Inference device, imaging apparatus, method for controlling inference device, and program

Abstract

To execute two or more machine learning models partially in parallel which require sequential execution.SOLUTION: An imaging apparatus has a first machine learning model having a plurality of learners and outputting likelihood to a plurality of classes, and a plurality of second machine learning models corresponding to the plurality of classes. The imaging apparatus executes in parallel the operation of the second machine learning model corresponding to a first class selected from the plurality of classes based on the likelihood calculated during the operations of the plurality of learners in the first machine learning model, and the operations of remainders of the plurality of learners in the first machine learning model. When a second class selected from the likelihood when all the operations of the plurality of learners in the first machine learning model are completed, is identical to the first class, the imaging apparatus continues the operation of the second machine learning model corresponding to the first class and outputs an inference result.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an inference device that performs inference in subject detection. [Background technology]

[0002] In recent years, camera settings have become more automated, allowing the camera to determine and set various parameters to automatically take appropriate photos. This automation can sometimes be achieved using machine learning models.

[0003] Known machine learning models include methods such as decision trees and random forests, which use multiple decision trees to ensemble learning of training data by bootstrap sampling. In recent years, a method called gradient boosting decision trees (GBDT) has also been used, which learns the residuals of inferred values ​​using multiple decision trees.

[0004] As cameras become more automated and more advanced judgments become necessary, multiple learning models will be used (ensemble learning), which will increase the amount of calculations. For example, in cases where the inference of one machine learning model cannot begin until the inference of the next machine learning model is completed, such as when machine learning models 2 and 3 are selected for use based on the judgment result of machine learning model 1, sequential calculations will be required.

[0005] Patent Document 1 is a method used for production equipment, but it calculates the remaining processing time of the prediction model, and if it is not enough, it uses intermediate values ​​to make a decision. In other words, it uses multiple machine learning models and uses the results without calculating all of the multiple weak classifiers. This method can reduce the amount of calculation when sequential calculation is required. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-179468 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, since the machine learning model of Patent Document 1 is supposed to guarantee the required accuracy by the designed number of weak classifiers, if the calculation is in progress, it will not be possible to calculate the original number of weak classifiers. Therefore, the accuracy of the calculation result is not necessarily guaranteed.

[0008] In addition, although the processing performance of recent CPUs has improved, there is a limit to the processing speed of battery-powered devices such as cameras, compared to PCs that are connected to external power sources. Decision trees are lightweight as machine learning models, but the calculation time becomes an issue when multiple decision trees are used sequentially. [Means for solving the problem]

[0009] In view of the above problem, an inference device according to the present invention comprises a first machine learning model having a plurality of learning devices and outputting likelihoods for a plurality of classes, and a plurality of second machine learning models corresponding to the plurality of classes, and executes an operation of the second machine learning model corresponding to a first class selected from the plurality of classes based on a likelihood calculated during the operation of the plurality of learning devices in the first machine learning model in parallel with the remaining operations of the plurality of learning devices in the first machine learning model, and if the second class selected from the likelihood when all operations of the plurality of learning devices in the first machine learning model are completed is the same as the first class, continues the operation of the second machine learning model corresponding to the first class and outputs an inference result, and if the first class and the second class are different, interrupts the operation of the second machine learning model corresponding to the first class, starts the operation of the second machine learning model corresponding to the second class, and outputs the inference result. Effect of the Invention

[0010] According to the present invention, even in the case of a machine learning model that requires multiple sequential executions, the second and subsequent machine learning models can be speculatively executed, and the final inference result may be obtained earlier than in the case of simple sequential execution. Furthermore, after the speculative execution of the machine learning model is started, the model currently being inferred and the next model can be calculated in parallel. [Brief description of the drawings]

[0011] [Figure 1] Block diagram of interchangeable lens camera system [Diagram 2] Flowchart of behavior detection process [Diagram 3] Flowchart of behavior detection process [Figure 4] Flowchart of behavior detection process [Diagram 5] Flowchart of behavior detection process [Figure 6] Flowchart of behavior detection process [Figure 7] Examples of captured and displayed images DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments for carrying out the present invention will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] First Embodiment 1 is a diagram showing the configuration of an interchangeable lens camera as an example of an imaging device according to a first embodiment of the present invention. The imaging device of this embodiment is composed of an interchangeable lens unit 10 and a camera 20. A lens control unit 106 that controls the overall operation of the lens and a camera control unit 30 that controls the overall operation of the camera system including the lens unit 10 can communicate with each other via a terminal provided on the lens mount.

[0014] First, the configuration of lens unit 10 will be described. Fixed lens 101, aperture 102, and focus lens 103 constitute an imaging optical system. Aperture 102 is driven by aperture drive unit 104 and controls the amount of light incident on image sensor 201, which will be described later. Focus lens 103 is driven by focus lens drive unit 105 and the focal distance of the imaging optical system changes depending on the position of focus lens 103. Aperture drive unit 104 and focus lens drive unit 105 are controlled by lens control unit 106 and determine the opening amount of aperture 102 and the position of focus lens 103.

[0015] The lens operation unit 107 is a group of input devices for the user to perform settings related to the operation of the lens unit 10, such as switching between AF (autofocus) / MF (manual focus) modes, adjusting the position of the focus lens by MF, setting the operating range of the focus lens, setting the image stabilization mode, etc. When the lens operation unit 107 is operated, the lens control unit 106 performs control according to the operation.

[0016] The lens control unit 106 controls the aperture driving unit 104 and the focus lens driving unit 105 in response to control commands and control information received from the camera control unit 30 (described later), and also transmits lens control information to the camera control unit 30.

[0017] In this embodiment, the interchangeable lens unit is detachably attached to the camera body, but the lens unit may be provided integrally with the camera body.

[0018] Next, a description will be given of the configuration of the camera 20. The camera 20 is configured so as to be able to obtain an image pickup signal from a light beam that has passed through the photographing optical system of the lens unit .

[0019] The image sensor 201 is composed of a CCD or CMOS sensor. A light beam incident from the photographing optical system of the lens unit 10 forms an image on the light receiving surface of the image sensor 201, and is photoelectrically converted into a signal charge according to the amount of incident light by a photodiode provided in each pixel arranged in the image sensor 201. The signal charge accumulated in each photodiode is sequentially read out from the image sensor 201 as a voltage signal according to the signal charge by a drive pulse output by a timing generator 214 according to a command from the camera control unit 30.

[0020] The CDS / AGC / AD converter 202 (hereinafter referred to as "converter 202" for convenience) performs correlated double sampling, gain adjustment, and AD conversion to remove reset noise on the imaging signal read out from the image sensor 201. The converter 202 outputs the imaging signal and AF signal that have been subjected to these processes to the image input controller 203 and the AF signal processing unit 204, respectively.

[0021] The image input controller 203 stores the imaging signal output from the converter 202 in the SDRAM 209 as an image signal via the bus 21. The image signal stored in the SDRAM 209 is read out by the display control unit 205 via the bus 21 and displayed on the display unit 206. In a recording mode in which the image signal is recorded, the image signal stored in the SDRAM 209 is recorded by the recording medium control unit 207 in a recording medium 208 such as a semiconductor memory.

[0022] The ROM 210 stores control programs and processing programs executed by the camera control unit 30, as well as various data required for executing these programs. The flash ROM 211 stores various setting information related to the operation of the camera 20 set by the user.

[0023] The joint detection unit 301 in the camera control unit 30 detects and stores each joint, organ, etc. of the target person in the imaging signal input from the image input controller 203. The detection method is a known method using a neural network or other machine learning model, and will not be described in detail.

[0024] The event detection unit 302 inputs the joint information output from the joint detection unit 301 and infers the sport event (class) that the target person is engaged in. The events are, for example, track and field, soccer, tennis, etc., and in this specification, they are events A, B, and C, but more types may be selected. These are assumed to estimate the probability likelihood of each of the events A to C, and in particular, in the third embodiment described later, a softmax function is used, and the sum of the probability likelihoods is 100%. The detection method is an ensemble learning model consisting of weak classifiers (learners), and in this embodiment, GBDT is used, but random forest and other methods may be used. These are assumed to use known methods and will not be described in detail.

[0025] The action detection units 303 to 305 input the joint information output from the joint detection unit 301, and infer the probability likelihood that the target person is performing a specific action. An example of an action is the action of swinging a racket in tennis. The detection method is an ensemble learning model consisting of weak classifiers (learners), and in this embodiment, GBDT is used, but random forest and other methods may be used. These are assumed to use known methods and will not be described in detail. There are multiple action detection units, and there are action detection units A to C corresponding to events A to C, which are used separately according to the algorithm described later. In addition, the multiple action detection units may each be a different Algorithm, or the same Algorithm may be used with parameters replaced.

[0026] The image processing unit 306 applies predetermined image processing to the image signal stored in the SDRAM 209. The image processing applied by the image processing unit 306 includes, but is not limited to, so-called development processing such as white balance adjustment processing, color interpolation (demosaic) processing, and gamma correction processing, as well as signal format conversion processing and scaling processing.

[0027] The camera control unit 30 controls each unit in the camera 20 while exchanging information with them. In addition, the camera control unit 30 executes various processes corresponding to user operations such as power ON / OFF, changing various settings, imaging processing, AF processing, and playback processing of recorded images, in response to an input from a camera operation unit 213 based on a user operation. Furthermore, the camera control unit 30 transmits control commands for the lens unit 10 (lens control unit 106) and information about the camera 20 to the lens control unit 106, and acquires information about the lens unit 10 from the lens control unit 106. The camera control unit 30 is configured by a microcomputer, and controls the entire camera system including the interchangeable lens 10 by executing a computer program stored in the ROM 210.

[0028] The following describes the processes performed by the camera 20. For each process described below, the camera control unit 30 controls each unit of the camera 20 in accordance with an imaging processing program, which is a computer program, and performs the following processes.

[0029] 2 is a flow chart showing a procedure for detecting the behavior of a person on the camera 20. Here, a conventional example that is not performed in the present invention is shown for comparison with the embodiments described later. S stands for step.

[0030] First, in S201, the camera control unit 30 extracts feature amounts from an image. In this embodiment, the joint detection unit 301 extracts human features, in this case the position coordinates of joints, from the image.

[0031] In S202, the sports event detection unit 302 estimates the sports event with the highest likelihood among those inferred using the features of the person as the sports event that the person is participating in. In this embodiment, the likelihoods of the three sports A, B, and C are output.

[0032] In S203, the specific behavior of the person is estimated from the feature amount of the person using the behavior detection units 303 to 305 corresponding to the event with the highest likelihood in S202. Since each behavior detection unit has learned the specific behavior of the respective sports event, the feature of the person that fits the behavior is output as a high likelihood.

[0033] By the above, it is possible to estimate that a person in an image is performing a specific action in a specific sport. However, in this case, the processing time of the event detection unit 203 and the event detection units 303 to 305 must be performed in order, so it takes the calculation time of two machine learning models. If there are multiple people in an image, these are executed for each person.

[0034] 3 is a flowchart showing a procedure for detecting the behavior of a person in the camera 20. S stands for step.

[0035] 2 is that the events detection unit 302 and the behavior detection units 303-305 each perform GBDT with the same number of decision trees, and the processing is divided. That is, the event detection unit 302 performs the event estimation processing for one person by dividing it into event determination 1 (S302) and event determination 2 (S303). Also, the behavior detection units 303-305 perform the behavior detection processing for one person by dividing it into behavior detection 1 (S304) and behavior detection 2 (S306). Here, the ratio of each divided process may be half, or the ratio may be changed as appropriate.

[0036] When the event detection unit 302 performs event determination 1 (S302), the processing of the first half of the GBDT's weak classifiers is completed. Generally, in GBDT, residuals are learned in sequence using weak classifiers during model training. Therefore, in event determination 1, partial processing of the weak classifiers in the event determination unit 302 is performed, and in event determination 2, all processing of the weak classifiers is completed. If the sum of the values ​​of the leaf nodes at a certain point in time of event determination is then multiplied by a sigmoid function, event estimation can be performed, albeit with low accuracy. This means that event estimation can be performed at the event determination 1 stage, albeit with low accuracy.

[0037] Hereinafter, the details of the flowchart in FIG. 3 will be described assuming that S303 and S304, and S306 and S307 are processed simultaneously in parallel.

[0038] First, in S301, the camera control unit 30 extracts features from a captured image. The image captured by the camera 20 is shown in Fig. 7(a) 701. In this embodiment, the joint detection unit 301 extracts features of a person 702 from the image 701, in this case the position coordinates 703 of each joint.

[0039] In S302, the event detection unit 302 uses the feature amounts of the person 702 to perform event determination 1 processing to estimate the sport that the person 702 is playing. As described above, event determination 1 is the first half of the divided estimation processing of the event detection unit 302, and low-precision event estimation is performed here. In other words, the event detection unit 302 estimates the highest likelihood among the likelihoods for each event calculated with low precision as the sport. Furthermore, the image 701 is displayed on the display unit 206 after development processing as shown in 704. At that time, the currently estimated event is displayed as shown in icon 705.

[0040] In S303, the sport detection unit 302 uses the feature amount of the person 702 to perform sport judgment 2 processing to estimate the sport that the person is playing. This completes all processing of the weak classifier in sport judgment, so that the sport can be estimated with the designed accuracy. Similarly, the estimated sport is displayed as icon 705.

[0041] In S304, behavior detection 1 corresponding to the first half of the behavior estimation process of the behavior detection units 303 to 305 corresponding to the event estimated with low accuracy in event determination 1 in S302 is speculatively performed. That is, behavior detection 1 is processed in parallel with event determination 2 in S303.

[0042] In S305, the camera control unit 30 compares the low-accuracy event estimation result in S302 with the event estimation result with a predetermined accuracy in S303. If the event estimation results match, the process proceeds to S306, and if they do not match, the process proceeds to S307.

[0043] In S306, since the event that is the premise of the speculatively executed event detection in S304 was correct, event detection 2, which is the latter half of the event estimation process of the event detection units 303 to 305 for the same event, is performed to estimate the person's event. In other words, event detection for the same event is continued. In this case, the event detection unit's process and the event detection unit's process, which originally had to be executed sequentially as shown in FIG. 2, can be processed in parallel. If the event detection unit and the event detection unit's process required the same amount of time and the first half and the second half were divided in half, the event could be estimated with the final accuracy in 3 / 4 of the processing time required for event estimation shown in the flow of FIG. 2.

[0044] In S307, the event that was the premise of the speculatively executed behavior detection in S304 was incorrect, so the processing of the behavior detection units 303 to 305 corresponding to the event estimated with the designed accuracy in S303 is performed without division (Behavior Detection'). In this case, although the execution of the speculative behavior detection 1 in S304 based on the estimated result of the event determination 1 in S302 was incorrect, the processing time is almost the same as the sequential execution shown in Figure 2, excluding the load of the determination process etc. in S305.

[0045] If there are multiple people in the image, the above process is executed for each person.

[0046] The captured image 701 is recorded in the storage medium 208 by the storage medium control unit 207. At that time, the event estimated in S302 and S303 and the action estimated in S306 and S307 are recorded in the storage medium 208 together with the image data.

[0047] As described above, according to the first embodiment of the present invention, even in the case of a machine learning model that requires multiple sequential executions, the second and subsequent machine learning models can be speculatively executed, and the final inference result may be obtained earlier than in the case of simple sequential execution. Furthermore, after the speculative execution is started, the model currently being inferred and the next model can be calculated in parallel.

[0048] Second Embodiment Hereinafter, a process in the case where the required calculation time is short will be described as a modification of the first embodiment. In this embodiment, except for the process that will be described separately, the same configuration and control as the first embodiment will be performed.

[0049] 4 is a flowchart showing a procedure for detecting the behavior of a person in the camera 20 according to the second embodiment. The camera control unit 30 controls each unit of the camera 20 according to an image capturing processing program, which is a computer program, and performs the following processes. In addition, the processes of S403 and S404 described later are processed simultaneously in parallel.

[0050] First, in S401, the camera control unit 30 extracts features from an image. Here, an explanation will be given by taking the image 701 in FIG.

[0051] In S402, the event detection unit 302 performs event determination 1 as a process for estimating the sport event being performed by the person 702, using the feature amount of the person 702. As described in S302 of Fig. 3, the event determination 1 corresponds to the first half of the divided processes in the estimation process of the event detection unit 302, and results in low-accuracy event estimation.

[0052] In S403, the sport detection unit 302 uses the feature amount of the person 702 to perform a sport judgment 2 process for estimating the sport the person is playing.

[0053] In S404, behavior detection 1 corresponding to the first half of the behavior estimation process of the behavior detection units 303 to 305 corresponding to the event estimated with low accuracy in event determination 1 in S402 is speculatively performed. That is, behavior detection 1 is performed in parallel with event determination 2 in S403.

[0054] In S405, the camera control unit 30 compares the low-accuracy event estimation result in S402 with the event estimation result with a predetermined accuracy in S403. If the respective estimation results match, the process of S406 is performed, and if they do not match, the process ends here. In this embodiment, unlike the first embodiment, if the event that is the premise of the speculatively executed behavior detection 1 in S304 is incorrect, the latter half of the process is not performed. In other words, since the behavior estimation process cannot be performed, the estimation result from the previous shooting may be used instead if necessary.

[0055] In S406, since the event that is the premise of the speculatively executed event detection in S304 was correct, the event detection unit 303-305 executes the latter half of the event event estimation process, event detection 2, to estimate the person's event. In this case, the event detection unit and event detection unit processes, which originally needed to be executed sequentially as shown in FIG. 2, can be executed in parallel.

[0056] As a result of the above, there are cases where behavior cannot be estimated, but if the event detection unit and behavior detection unit require the same amount of time for processing and the first half and second half are divided in half, then if the speculative execution is correct, the processing time will be 3 / 4 of the processing time required for the flow in Figure 2. Also, if the speculative execution is incorrect, processing can be completed in 1 / 2 the processing time of the flow chart in Figure 2.

[0057] As described above, according to the second embodiment of the present invention, even in the case of a machine learning model that requires multiple sequential executions, the second and subsequent machine learning models can be speculatively executed, and the final inference result may be obtained earlier than in the case of simple sequential execution. After the speculative execution is started, the model currently being inferred and the next model can be calculated in parallel. If the speculative execution is incorrect, the calculation is terminated, and the overall processing time can be designed to be shorter than the execution time of the two machine learning models.

[0058] Third embodiment Hereinafter, as a modified example of the first embodiment, an example in which the computational resources of the behavior detection unit are distributed according to the likelihood of each event will be described. In this embodiment, except for the processing to be described separately, the same configuration and control as in the first embodiment will be performed.

[0059] 5(a) is a flowchart showing a procedure for detecting the behavior of a person in the camera 20 according to the third embodiment. The camera control unit 30 controls each unit of the camera 20 according to an image capturing processing program, which is a computer program, and performs the following processing. S503, S504, S505, and S506 are processed simultaneously in parallel.

[0060] First, in S501, the camera control unit 30 extracts features from an image. Here, an explanation will be given by taking the image 701 in FIG.

[0061] In S502, the event detection unit 302 performs event determination 1 processing to estimate the sport event performed by the person 702 using the feature amount of the person 702. As described above, event determination 1 is the first half of the divided estimation processing by the event detection unit 302, and here the event is estimated with low accuracy.

[0062] In S503, the sport detection unit 302 performs a sport determination 2 process to estimate the sport that the person 702 is playing by using the feature amount of the person 702.

[0063] In S504 to S506, the first half of the behavior estimation process of the behavior detection units 303 to 305 corresponding to the events estimated with low accuracy in S502 is speculatively performed. Here, behavior detection 1A, behavior detection 1B, and behavior detection 1C performed in parallel correspond to the multiple events estimated in S502, respectively, and the following description assumes that three events (A to C) were estimated in S502. Note that the number of behavior estimation processes performed in parallel may be changed depending on the number of estimated events.

[0064] 5(b) is an example of inference likelihood and computational resource distribution when there are three events to be inferred. As shown in this table, in behavior detection 1A (S504), if the inference likelihood for event A inferred in S502 is 0.6, behavior detection unit A303 executes inference processing with a computational resource ratio of 0.6 until S503 ends. Similarly, processing for S505 and S506 is performed with a computational resource ratio corresponding to the estimated likelihood.

[0065] When the process of S503 is completed, S504 to S506 are temporarily suspended. When S503 is completed, the event detection unit 302 has completed event estimation with the designed accuracy, and the likelihood that all event estimation processes have been completed is calculated.

[0066] In S507, the remaining calculation of the behavior detection that was interrupted at the time of the end of S503 is performed for the event with the highest likelihood estimated in S503, and the final behavior detection is estimated. Here, it is assumed that the machine learning models of the event detection unit 302 and the behavior detection units A 303 to 305 all take the same processing time, and that the event detection unit 302 and the behavior detection units A 303 to 305 are executed by different processors with the same processing speed. In this case, as a best example, if the inference in S502 is the event A and the inference in S503 is also the event A, 0.6×0.5=30% of the processing can be speculatively executed in parallel in S504, and the remaining 70% of the processing is performed in S507. Therefore, the calculation can be performed in a total processing time of 170 / 200=85% of the processing time required for the flowchart in FIG. 2.

[0067] On the other hand, in the worst case scenario, if the inference in S502 is type A and the inference in S503 is type C, then 0.1 x 0.5 = 5% of the processing can be speculatively executed in parallel in S504, and the remaining 95% of the processing is performed in S507. Therefore, the calculation can be performed in 195 / 200 = 97.5% of the total processing time required for the flow in Figure 2.

[0068] As described above, according to the third embodiment of the present invention, even in the case of a machine learning model that requires multiple sequential executions, the second and subsequent machine learning models can be speculatively executed, and the final inference result may be obtained earlier than in the case of simple sequential execution. After the speculative execution is started, the model currently being inferred and the next model can be calculated in parallel. During the parallel calculation, the speculative inference is allocated to the calculation resources based on the likelihood at that time, so that even if the final inference result of the behavior detection unit 302 differs from the inference result in S502, the processing time can be shortened compared to the original sequential execution. In this embodiment, an example in which the calculation resources are allocated based on the likelihood of each type has been introduced, but allocation based on other conditions may also be used.

[0069] (Fourth embodiment) Hereinafter, as a modified example of the first embodiment, an example will be described in which the behavior detection unit 302 is reviewed multiple times in parallel with the event detection unit 302. In this embodiment, except for the processing that will be described separately, the same configuration and control as in the first embodiment will be performed.

[0070] 6 is a flow chart showing the procedure for detecting the behavior of a person in the camera 20 according to the fourth embodiment. The camera control unit 30 controls each unit of the camera 20 according to an image capture processing program, which is a computer program, and performs the following processing. It is assumed that S603 and S604, and S605 and S606 are processed simultaneously in parallel. In this embodiment, the event determination processing performed by the event detection unit 302 is divided into three, S602, S603, and S604, but may be further divided.

[0071] First, in S601, the camera control unit 30 extracts features from an image. Here, an explanation will be given by taking the image 701 in FIG.

[0072] In S602, the sport detection unit 302 performs a sport judgment 1 process to estimate the sport played by the person 702 using the feature amount of the person 702, and estimates the sport.

[0073] In S603, the sport detection unit 302 performs a sport judgment 2 process to estimate the sport being played by the person 702 using the feature amount of the person 702, and estimates the sport.

[0074] In S604, a part of the behavior estimation process of the behavior detection units 303 to 305 corresponding to the event estimated in S602 is performed until the process of S603 is completed.

[0075] In S605, the event detection unit 302 uses the features of the person 702 to perform event determination process 3 to estimate the sport event that the person 702 is participating in, and estimates the event using the likelihood determined when all event estimation processes are completed.

[0076] In S606, if the event estimated in S603 is the same as the event estimated in S602, part of the process following S604 is carried out. On the other hand, if the estimation result in S603 is different, part of the corresponding behavior estimation process of the behavior detection units 303 to 305 is reviewed, and recalculation processing is carried out again until the processing of S605 is completed.

[0077] In S607, if the event estimated in S605 is the same as the event estimated in S604, the remaining processing following S606 is performed, and if the estimation result in S605 is different, a corresponding calculation process is performed to review the behavior estimation process of the behavior detection units 303 to 305.

[0078] In this way, although the present embodiment reviews the speculatively executed action detection two times, the speculatively executed action detection is reviewed multiple times for each calculation. In the case of an ensemble learning model and especially a GBDT, the more weak classifiers are processed, the closer the estimation result will be to the final result. Therefore, by reviewing the speculatively executed action detection multiple times, it is possible to reduce unnecessary speculative execution.

[0079] As described above, according to the fourth embodiment of the present invention, even in the case of a machine learning model that requires multiple sequential executions, the second and subsequent machine learning models can be speculatively executed, and the final inference result may be obtained earlier than in the case of simple sequential execution. After the speculative execution is started, the model currently being inferred and the next model can be calculated in parallel. And, by reviewing the speculative execution multiple times, unnecessary speculative execution can be reduced.

[0080] (Other embodiments) Except for the processing described below, the configuration and control are the same as those in the first embodiment.

[0081] In the present invention, for example, S303 and S304 are uniformly described as parallel processing, and are basically described as being executed in parallel by a multiprocessor. However, this may be time-divisionally processed by one or more processors.

[0082] The present invention describes an embodiment of a camera. In the case of a camera, the first behavior detection is performed when the power is turned on, when live view is started, when continuous still image shooting starts, when video recording mode is started, when video recording starts, etc. This also applies when the inference device itself is turned on or when the mode is changed. In this case, it is possible to consider processing that does not perform speculative execution the first time (immediately after startup, etc.) just to be sure. Also, it is possible to consider processing that does not perform speculative execution even if the type detected is different from the last time. In this case, the last time refers to the previous frame in the case of video or live view, and refers to the previous captured frame in the case of still images when there is no live view between frames.

[0083] Furthermore, the behavior detection units 303 to 305 output behavior likelihoods, but generally a threshold is set, and if the likelihood is equal to or greater than the threshold, it is determined that a specific behavior has been performed.

[0084] When performing speculative execution, it is possible to offset the likelihood of action estimation lower or to raise the threshold, making it harder to detect incorrect actions.

[0085] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0086] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. [Explanation of symbols]

[0087] 10 Lens unit 101 Fixed Lens 102 Aperture 103 Focus Lens 104 Aperture drive unit 105 Focus lens drive unit 106 Lens control unit 107 Lens operation section 20 Camera 21 Bus 201 Image sensor 202 CDS / AGC / AD Converter 203 Image Input Controller 204 AF signal processing section 205 Display control unit 206 Display section 207 Recording medium control section 208 Recording media 209 SDRAM 210 ROM 211 Flash ROM 213 Camera operation unit 214 Timing Generator 30 Camera control unit 301 Joint detection unit 302 Event detection unit 303 Behavior Detection Unit A 304 Behavior Detection Unit B 304 Behavior Detection Unit C

Claims

1. Having an inference means for performing inference processing, The inference means is Execute a calculation using a first learning model that outputs likelihoods for a plurality of classes; Executing, in parallel with the calculation of the first learning model, a calculation of a second learning model corresponding to one or more classes selected from the plurality of classes based on the likelihood calculated at a first time point during the calculation of the first learning model; An inference device characterized in that, when the calculation of the first learning model is different from one or more classes selected from the plurality of classes based on the likelihood calculated at the first time point, the calculation of the second learning model corresponding to the class selected from the likelihood calculated at the second time point is started.

2. An inference device as described in Claim 1, characterized in that if the class selected from the likelihood calculated at the second time point is different from one or more classes selected from the multiple classes based on the likelihood calculated at the first time point, calculation of the second learning model corresponding to one or more classes selected from the multiple classes based on the likelihood calculated at the first time point is interrupted.

3. An inference device as described in Claim 1, characterized in that if a class selected in the first learning model based on the likelihood calculated at the second time point matches one or more classes selected from the multiple classes based on the likelihood calculated at the first time point, the device continues calculating the second learning model corresponding to the matched class.

4. An inference device as described in claim 1, characterized in that it performs calculations of the multiple second learning models using a resource ratio based on the likelihood of each of the multiple classes calculated at the first point in time.

5. An inference device as described in claim 1, characterized in that the calculation of the first learning model and the calculation of the multiple second learning models are performed in a time-division manner.

6. The inference device described in Claim 1, characterized in that the first learning model and the second learning model are gradient boosting decision trees.

7. further comprising a display means for displaying an image; 2. The inference device according to claim 1, wherein said display means displays information about the class being calculated during the calculation of said second learning model.

8. An imaging means; an inference device according to claim 1, wherein the image acquired from the imaging means is used for inference; An imaging device comprising:

9. A first calculation step using a first learning model that outputs likelihoods for multiple classes; a second calculation step using a plurality of second learning models corresponding to the plurality of classes, In the first calculation step, one or more classes are selected from the plurality of classes based on a likelihood calculated at a first point in time during calculation of the first learning model; the second calculation step executes calculation of a second learning model corresponding to the one or more classes in parallel with calculation of the first learning model in the first calculation step; In the first calculation step, a class is selected from the plurality of classes based on a likelihood calculated at a second time point that is later than the first time point in the calculation of the first learning model; A control method characterized in that, in the second calculation step, if the class selected based on the likelihood calculated at the second time point is different from one or more classes selected from the plurality of classes based on the likelihood calculated at the first time point, calculation of a second learning model corresponding to the class selected based on the likelihood calculated at the second time point is started.

10. A program for causing a computer to execute the method for controlling an inference device according to claim 9.

11. A computer-readable storage medium storing the program according to claim 10.

Images