Information processing device and information processing method

Abstract

To provide an information processing apparatus and an information processing method for completing learning related to frames by the timing to perform inference related to the frames.SOLUTION: In a method, when inference related to a first frame is completed but learning related to the first frame is not completed in a first frame period corresponding to the first frame, the learning related to the first frame and inference and learning related to a second frame subsequent to the first frame are executed in a second frame period corresponding to the second frame.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to techniques relating to inference and learning. [Background technology]

[0002] Machine learning is an effective method for tracking objects in images across multiple frames. In general, machine learning object tracking uses a discriminant model such as a neural network or a support vector machine to detect the target for each frame. This step is called inference, and the step of updating the parameters of the discriminant model based on the results of inference is called learning.

[0003] In object tracking using machine learning, online learning is often used. Non-Patent Document 1 discloses a method for object tracking using a neural network. In the method disclosed in Non-Patent Document 1, some of the parameters of a discrimination model are updated using the result of inference for each frame. This makes it possible to track objects with various characteristics using one type of discrimination model.

[0004] On the other hand, in the process of actually tracking an object, the process may be interrupted in the middle of a frame period. Patent Document 1 discloses a multi-camera system that uses online learning to track a subject. In the camera system disclosed in Patent Document 1, a reset is performed by a user's operation, and the system returns to an initial state. Patent Document 2 discloses a method for performing online learning in object recognition using machine learning. In the method disclosed in Patent Document 2, online learning is interrupted when input data that is an outlier is received, and after recovery, the process is resumed using the latest learning result before the interruption. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-139725 A [Patent Document 2] Special Publication No. 2018-509712 [Non-patent literature]

[0006] [Non-Patent Document 1] “Discriminative and Robust Online Learning for Siamese Visual Tracking”, J. Zhou et al., Vol 34 No 07: AAAI-20 Technical Tracks 7(2020) Summary of the Invention [Problem to be solved by the invention]

[0007] Some applications require that the tracking state of a target be maintained even after resuming from an interruption. For example, consider a case where a surveillance camera is equipped with a function that records still images when it recognizes a specific movement of the target to be tracked. In such a case, an interrupt occurs due to the still image recording process, which has a higher priority than the tracking process. This causes the tracking process to be interrupted in the middle of a frame period, but it is desirable to maintain the tracking state of the target in the next frame and beyond.

[0008] In order to maintain the tracking state of the target, it is important to keep the parameters of the discrimination model up to date. The orientation and size of the target to be tracked in the image continue to change, so the features change slightly for each frame. If detection of the target by inference for each frame fails due to this change in features, the tracking state is released. To prevent this, it is necessary to perform learning for each frame and perform inference using learning parameters that reflect the latest features.

[0009] However, in the camera system disclosed in Patent Document 1, the system returns to an initial state, so that in order to track the same object as before the interruption, online learning must be repeated. In the method disclosed in Patent Document 2, tracking can be resumed using the learning parameters before the interruption, but the results of online learning during the frame period in which the interruption occurred are not reflected. The present invention provides a technique for completing frame-related learning by the time when frame-related inference is performed. [Means for solving the problem]

[0010] One aspect of the present invention includes a processing means for performing inference related to each frame and learning related to the frame during a frame period corresponding to the frame, and when inference related to the first frame is completed during a first frame period corresponding to the first frame but learning related to the first frame is not completed, the processing means performs inference related to the first frame and learning related to the second frame during a second frame period corresponding to a second frame following the first frame. Effect of the Invention

[0011] According to the configuration of the present invention, learning related to a frame can be completed by the time when inference related to the frame is performed. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram showing an example of a configuration of a discriminant model and a data flow in the discriminant model. [Diagram 2] Time chart. [Diagram 3] Time chart. [Figure 4] 4 is a flowchart of a process performed by an information processing device. [Diagram 5] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device. [Figure 6] FIG. 4 is a block diagram showing an example of the arrangement of a calculation unit 507. [Figure 7] Time chart. [Figure 8] 4 is a flowchart of a process performed by an information processing device. [Figure 9] Time chart. [Figure 10] 4 is a flowchart of a process performed by an information processing device. [Figure 11] Time chart. [Figure 12] 4 is a flowchart of a process performed by an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] [First embodiment] In this embodiment, in a configuration in which inference related to each frame and learning related to the frame are performed in a frame period corresponding to the frame, if inference related to the second frame is completed in a second frame period corresponding to a second frame that is older than a first frame, and learning related to the second frame is not completed in the second frame period, learning related to the second frame and inference and learning related to the first frame are performed in a first frame period corresponding to the first frame. This makes it possible to complete learning before the next inference that requires the results of the learning.

[0015] In this embodiment, a case will be described in which a tracking process is performed by using an identification model for identifying a tracking target in a frame as an inference process and then tracking the tracking target. In this embodiment, a convolutional neural network, which is an example of a hierarchical neural network, is used as such an identification model, but other types of models may also be used.

[0016] An example of the configuration of a discriminative model according to this embodiment and a data flow in the discriminative model will be described with reference to the block diagram of Fig. 1. As shown in Fig. 1, the discriminative model has a first convolutional layer 101 and a second convolutional layer 102.

[0017] In the first convolutional layer 101, a first coefficient 104 (weighting coefficient in the first convolutional layer 101) is set as a learned parameter. In the second convolutional layer 102, a second coefficient 106 (weighting coefficient in the second convolutional layer 102) is set as a learned parameter. The first convolutional layer 101 in which the first coefficient 104 is set processes the input frame 103 and outputs an intermediate output 105. The second convolutional layer 102 in which the second coefficient 106 is set processes the intermediate output 105 and outputs a tracking result 107 indicating tracked target information such as the position and size of the tracked target included in the frame 103.

[0018] The identification model operates in the same manner for each input frame, and performs processing (inference processing) to output a tracking result corresponding to the frame. Each input frame may be an image of each frame in a captured video, or may be a still image captured periodically or irregularly.

[0019] In addition, in this embodiment, the second convolutional layer 102 of the discriminative model supports online learning. That is, the second coefficients 106 are learned (updated) for each frame so that the second convolutional layer 102 can output a tracking result that is more accurate tracked target information.

[0020] In the online learning, the second coefficient 106 is learned (updated) using the intermediate output 105 input to the second convolutional layer 102 and the tracking result 107 output from the second convolutional layer 102. For example, a map is prepared in advance as correct answer data, in which a label value "1" is assigned to the area of ​​the tracking target in the frame 103 input to the first convolutional layer 101 and a label value "0" is assigned to the area other than the area of ​​the tracking target in the frame 103. Then, the frame 103 is input to the first convolutional layer 101 to obtain the intermediate output 105 output from the first convolutional layer 101 and the tracking result 107 output from the second convolutional layer 102. Then, the difference (error) between the tracking result 107 and the correct answer data is obtained, and the second coefficient 106 is updated using the error and the intermediate output 105 by a known method such as the backpropagation method. Since such an online learning method is a known technique, further description thereof will be omitted. Also, the online learning method is not limited to a specific method.

[0021] As mentioned above, updating the second coefficient 106 by online learning is essential to maintain the tracking state of the tracked target. The characteristics of the tracked target moving within a frame change for each frame. In order to continue to detect such a tracked target by inference processing for each frame, i.e., processing of the discriminative model shown in FIG. 1, it is necessary to perform online learning for each frame and continue to obtain the second coefficient 106 corresponding to the change in the characteristics.

[0022] FIG. 2 shows a time chart showing the allocation of time when performing inference processing and learning processing for each frame. In the first frame period, which is a processing period corresponding to the first frame, first, an inference processing 201 using a discriminative model is performed for the first frame, and then, using the result of the inference processing 201, a learning processing 202 (online learning of the second convolution layer 102) of the discriminative model is performed to update the second coefficient 106. Thereafter, in the second frame period, which is a processing period corresponding to the second frame, first, an inference processing using a discriminative model is performed for the second frame, and then, using the result of the inference processing, a learning processing (online learning of the second convolution layer 102) of the discriminative model is performed to update the second coefficient 106. Thereafter, in the third frame period, which is a processing period corresponding to the third frame, first, an inference processing using a discriminative model is performed for the third frame, and then, using the result of the inference processing, a learning processing (online learning of the second convolution layer 102) of the discriminative model is performed to update the second coefficient 106. In this manner, by performing the inference process and the learning process for each frame period, a tracking process is performed to track the tracking target while updating the second coefficient 106.

[0023] A time chart according to this embodiment is shown in Fig. 3. Fig. 3 shows a case where the learning process 302 in the first frame period is interrupted by an interrupt instruction from a user. Note that the interrupt instruction is not limited to an instruction input by a user, but may be, for example, an instruction issued by a control unit such as a processor of the device when some condition is satisfied, or an instruction received from an external device.

[0024] In this case, in the second frame period, first, the learning process 302 that was originally supposed to be executed in the first frame period is executed. Then, when the learning process 302 is completed, the remaining time of the second frame period is used to execute an inference process 303 using a discriminative model for the second frame, and then, using the result of the inference process 303, a learning process 304 of the discriminative model is executed to update the second coefficient 106. In the third frame period, as in Fig. 2, first, an inference process 305 using a discriminative model is executed for the third frame, and then, using the result of the inference process 305, a learning process 306 of the discriminative model is executed to update the second coefficient 106.

[0025] As a result, the learning process 302 can be completed by the time the inference process 303 in the second frame period is executed, so that the second coefficient 106 of the discrimination model used in the inference process 303 can be updated before the start of the inference process 303. In other words, even if the learning process 302 in the first frame period is interrupted, it is possible to reflect the result of the learning process 302 in the first frame in the inference process 303 in the second frame.

[0026] Next, a hardware configuration example of an information processing device according to this embodiment that performs the above processing will be described with reference to the block diagram of Fig. 5. Computer devices such as PCs (personal computers), smartphones, and tablet terminal devices can be applied to the information processing device according to this embodiment.

[0027] The CPU 506 executes various processes using computer programs and data stored in the RAM 509 and the ROM 508. As a result, the CPU 506 controls the operation of the entire information processing device, and executes or controls various processes described as processes performed by the information processing device. The number of CPUs 506 is not limited to one, and may be multiple.

[0028] The RAM 509 has an area for storing computer programs and data loaded from the ROM 508 or data storage unit 502, and an area for storing computer programs and data received from an external device via the communication unit 503. The RAM 509 further has work areas used when the CPU 506, the calculation unit 507, and the image processing unit 505 execute various processes. In this way, the RAM 509 can provide various areas as appropriate.

[0029] The ROM 508 stores setting data for the information processing device, computer programs and data related to the startup of the information processing device, computer programs and data related to the basic operation of the information processing device, etc. The computer programs and data stored in the ROM 508 are loaded into the RAM 509 as appropriate under the control of the CPU 506, and are processed by the CPU 506.

[0030] The input unit 501 is a user interface such as a keyboard, a mouse, or a touch panel, and the user can operate it to input various instructions to the CPU 506 , the calculation unit 507 , and the image processing unit 505 .

[0031] The data storage unit 502 is a large-capacity information storage device such as a hard disk drive device, etc. The data storage unit 502 stores computer programs and data for causing the CPU 506, the calculation unit 507, and the image processing unit 505 to execute or control various processes described as processes performed by the OS (operating system) and the information processing device.

[0032] The computer program stored in the data storage unit 502 includes a computer program for causing the CPU 506 and the calculation unit 507 to execute or control the operation of the above-mentioned discriminative model (first convolutional layer 101, second convolutional layer 102). In addition, the data stored in the data storage unit 502 includes parameters of the above-mentioned discriminative model (first coefficient 104, second coefficient 106).

[0033] The computer programs and data stored in the data storage unit 502 are loaded into the RAM 509 as appropriate under the control of the CPU 506 , and are then processed by the CPU 506 , the calculation unit 507 , and the image processing unit 505 .

[0034] The data storage unit 502 may be a device that reads and writes computer programs and data from and to a memory device. Examples of memory devices include flexible disks, CD-ROMs, CD-Rs, DVDs, memory cards, CF cards, smart media, SD cards, memory sticks, xD picture cards, and USB memories. In this case, the computer programs and data described above as being stored in the data storage unit 502 are stored in this memory device. In this case, the data storage unit 502 loads the computer programs and data stored in the memory device into the RAM 509 under the control of the CPU 506.

[0035] Furthermore, a part of the RAM 509 may be used as the data storage unit 502. In addition to or instead of the data storage unit 502, a storage device of a device connected via the communication unit 503 may be used.

[0036] The communication unit 503 is a communication I / F for performing data communication with an external device. In Fig. 5, the input unit 501, the data storage unit 502, and the display unit 504 are all included in the information processing device, but these devices may be connected to the information processing device via the communication unit 503.

[0037] The display unit 504 has a liquid crystal screen or a touch panel screen, and displays, as images or characters, the results of processing by the CPU 506, the calculation unit 507, and the image processing unit 505. The display unit 504 may be a projection device such as a projector that projects images and characters.

[0038] Note that, like a known touch screen device, the display unit 504 and the input unit 501 may be integrated. In that case, an input via the touch screen is treated as an input to the input unit 501.

[0039] When the image processing unit 505 receives a command from the CPU 506, it performs image processing such as range adjustment of pixel values ​​of the image stored in the data storage unit 502 in accordance with the command, and stores the processed image in the RAM 509.

[0040] When the calculation unit 507 receives a command from the CPU 506, it performs calculations necessary for the above-mentioned inference processing and learning processing. The calculation unit 507 has an internal memory necessary for executing the calculations. Various data shown in Fig. 1, such as the frame 103, the first coefficient 104, the second coefficient 106, and the tracking result 107, are all stored in the internal memory of the calculation unit 507 or in the RAM 509, and the calculation unit 507 reads and writes these data during the calculation process.

[0041] The input unit 501 , data storage unit 502 , communication unit 503 , display unit 504 , image processing unit 505 , CPU 506 , calculation unit 507 , ROM 508 , and RAM 509 are all connected to a system bus 510 .

[0042] Next, the process performed by the information processing device to perform the above-mentioned inference process and learning process for the current frame period will be described with reference to the flowchart in Fig. 4. Note that the information processing device can perform the inference process and learning process for each frame period (each frame) by performing the process according to the flowchart in Fig. 4 for each frame period (each frame).

[0043] In step S401, the CPU 506 determines whether or not first information indicating that the inference process has been completed in the frame period immediately preceding the current frame period (the previous frame period) is stored in the RAM 509. The CPU 506 also determines whether or not second information indicating that the learning process has been completed in the previous frame period is stored in the RAM 509.

[0044] If it is determined that the first information is stored in RAM 509 but the second information is not stored, CPU 506 determines that "inference processing has been completed in the previous frame period but learning processing has not been completed," and processing proceeds to step S402.

[0045] Also, if it is determined that both the first information and the second information are stored in RAM 509, CPU 506 determines that "the inference process and the learning process have been completed in the previous frame period", and the process proceeds to step S404.

[0046] Furthermore, if it is determined that neither the first information nor the second information is stored in RAM 509, the process proceeds to step S404. If neither the first information nor the second information is stored in RAM 509, and if restarting in the next frame period from the inference process that was to be performed in the previous frame period results in accumulation of frame processing discrepancies, the process that was to be performed in the previous frame period may be discarded.

[0047] It should be noted that if the current frame period is the first frame period, CPU 506 does not determine whether or not the first information and the second information are stored in RAM 509, and the process proceeds to step S404.

[0048] In step S402, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The calculation unit 507 performs learning processing for a frame (previous frame) corresponding to the previous frame period in accordance with the operation command (operation instruction). Then, the calculation unit 507 notifies the CPU 506 of the completion of the processing.

[0049] In step S403, the calculation unit 507 determines whether the learning process for the previous frame is completed. Also, in step S403, the CPU 506 determines whether an interrupt instruction has been issued. The interrupt instruction may be issued by the CPU 506 in response to a user operating the input unit 501, or may be issued by the CPU 506 in response to detection that some condition has been satisfied.

[0050] If the result of this determination is that the learning process for the previous frame is complete, the process proceeds to step S404. If the learning process for the previous frame is not yet complete, the process proceeds to step S402, where the learning process continues. If the CPU 506 determines that an interrupt instruction has been issued, the process for the current frame according to the flowchart in FIG. 4 ends.

[0051] In step S404, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The calculation unit 507 performs inference processing for a frame (current frame) corresponding to the current frame period in accordance with the operation command (operation instruction). The calculation unit 507 then stores the tracking result 107 in the RAM 509 and notifies the CPU 506 of the completion of processing.

[0052] In step S405, the calculation unit 507 determines whether the inference process for the current frame is completed (whether the tracking result 107 of the discrimination model in which the current frame is input as the frame 103 is obtained). Also, in step S405, the CPU 506 determines whether an interrupt instruction is issued.

[0053] If it is determined that the inference process for the current frame is complete (the tracking result 107 of the discriminative model in which the current frame is input as the frame 103 has been obtained), the process proceeds to step S406.

[0054] Also, if the inference process for the current frame is not yet complete (the tracking result 107 of the discriminative model in which the current frame is input as the frame 103 has not yet been obtained), the process proceeds to step S404 to continue the inference process.

[0055] Moreover, if the CPU 506 determines that an interruption instruction has been issued, the processing for the current frame according to the flowchart in FIG. 4 ends.

[0056] In step S406, CPU 506 stores (records) the first information in RAM 509. As a result, when the determination process of step S401 is performed for the frame period next to the current frame period (the frame period corresponding to the frame following the current frame) after the process of step S406, it will be determined that the first information indicating that the inference process has been completed in the previous frame period is stored in RAM 509.

[0057] In step S407, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The calculation unit 507 performs learning processing for a frame (current frame) corresponding to the current frame period in accordance with the operation command (operation instruction). Then, the calculation unit 507 notifies the CPU 506 of the completion of the processing.

[0058] In step S408, the calculation unit 507 determines whether or not the learning process for the current frame is completed. Also, in step S408, the CPU 506 determines whether or not an interruption instruction has been issued.

[0059] If the result of this determination is that the learning process for the current frame is complete, the process proceeds to step S409. If the learning process for the current frame is not yet complete, the process proceeds to step S407 to continue the learning process.

[0060] Moreover, if the CPU 506 determines that an interruption instruction has been issued, the processing for the current frame according to the flowchart in FIG. 4 ends.

[0061] In step S409, CPU 506 stores (records) the second information in RAM 509. As a result, when the determination process of step S401 is performed for the frame period next to the current frame period (the frame period corresponding to the frame following the current frame) after the process of step S409, it is determined that the second information indicating that the learning process has been completed in the previous frame period is stored in RAM 509.

[0062] The process according to the flowchart in Fig. 4 will be described taking as an example the case in Fig. 3. First, the process according to the flowchart in Fig. 4 is carried out for the first frame period.

[0063] Since the first frame period is assumed to be the initial frame period, the process proceeds from step S401 to step S404. In step S404, the calculation unit 507 performs inference process 301 for the frame corresponding to the first frame period, and when the inference process 301 is completed, the process proceeds to step S406 via step S405. In step S406, the CPU 506 stores the first information in the RAM 509. Then, in step S407, the calculation unit 507 performs learning process 302 for the frame corresponding to the first frame period, but since an interrupt instruction was issued during the learning process 302, the process according to the flowchart in FIG. 4 for the first frame period ends.

[0064] Next, processing is performed according to the flowchart in FIG. 4 for the second frame period. However, since the first information is stored in the RAM 509 but the second information is not stored therein, processing proceeds from step S401 to step S402.

[0065] In step S402, the calculation unit 507 performs the learning process 302 for a frame corresponding to the first frame period, and when the learning process 302 is completed, the process proceeds to step S404 via step S403. In step S404, the calculation unit 507 performs the inference process 303 for a frame corresponding to the second frame period, and when the inference process 303 is completed, the process proceeds to step S406 via step S405. In step S406, the CPU 506 stores the first information in the RAM 509. Thereafter, in step S407, the calculation unit 507 performs the learning process 304 for a frame corresponding to the second frame period, and when the learning process 304 is completed, the process proceeds to step S409 via step S408. In step S409, the CPU 506 stores the second information in the RAM 509.

[0066] Next, the process according to the flowchart in FIG. 4 is performed for the third frame period, but since the first information and the second information are stored in the RAM 509, the process proceeds from step S401 to step S404.

[0067] In step S404, the calculation unit 507 performs inference processing 305 for a frame corresponding to the third frame period, and when the inference processing 305 is completed, the process proceeds to step S406 via step S405. In step S406, the CPU 506 stores the first information in the RAM 509. Thereafter, in step S407, the calculation unit 507 performs learning processing 306 for a frame corresponding to the third frame period, and when the learning processing 306 is completed, the process proceeds to step S409 via step S408. In step S409, the CPU 506 stores the second information in the RAM 509.

[0068] In this way, according to this embodiment, even if the process is interrupted by an interrupt instruction in the previous frame period, the learning process that should have been performed in the previous frame period is completed before the subsequent inference process, so that the second coefficient 106 can be appropriately updated in the current frame period. This makes it possible to prevent tracking failures caused by the second coefficient 106 not being updated.

[0069] <Modification of the first embodiment> In the modified example of the first embodiment, the learning process for the previous frame and the inference process for the current frame are processed in parallel. By adopting such a configuration, in addition to the effect of the first embodiment, it is possible to prevent an increase in the processing time during the current frame period. Below, the difference from the first embodiment will be described, and unless otherwise specified below, it is assumed that the second embodiment is the same as the first embodiment.

[0070] A configuration example of the calculation unit 507 according to this modified example is shown in the block diagram of Fig. 6. When the control unit 601 receives an operation command (operation instruction) from the CPU 506, it controls the operation of each of the inference calculation unit 601 and the learning calculation unit 602 in response to the operation command (operation instruction).

[0071] The inference calculation unit 601 is a calculation unit that performs the above inference processing, and the learning calculation unit 602 is a calculation unit that performs the above learning processing, and each is a calculation unit that can operate independently. The inference calculation unit 601 and the learning calculation unit 602 perform processing using data stored in the memory unit 604, and store the results of the processing in the memory unit 604.

[0072] The functional units shown in Fig. 6 may be substituted with other functional units shown in Fig. 5. For example, the control unit 603 may be substituted with the CPU 506, the storage unit 604 may be substituted with the RAM 509, and the inference calculation unit 601 and the learning calculation unit 602 may be substituted with the CPU 506.

[0073] A time chart according to this modified example is shown in Fig. 7. Fig. 7 shows a case where the learning process 703 in the first frame period is interrupted by an interrupt instruction.

[0074] An inference process 701 performed in the first frame period is a process by the first convolutional layer 101 (a process for obtaining the intermediate output 105 from the frame 103) among the processes included in the above inference process 301. An inference process 702 performed in the first frame period is a process by the second convolutional layer 102 (a process for obtaining the tracking result 107 from the intermediate output 105) among the processes included in the above inference process 301.

[0075] An inference process 704 performed in the second frame period is a process by the first convolutional layer 101 among the processes included in the inference process 303. An inference process 705 performed in the second frame period is a process by the second convolutional layer 102 among the processes included in the inference process 303.

[0076] An inference process 707 performed in the third frame period is a process by the first convolutional layer 101 among the processes included in the inference process 305. An inference process 708 performed in the third frame period is a process by the second convolutional layer 102 among the processes included in the inference process 305.

[0077] When the inference process 701 and the inference process 702 are completed in the first frame period, the learning process 703 starts. However, an interrupt instruction is issued midway through the learning process 703, and as a result, the learning process 703 is interrupted.

[0078] In the second frame period, the interrupted learning process 703 is started first. Here, since the inference process 704 is a process using the first convolutional layer 101, the first coefficient 104 is used but the second coefficient 106 is not used. In other words, since the inference process 704 is not affected by the learning process 703, it may be executed in parallel with the learning process 703. In FIG. 7, in the second frame period, the interrupted learning process 703 is started and the inference process 704 is also started, so that the learning process 703 and the inference process 704 are executed in parallel. Then, since the inference process 705 is a process performed using the second convolutional layer 102 in which the second coefficient 106 updated by the learning process 703 is set, it is started after the learning process 703 is completed. In this modification, since the inference calculation unit 601 and the learning calculation unit 602 operate independently, the learning process 703 and the inference process 704 can be executed in parallel. 7, even if the learning process 703 is executed again in the second frame, the time required for all the processes in the second frame period remains the same as in the first embodiment. In other words, compared to the first embodiment, it is possible to shorten the processing time in the second frame period by the amount of the learning process 703.

[0079] Then, when the inference process 705 is completed, the learning process 706 corresponding to the second frame period is started. Then, in the third frame period, the inference process 707 and the inference process 708 are executed, and after the inference process 708 is completed, the learning process 709 is executed.

[0080] Next, the process performed by the information processing device to perform the above-mentioned inference process and learning process for the current frame period will be described with reference to the flowchart in Fig. 8. Note that the information processing device can perform the inference process and learning process for each frame period (each frame) by performing the process according to the flowchart in Fig. 8 for each frame period (each frame).

[0081] Hereinafter, in the inference processing, the processing performed using the first convolutional layer 101 is referred to as the first inference processing, and in the inference processing, the processing performed using the second convolutional layer 102 is referred to as the second inference processing.

[0082] In step S801, CPU 506 determines whether or not fourth information indicating that the second inference process has been completed in the frame period immediately preceding the current frame period (previous frame period) is stored in RAM 509. CPU 506 also determines whether or not second information indicating that the learning process has been completed in the previous frame period is stored in RAM 509.

[0083] If it is determined that the fourth information is stored in RAM 509 but the second information is not stored, CPU 506 determines that "the first inference process and the second inference process have been completed in the previous frame period, but the learning process has not been completed." Then, the process proceeds to step S802.

[0084] Also, if the result of this determination is that both the fourth information and the second information are stored in RAM 509, CPU 506 determines that "the first inference process, the second inference process, and the learning process have been completed in the previous frame period," and the process proceeds to step S803.

[0085] Furthermore, if it is determined that neither the fourth information nor the second information is stored in RAM 509, the process proceeds to step S803. If neither the fourth information nor the second information is stored in RAM 509, and if restarting the inference process that was to be performed in the previous frame period in the next frame period does not allow for the accumulation of frame processing discrepancies, the process that was to be performed in the previous frame period may be discarded.

[0086] It should be noted that if the current frame period is the first frame period, CPU 506 does not determine whether or not the fourth information and the second information are stored in RAM 509, and the process proceeds to step S803.

[0087] In step S802, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The learning calculation unit 602 in the calculation unit 507 performs learning processing on a frame corresponding to the previous frame period (previous frame) in accordance with the operation command (operation instruction).

[0088] In step S803, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The inference calculation unit 601 in the calculation unit 507 performs a first inference process for a frame (current frame) corresponding to the current frame period in accordance with the operation command (operation instruction). The calculation unit 507 then stores the intermediate output 105, which is the result of the first inference process, in the RAM 509, and notifies the CPU 506 of the completion of the process.

[0089] In step S804, the calculation unit 507 determines whether the learning process for the previous frame is completed, and whether the first inference process for the current frame is completed. In addition, in step S405, the CPU 506 determines whether an interrupt instruction has been issued.

[0090] If the result of this determination is that the learning process for the previous frame and the first inference process for the current frame are not yet complete, the process proceeds to step S802. If the result of this determination is that the learning process for the previous frame is complete but the first inference process for the current frame is not yet complete, the process proceeds to step S803. If the result of this determination is that both the learning process for the previous frame and the first inference process for the current frame are complete, the process proceeds to step S805. If the result of this determination is that CPU 506 determines that an interrupt instruction has been issued, the process for the current frame according to the flowchart in FIG. 8 ends.

[0091] In step S805, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The inference calculation unit 601 in the calculation unit 507 performs a second inference process for a frame (current frame) corresponding to the current frame period in accordance with the operation command (operation instruction). The calculation unit 507 then stores the tracking result 107, which is the result of the second inference process, in the RAM 509, and notifies the CPU 506 of the completion of the process.

[0092] In step S806, the calculation unit 507 determines whether or not the second inference process for the current frame is completed. Also, in step S806, the CPU 506 determines whether or not an interruption instruction has been issued.

[0093] If the result of this determination is that the second inference process for the current frame has not yet been completed, the process proceeds to step S805, where the second inference process continues. If the result of this determination is that the second inference process for the current frame has been completed, the process proceeds to step S807. If the result of this determination is that the CPU 506 has issued an interrupt instruction, the process for the current frame according to the flowchart in FIG. 8 ends.

[0094] In step S807, the CPU 506 stores the fourth information in the RAM 509. As a result, when the determination process of step S801 is performed for the frame period next to the current frame period (the frame period corresponding to the frame following the current frame) after the process of step S807, it will be determined that the fourth information indicating that the second inference process has been completed in the previous frame period is stored in the RAM 509.

[0095] In step S808, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The learning calculation unit 602 in the calculation unit 507 performs learning processing for a frame (current frame) corresponding to the current frame period in accordance with the operation command (operation instruction). Then, the calculation unit 507 notifies the CPU 506 of the completion of the processing.

[0096] In step S809, the calculation unit 507 determines whether or not the learning process for the current frame is completed. Also, in step S809, the CPU 506 determines whether or not an interruption instruction has been issued.

[0097] As a result of this determination, if the learning process for the current frame is completed, the process proceeds to step S810. If the learning process for the current frame is not yet completed, the process proceeds to step S808, where the learning process continues. If CPU 506 determines that an interrupt instruction has been issued, the process for the current frame according to the flowchart in FIG. 8 ends. In step S810, CPU 506 stores the second information in RAM 509.

[0098] The process according to the flowchart in Fig. 8 will be described by taking as an example the case in Fig. 7. First, the process according to the flowchart in Fig. 8 is carried out for the first frame period.

[0099] Since the first frame period is assumed to be the first frame period, the process proceeds from step S801 to step S803. In step S803, the calculation unit 507 performs inference processing 701 for the frame corresponding to the first frame period, and when the inference processing 701 is completed, the process proceeds to step S805 via step S804. In step S805, the calculation unit 507 performs inference processing 702 for the frame corresponding to the first frame period, and when the inference processing 702 is completed, the process proceeds to step S807 via step S806. In step S807, the CPU 506 stores the fourth information in the RAM 509. Then, in step S808, the calculation unit 507 performs learning processing 703 for the frame corresponding to the first frame period, but since an interrupt instruction is issued during the learning processing 703, the process according to the flowchart of FIG. 8 for the first frame period ends.

[0100] Next, processing is performed according to the flowchart in FIG. 8 for the second frame period. However, since the fourth information is stored in the RAM 509 but the second information is not stored therein, processing proceeds from step S801 to step S802.

[0101] In step S802, the calculation unit 507 performs a learning process 703 for a frame corresponding to the first frame period, and in step S803, the calculation unit 507 performs an inference process 704 for a frame corresponding to the second frame period. When the inference process 704 and the learning process 703 are completed, the process proceeds to step S805 via step S804, and in step S805, the calculation unit 507 performs an inference process 705 for a frame corresponding to the second frame period. When the inference process 705 is completed, the process proceeds from step S806 to step S807, and in step S807, the CPU 506 stores the fourth information in the RAM 509. Thereafter, in step S808, the calculation unit 507 performs a learning process 706 for a frame corresponding to the second frame period, and when the learning process 706 is completed, the process proceeds to step S810 via step S809. In step S810, the CPU 506 stores the second information in the RAM 509.

[0102] Next, the process according to the flowchart in FIG. 8 is performed for the third frame period, but since the fourth information and the second information are stored in the RAM 509, the process proceeds from step S801 to step S803.

[0103] In step S803, the calculation unit 507 performs inference processing 707 for a frame corresponding to the third frame period, and when the inference processing 707 is completed, the process proceeds to step S805 via step S804. In step S805, the calculation unit 507 performs inference processing 708 for a frame corresponding to the third frame period, and when the inference processing 708 is completed, the process proceeds to step S807 via step S806. In step S807, the CPU 506 stores the fourth information in the RAM 509. Thereafter, in step S808, the calculation unit 507 performs learning processing 709 for a frame corresponding to the third frame period, and when the learning processing 709 is completed, the process proceeds to step S810 via step S809. In step S810, the CPU 506 stores the second information in the RAM 509.

[0104] In this manner, according to this modified example, the first inference process that does not use the second coefficient 106 in the inference process for the current frame (is not affected by the learning process) and the learning process for the previous frame can be executed in parallel, thereby making it possible to shorten the processing time during the frame period.

[0105] [Second embodiment] The following describes the differences from the first embodiment, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. In this embodiment, an example of a method for restarting inference processing in the previous frame period when the inference processing is interrupted during the previous frame period is shown. By the method shown below, processing that requires the results of the learning processing in the previous frame period, i.e., the inference processing and learning processing in the previous frame period are completed before the inference processing of the current frame starts. process It is possible to complete the above.

[0106] A time chart according to this embodiment is shown in Fig. 9. Fig. 9 shows a case where an inference process 901 in the first frame period is interrupted by an interrupt instruction.

[0107] When the inference process 901 in the first frame period is interrupted, the learning process 902 that was supposed to be performed in the first frame period is not performed in the first frame period, and the interrupted inference process 901 is first executed in the second frame period. Then, when the inference process 901 is completed, the learning process 902 is executed. Then, after the learning process 902 is completed, the inference process 903 and the learning process 904 are performed in the second frame period, but a timeout occurs at the end timing of the second frame period, and the processing being executed is interrupted. In the example of FIG. 9, the inference process 903 is completed, but the learning process 904 is interrupted due to a timeout. Then, in the third frame period, the learning process 904 that was interrupted due to a timeout is first executed, and then the inference process 905 and the learning process 906 to be performed in the third frame period are executed.

[0108] This allows the second coefficient 106 to be updated by the learning process 902 before the inference process 903 for the second frame period is executed, and the second coefficient 106 to be updated by the learning process 904 before the inference process 905 for the third frame period is executed.

[0109] In other words, even if the interruption of inference process 901 during the first frame period is instructed by a user operation, the results of learning process 902 can be reflected in inference process 903, and the results of learning process 904 can be reflected in inference process 905.

[0110] Next, the process performed by the information processing device to perform the above-mentioned inference process and learning process for the current frame period will be described with reference to the flowchart in Fig. 10. Note that the information processing device can perform the inference process and learning process for each frame period (each frame) by performing the process according to the flowchart in Fig. 10 for each frame period (each frame).

[0111] In step S1001, CPU 506 determines whether or not first information indicating that inference processing has been completed in the frame period immediately preceding the current frame period (previous frame period) is stored in RAM 509. CPU 506 also determines whether or not second information indicating that learning processing has been completed in the previous frame period is stored in RAM 509.

[0112] If it is determined that the first information is stored in RAM 509 but the second information is not stored, CPU 506 determines that "inference processing has been completed in the previous frame period but learning processing has not been completed," and processing proceeds to step S1004.

[0113] Furthermore, if it is determined that both the first information and the second information are stored in RAM 509, CPU 506 determines that "the inference process and the learning process have been completed in the previous frame period", and the process proceeds to step S1006.

[0114] Furthermore, if it is determined that neither the first information nor the second information is stored in RAM 509, the process proceeds to step S1002. If neither the first information nor the second information is stored in RAM 509, and if restarting in the next frame period from the inference process that was to be performed in the previous frame period results in accumulation of frame processing discrepancies, the process that was to be performed in the previous frame period may be discarded.

[0115] It should be noted that if the current frame period is the first frame period, CPU 506 does not determine whether or not the first information and the second information are stored in RAM 509, and the process proceeds to step S1006.

[0116] In step S1002, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The calculation unit 507 performs inference processing on a frame (previous frame) corresponding to the previous frame period in accordance with the operation command (operation instruction). The calculation unit 507 then stores the tracking result 107 in the RAM 509 and notifies the CPU 506 of the completion of processing.

[0117] In step S1003, the calculation unit 507 determines whether or not the inference process for the previous frame is completed. Also, in step S1003, the CPU 506 determines whether or not an interruption instruction has been issued.

[0118] If the result of this determination is that the inference process for the previous frame is complete, the process proceeds to step S1004. If the inference process for the previous frame is not yet complete, the process proceeds to step S1002 to continue the inference process. If the CPU 506 detects the occurrence of an interrupt instruction, the process for the current frame according to the flowchart in FIG. 10 ends.

[0119] In step S1004, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The calculation unit 507 performs frame ( before The calculation unit 507 then performs the learning process for each frame. Then, the calculation unit 507 notifies the CPU 506 of the completion of the process.

[0120] In step S1005, the calculation unit 507 determines whether or not the learning process for the previous frame is completed. Also, in step S1005, the CPU 506 determines whether or not an interruption instruction has been issued.

[0121] If the result of this determination is that the learning process for the previous frame has been completed, the process proceeds to step S1006. If the learning process for the previous frame has not yet been completed, the process proceeds to step S1004 to continue the learning process.

[0122] Moreover, if the CPU 506 detects that an interruption instruction has been issued, the processing for the current frame according to the flowchart in FIG. 10 ends.

[0123] In step S1006, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The calculation unit 507 performs inference processing for a frame (current frame) corresponding to the current frame period in accordance with the operation command (operation instruction). The calculation unit 507 then stores the tracking result 107 in the RAM 509 and notifies the CPU 506 of the completion of processing.

[0124] In step S1007, the calculation unit 507 determines whether the inference process for the current frame is completed. Also, in step S1007, the CPU 506 determines whether an interrupt instruction has occurred (whether an interrupt instruction has been input in response to a user operation, or whether the CPU 506 or the like has issued an interrupt instruction due to the above-mentioned timeout).

[0125] If the result of this determination is that the inference process for the current frame is complete, the process proceeds to step S1008. If the inference process for the current frame is not yet complete, the process proceeds to step S1006 to continue the inference process. If an interrupt instruction has been issued, the process for the current frame according to the flowchart in FIG. 10 ends.

[0126] In step S1008, CPU 506 stores the first information in RAM 509. As a result, when the determination process of step S1001 is performed for the frame period next to the current frame period (the frame period corresponding to the frame following the current frame) after the process of step S1008, it will be determined that the first information indicating that the inference process has been completed in the previous frame period is stored in RAM 509.

[0127] In step S1009, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The calculation unit 507 performs learning processing for a frame (current frame) corresponding to the current frame period in accordance with the operation command (operation instruction). Then, the calculation unit 507 notifies the CPU 506 of the completion of the processing.

[0128] In step S1010, the calculation unit 507 determines whether the learning process for the current frame is completed. Also, in step S1010, the CPU 506 determines whether an interrupt instruction has occurred (whether an interrupt instruction has been input in response to a user operation, or whether the CPU 506 or the like has issued an interrupt instruction due to the above-mentioned timeout).

[0129] If the result of this determination is that the learning process for the current frame is complete, the process proceeds to step S1011. If the learning process for the current frame is not yet complete, the process proceeds to step S1009 to continue the learning process. If an interrupt instruction has been issued, the process for the current frame according to the flowchart in FIG. 10 ends.

[0130] In step S1011, CPU 506 stores the second information in RAM 509. As a result, when the determination process of step S1001 is performed for the frame period next to the current frame period (the frame period corresponding to the frame following the current frame) after the process of step S1011, it will be determined that the second information indicating that the learning process has been completed in the previous frame period is stored in RAM 509.

[0131] In this manner, in this embodiment, even if the process is interrupted by an interrupt instruction, the learning process that should be performed in the previous frame period is completed before the subsequent inference process, so that the second coefficient 106 can be appropriately updated in the current frame period. This makes it possible to prevent tracking failures caused by the second coefficient 106 not being updated.

[0132] <Modification of the second embodiment> In a modified example of the second embodiment, the learning process for the previous frame and the inference process for the current frame are processed in parallel. By adopting such a configuration, in addition to the effect of the second embodiment, it is possible to prevent an increase in processing time during the current frame period. Furthermore, when the first inference process is completed in the previous frame, it is possible to reduce the increase in processing time by restarting from the second inference process. Below, the differences from the second embodiment are explained, and unless otherwise noted below, it is assumed that the second embodiment is the same as the first embodiment.

[0133] A time chart according to this modified example is shown in Fig. 11. Fig. 11 shows a case where the inference process 1102 in the first frame period is interrupted by an interrupt instruction.

[0134] An inference process 1101 performed in the first frame period is a process by the first convolutional layer 101 among the processes included in the inference process 301. Also, an inference process 1102 performed in the first frame period is a process by the second convolutional layer 102 among the processes included in the inference process 301.

[0135] When inference process 1101 and inference process 1102 are completed in the first frame period, learning process 1103 is started. However, an interrupt instruction is issued midway through inference process 1102, so that inference process 1102 and learning process 1103 are interrupted.

[0136] In the second frame period, the interrupted inference process 1102 is executed, and after the inference process 1102 is completed, the learning process 1103 and the inference process 1104 (the process by the first convolution layer 101 among the processes included in the inference process 302) are executed in parallel. Since the inference process 1104 is a process using the first convolution layer 101, the first coefficient 104 is used but the second coefficient 106 is not used. In other words, since the inference process 1104 is not affected by the learning process 1103, it may be executed in parallel with the learning process 1103. Then, the inference process 1105 is started after the learning process 1103 is completed, since it is executed using the second convolution layer 102 in which the second coefficient 106 updated by the learning process 1103 is set. Then, when the inference process 1105 is completed, the learning process 1106 is executed. The third frame period is the same as that of FIG. 7.

[0137] Next, the process performed by the information processing device to perform the above-mentioned inference process and learning process for the current frame period will be described with reference to the flowchart in Fig. 12. Note that the information processing device can perform the inference process and learning process for each frame period (each frame) by performing the process according to the flowchart in Fig. 12 for each frame period (each frame).

[0138] In step S1201, CPU 506 determines whether or not third information indicating that the first inference process has been completed in the frame period immediately preceding the current frame period (previous frame period) is stored in RAM 509. CPU 506 also determines whether or not fourth information indicating that the second inference process has been completed in the previous frame period is stored in RAM 509. CPU 506 also determines whether or not second information indicating that the learning process has been completed in the previous frame period is stored in RAM 509.

[0139] If it is determined that the third information is stored in RAM 509, but the fourth information and the second information are not stored, CPU 506 determines that "the first inference process has been completed in the previous frame period, but the second inference process and the learning process have not been completed." Then, the process proceeds to step S1204.

[0140] Furthermore, if the result of this determination is that the third information and the fourth information are stored in RAM 509, but the second information is not stored, CPU 506 determines that "the first inference process and the second inference process have been completed in the previous frame period, but the learning process has not been completed." Then, the process proceeds to step S1206.

[0141] Also, if the result of this determination is that the third information, fourth information, and second information are stored in RAM 509, CPU 506 determines that "the first inference process, the second inference process, and the learning process have been completed in the previous frame period," and the process proceeds to step S1207.

[0142] Furthermore, if it is determined that none of the third information, fourth information, and second information is stored in RAM 509, the process proceeds to step S1202. If none of the third information, fourth information, and second information is stored in RAM 509, and if it is not acceptable for frame processing deviations to accumulate when inference processing that was to be performed in the previous frame period is resumed in the next frame period, the processing that was to be performed in the previous frame period may be discarded.

[0143] It should be noted that if the current frame period is the first frame period, CPU 506 does not determine whether or not the third information, the fourth information, and the second information are stored in RAM 509, and the process proceeds to step S1207.

[0144] In step S1202, CPU 506 issues an operation command (operation instruction) to calculation unit 507. Inference calculation unit 601 in calculation unit 507 performs a first inference process for a frame (previous frame) corresponding to the previous frame period in accordance with the operation command (operation instruction). Then, calculation unit 507 stores intermediate output 105, which is a result of the first inference process, in RAM 509, and notifies CPU 506 of the completion of the process.

[0145] In step S1203, the calculation unit 507 determines whether or not the first inference process for the previous frame is completed. Also, in step S1203, the CPU 506 determines whether or not an interruption instruction has been issued.

[0146] If the result of this determination is that the first inference process for the previous frame has not yet been completed, the process proceeds to step S1202, where the first inference process continues. If the result of this determination is that the first inference process for the previous frame has been completed, the process proceeds to step S1204. If the result of this determination is that the CPU 506 has issued an interrupt instruction, the process for the current frame according to the flowchart in FIG. 12 ends.

[0147] In step S1204, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The inference calculation unit 601 in the calculation unit 507 performs a second inference process for the previous frame in accordance with the operation command (operation instruction). The calculation unit 507 then stores the tracking result 107, which is the result of the second inference process, in the RAM 509, and notifies the CPU 506 of the completion of the process.

[0148] In step S1205, the calculation unit 507 determines whether or not the second inference process for the previous frame is completed. Also, in step S1205, the CPU 506 determines whether or not an interruption instruction has been issued.

[0149] If the result of this determination is that the second inference process for the previous frame has not yet been completed, the process proceeds to step S1204, where the second inference process continues. If the result of this determination is that the second inference process for the previous frame has been completed, the process proceeds to step S1206. If the result of this determination is that the CPU 506 has issued an interrupt instruction, the process for the current frame according to the flowchart in FIG. 12 ends.

[0150] In step S1206, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The learning calculation unit 602 in the calculation unit 507 performs learning processing on the previous frame in accordance with the operation command (operation instruction). Then, the calculation unit 507 notifies the CPU 506 of the completion of the processing.

[0151] In step S1207, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The inference calculation unit 601 in the calculation unit 507 performs a first inference process for the current frame in accordance with the operation command (operation instruction). The calculation unit 507 then stores the intermediate output 105, which is the result of the first inference process, in the RAM 509, and notifies the CPU 506 of the completion of the process.

[0152] In step S1208, the calculation unit 507 determines whether the learning process for the previous frame is completed, and whether the first inference process for the current frame is completed. Also, in step S405, the CPU 506 determines whether an interrupt instruction has been issued.

[0153] If it is determined that the learning process for the previous frame and the first inference process for the current frame are not yet complete, the process proceeds to step S1206. If it is determined that the learning process for the previous frame is complete but the first inference process for the current frame is not yet complete, the process proceeds to step S1207. If it is determined that both the learning process for the previous frame and the first inference process for the current frame are complete, the process proceeds to step S1209. If it is determined that an interrupt instruction has been issued by CPU 506, the process for the current frame according to the flowchart in FIG. 12 ends.

[0154] In step S1209, CPU 506 stores (records) the third information in RAM 509. As a result, when the determination process of step S1201 is performed for the frame period next to the current frame period (the frame period corresponding to the frame following the current frame) after the process of step S1209, it will be determined that the third information indicating that the first inference process has been completed in the previous frame period is stored in RAM 509.

[0155] In step S1210, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The inference calculation unit 601 in the calculation unit 507 performs a second inference process for the current frame in accordance with the operation command (operation instruction). The calculation unit 507 then stores the tracking result 107, which is the result of the second inference process, in the RAM 509, and notifies the CPU 506 of the completion of the process.

[0156] In step S1211, the calculation unit 507 determines whether or not the second inference process for the current frame is completed. Also, in step S1211, the CPU 506 determines whether or not an interruption instruction has been issued.

[0157] If the result of this determination is that the second inference process for the current frame has not yet been completed, the process proceeds to step S1210, where the second inference process continues. If the result of this determination is that the second inference process for the current frame has been completed, the process proceeds to step S1212. If the result of this determination is that the CPU 506 has issued an interrupt instruction, the process for the current frame according to the flowchart in FIG. 12 ends.

[0158] In step S1212, CPU 506 stores (records) the fourth information in RAM 509. As a result, when the determination process of step S1201 is performed for the frame period next to the current frame period (the frame period corresponding to the frame following the current frame) after the process of step S1212, it will be determined that the fourth information indicating that the second inference process has been completed in the previous frame period is stored in RAM 509.

[0159] In step S1213, the CPU 506 issues an operation command (operation instruction) to the calculation unit 507. The learning calculation unit 602 in the calculation unit 507 performs learning processing for the current frame in accordance with the operation command (operation instruction). Then, the calculation unit 507 notifies the CPU 506 of the completion of the processing.

[0160] In step S1214, the calculation unit 507 determines whether or not the learning process for the current frame is completed. Also, in step S1214, the CPU 506 determines whether or not an interruption instruction has been issued.

[0161] As a result of this determination, if the learning process for the current frame is completed, the process proceeds to step S1215. If the learning process for the current frame is not yet completed, the process proceeds to step S1213, where the learning process continues. If CPU 506 determines that an interrupt instruction has been issued, the process for the current frame according to the flowchart in FIG. 12 ends. In step S1215, CPU 506 stores the second information in RAM 509.

[0162] In this manner, according to this modified example, as in the second embodiment, even if processing is interrupted by an interrupt instruction, the learning processing that would have been performed in the previous frame period is completed before the subsequent second inference processing, so that the second coefficient 106 can be appropriately updated in the current frame period.

[0163] Furthermore, in this modification, like the modification of the first embodiment, the first inference process in the current frame and the learning process in the previous frame can be executed in parallel, making it possible to shorten the processing time during a frame period.

[0164] Furthermore, in this modification, taking Fig. 11 as an example, it is possible to reduce an increase in the processing time of the inference process in the second frame period. In this modification, the inference process 903 described in the second embodiment is divided into a first inference process 1104 and a second inference process 1105, and in the second frame period, processing is resumed from the second inference process 1105, thereby reducing the processing time for the first inference process 1104.

[0165] [Third embodiment] In the above embodiment and modified example, the second coefficient 106, which is a parameter necessary for the second inference process, is updated by the learning process. However, the first coefficient 104, which is a parameter necessary for the first inference process, may be updated by the learning process. In that case, the learning process for the previous frame needs to be completed before the first inference process for the current frame. In the case of FIG. 11, in the second frame period, the inference process 1102 and the learning process 1103 are executed in parallel, and after the execution is completed, the inference process 1104 and the learning process of the first coefficient 104 in the learning process 1106 are executed in parallel. Then, the inference process 1105 is executed next, and when the execution is completed, the learning process of the second coefficient 106 is executed in the learning process 1106. In other words, the learning process can be started immediately after the first inference process, and the second inference process can be executed without waiting for the completion of the learning process. Therefore, for example, the second inference process and the learning process for the same frame may be processed in parallel.

[0166] In addition, in the inference, post-processing may be performed to generate a tracking result to be finally output, separate from the inference result. On the other hand, if the post-processing result is not used in learning, the post-processing may be performed only in the inference for the current frame, that is, the post-processing in the inference for the previous frame may be omitted. This makes it possible to reduce the processing time for post-processing when performing inference processing for the previous frame compared to when performing inference processing for the current frame.

[0167] In the modified example of the second embodiment, the inference process is divided into a first inference process and a second inference process, and in the case of FIG. 11, the first estimation process for the first frame is performed in the first frame period, and the second estimation process is started in the second frame period. However, the learning process may also be divided into a first learning process and a second learning process (for example, the first half of the learning process is the first learning process, and the second half is the second learning process). For example, if the first learning process is completed in the first frame period but the second learning process is not completed, the second learning process may be started in the second frame period, and after the second learning process is completed, the process may be performed in the same manner as in the third frame period of FIG. 7 or FIG. 11. This makes it possible to reduce the processing time for the first learning process, as in the modified example of the second embodiment.

[0168] In addition, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0169] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0170] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0171] The invention of this specification includes the following information processing device, information processing method, and computer program.

[0172] (Item 1) A processing means for executing inference related to each frame and learning related to each frame during a frame period corresponding to each frame, When inference related to a first frame is completed during a first frame period corresponding to the first frame but learning related to the first frame is not completed, the processing means executes learning related to the first frame and inference and learning related to the second frame during a second frame period corresponding to a second frame following the first frame. 23. An information processing apparatus comprising:

[0173] (Item 2) The processing means includes: When the inference for the frame is completed, record information indicating that the inference is completed; When learning related to the frame is completed, information indicating that the learning is completed is recorded. 2. The information processing device according to item 1,

[0174] (Item 3) 3. The information processing device according to item 2, wherein when the processing means detects an interrupt instruction during execution of an inference, the processing means does not record information indicating that the inference has been completed.

[0175] (Item 4) 4. The information processing device according to item 2 or 3, wherein when the processing means detects an interrupt instruction during execution of learning, the processing means does not record information indicating that the learning has been completed.

[0176] (Item 5) The processing means includes: When the inference related to the first frame is completed but the learning related to the first frame is not completed, in the second frame period, the learning related to the first frame is executed and then the inference and the learning related to the second frame are executed. 5. The information processing device according to any one of items 1 to 4.

[0177] (Item 6) The processing means includes: When the inference related to the first frame is completed but the learning related to the first frame is not completed, a first inference not affected by learning in the inference related to the second frame and the learning related to the first frame are executed in parallel during the second frame period, and after the execution is completed, a second inference affected by learning in the inference related to the second frame and the learning related to the second frame are executed. 5. The information processing device according to any one of items 1 to 4.

[0178] (Item 7) The processing means includes: When neither the inference nor the learning related to the first frame is completed, in the second frame period, the inference and the learning related to the first frame are performed, and then the inference and the learning related to the second frame are performed. The processing that is not completed by the end timing of the second frame period is performed in a frame period corresponding to a subsequent frame following the second frame, and then the inference and the learning related to the subsequent frame are performed. 5. The information processing device according to any one of items 1 to 4.

[0179] (Item 8) The processing means includes: When a first inference not affected by learning in an inference related to the first frame is completed, but a second inference affected by learning in an inference related to the first frame and the learning related to the first frame are not completed, in the second frame period, after the second inference is executed, the first inference not affected by learning in an inference related to the second frame and the learning related to the first frame are executed in parallel, and after the execution is completed, the second inference affected by learning in an inference related to the second frame is executed, and after the execution is completed, the learning related to the second frame is performed. 5. The information processing device according to any one of items 1 to 4.

[0180] (Item 9) The processing means includes: When a first inference not affected by learning in an inference related to the first frame is completed, but a second inference affected by learning in an inference related to the first frame and the learning related to the first frame are not completed, the second inference and the learning related to the first frame are executed in parallel during the second frame period, and after the execution is completed, a first inference affected by learning in an inference related to the second frame is executed, after the execution is completed, a first learning related to the second frame is executed, after the execution is completed, a second inference affected by learning in an inference related to the second frame is executed, and after the execution is completed, a second learning related to the second frame is executed. 5. The information processing device according to any one of items 1 to 4.

[0181] (Item 10) The information processing device according to any one of items 1 to 4, wherein, when a first learning related to a first frame is completed in a first frame period corresponding to the first frame but a second learning related to the first frame is not completed in the first frame period, the processing means first starts the second learning in a second frame period corresponding to the second frame.

[0182] (Item 11) The processing means includes: If neither the inference nor the learning related to the first frame has been completed, the inference and the learning related to the first frame are not performed. 11. The information processing device according to any one of items 1 to 10.

[0183] (Item 12) The processing means includes: The information processing device described in any one of items 1 to 11, characterized in that in the inference, post-processing is performed to generate a tracking result, and if the result of the post-processing is not used for learning, the post-processing is performed only in the inference related to the current frame.

[0184] (Item 13) The inference is a tracking process of identifying a target in a frame and tracking the target using an identification model that identifies the target in the frame; The learning is based on the result of the inference. To be carried out A learning process for the discrimination model. 13. The information processing device according to any one of items 1 to 12.

[0185] (Item 14) An information processing method performed by an information processing device, The processing means of the information processing device includes a processing step of executing, during a frame period corresponding to each frame, inference related to the frame and learning related to the frame, In the processing step, when inference related to the first frame is completed during a first frame period corresponding to the first frame but learning related to the first frame is not completed, learning related to the first frame and inference and learning related to the second frame are executed during a second frame period corresponding to a second frame following the first frame. 23. An information processing method comprising:

[0186] (Item 15) A computer program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 13.

[0187] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0188] 501: Input section 502: Data storage section 503: Communication section 504: Display section 505: Image processing section 506: CPU 507: Calculation section 508: ROM 509: RAM 510: System bus

Claims

1. A processing means for executing inference related to each frame and learning related to each frame during a frame period corresponding to each frame, When inference related to a first frame is completed during a first frame period corresponding to the first frame but learning related to the first frame is not completed, the processing means executes learning related to the first frame and inference and learning related to the second frame during a second frame period corresponding to a second frame following the first frame.

23. An information processing apparatus comprising:

2. The processing means includes: When the inference for the frame is completed, record information indicating that the inference is completed; When learning related to the frame is completed, information indicating that the learning is completed is recorded.

2. The information processing apparatus according to claim 1,

3. 3. The information processing apparatus according to claim 2, wherein said processing means, when detecting an interrupt instruction during execution of an inference, does not record information indicating that the inference has been completed.

4. 3. The information processing apparatus according to claim 2, wherein said processing means, when detecting an instruction to interrupt learning during execution of the learning, does not record information indicating that the learning has been completed.

5. The processing means includes: When the inference related to the first frame is completed but the learning related to the first frame is not completed, in the second frame period, the learning related to the first frame is executed and then the inference and the learning related to the second frame are executed.

2. The information processing apparatus according to claim 1,

6. The processing means includes: When the inference related to the first frame is completed but the learning related to the first frame is not completed, a first inference not affected by the learning related to the first frame in the inference related to the second frame and the learning related to the first frame are executed in parallel during the second frame period, and after the execution is completed, a second inference affected by the learning in the inference related to the second frame and the learning related to the second frame are executed.

2. The information processing apparatus according to claim 1,

7. The processing means includes: When neither the inference nor the learning related to the first frame is completed, in the second frame period, the inference and the learning related to the first frame are performed, and then the inference and the learning related to the second frame are performed. The processing that is not completed by the end timing of the second frame period is performed in a frame period corresponding to a subsequent frame following the second frame, and then the inference and the learning related to the subsequent frame are performed.

2. The information processing apparatus according to claim 1,

8. The processing means includes: In the case where a first inference not affected by learning related to a frame preceding the first frame is completed in an inference related to the first frame, but a second inference affected by learning in an inference related to the first frame and learning related to the first frame are not completed, in the second frame period, after the second inference is executed, a first inference not affected by learning related to the first frame in an inference related to the second frame and learning related to the first frame are executed in parallel, and after the execution is completed, a second inference affected by learning in an inference related to the second frame is executed, and after the execution is completed, learning related to the second frame is performed.

2. The information processing apparatus according to claim 1,

9. The processing means includes: When a first inference that is affected by learning in an inference related to the first frame has been completed but a second inference that is affected by learning in an inference related to the first frame and the learning related to the first frame have not been completed, the second inference and the learning related to the first frame are executed in parallel during the second frame period, and after the execution is completed, a first inference that is affected by learning in an inference related to the second frame is executed, after the execution is completed, a first learning related to the second frame is executed, after the execution is completed, a second inference that is affected by learning in an inference related to the second frame is executed, and after the execution is completed, a second learning related to the second frame is executed, a first inference that is influenced by learning in an inference related to the first frame is influenced by a first learning related to a frame preceding the first frame; a second inference that is influenced by learning in an inference related to the first frame is influenced by a second learning related to a frame preceding the first frame; a first inference that is influenced by learning in an inference related to the second frame is influenced by a first learning related to the first frame; A second inference that is influenced by learning in an inference related to the second frame is influenced by a second learning related to the first frame.

2. The information processing apparatus according to claim 1,

10. 2. The information processing device according to claim 1, characterized in that, when a first learning for a first frame is completed in a first frame period corresponding to a first frame, but a second learning for the first frame is not completed in the first frame period, the processing means first starts the second learning in a second frame period corresponding to the second frame.

11. The processing means includes: If neither the inference nor the learning related to the first frame has been completed, the inference and the learning related to the first frame are not performed.

2. The information processing apparatus according to claim 1,

12. The processing means includes:

2. The information processing apparatus according to claim 1, wherein in the inference, post-processing is performed to generate a tracking result, and if the result of the post-processing is not used for learning, the post-processing is performed only in the inference related to the current frame.

13. The inference is a tracking process of identifying a target in a frame and tracking the target using an identification model that identifies the target in the frame; The learning is a learning process of the discriminative model that is performed based on the result of the inference.

2. The information processing apparatus according to claim 1,

14. A method for detecting a moving object to be tracked, comprising: a processing means for performing, during a frame period corresponding to each frame in which a moving object is included as an object, inference regarding the object in the frame and learning regarding the object in the frame; When inference related to the subject in a first frame period corresponding to a first frame is completed but learning related to the subject in the first frame is not completed, the processing means executes, in a second frame period corresponding to a second frame following the first frame, learning related to the subject in the first frame, inference related to the subject in the second frame using a result of the learning related to the subject in the first frame, and learning related to the subject in the second frame using a result of the inference related to the subject in the second frame.

23. An information processing apparatus comprising:

15. An information processing method performed by an information processing device, The processing means of the information processing device includes a processing step of executing, during a frame period corresponding to each frame, inference related to the frame and learning related to the frame, In the processing step, when inference related to a first frame is completed during a first frame period corresponding to the first frame but learning related to the first frame is not completed, learning related to the first frame and inference and learning related to the second frame are executed during a second frame period corresponding to a second frame following the first frame.

23. An information processing method comprising:

16. A computer program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 13.

Images