Inference processing device, inference processing method, and inference processing program

Abstract

This inference processing device performs inference processing for outputting positional information of an object by using a moving image as an input, and comprises: a division processing unit for performing object detection by dividing each frame image of the moving image; a counting unit for counting the number of new objects and the number of discarded objects within respective partial surfaces, which are partial images obtained by said dividing, for each of the partial surfaces; an allocation control unit for determining, on the basis of the value counted by the counting unit, the partial surface on which inference processing is to be performed, for each frame image; an overall processing unit for performing object detection by reducing the overall frame image in size; and a synthesis processing unit for obtaining a final object detection result by synthesizing an object detection result from the division processing unit and an object detection result from the overall processing unit.

Description

Inference processing device, inference processing method, and inference processing program

[0001] The disclosed technology relates to an inference processing device, an inference processing method, and an inference processing program.

[0002] Edge AI systems that perform convolutional neural network (CNN) inference processing such as object detection and pose estimation on edge devices such as surveillance cameras and drones have been attracting attention. However, these CNN inferences are limited in the input image size that can be processed at one time due to limitations in the model itself and memory capacity constraints on the edge device.

[0003] Object detection is a device that detects metadata from an input image, including the position of an object (a rectangular frame surrounding the object), attributes (type of object, such as person or car), and reliability. Pose estimation is a device that detects skeletal information of a person contained in the input image (information on feature points such as the eyes, nose, and joints, and the edges connecting them).

[0004] To enable CNN inference processing such as object detection and pose estimation for high-resolution images, Patent Literature 1 and Non-Patent Literature 1 disclose methods for performing inference processing on an entire image obtained by reducing an input image to the above-mentioned constrained image size and on divided surfaces, and then synthesizing the results (see FIG. 1). For example, if the input size is 608 x 608, for ultra-high-definition video such as 4K (3840 x 2160), the number of image divisions in the division process would be approximately 28, resulting in a huge amount of calculation. Another method for reducing the amount of calculation has been proposed: narrowing down (thinning out) the partial surfaces on which object detection is performed for the same video frame. When thinning out, in order to prioritize inference processing for partial surfaces with large scene changes, the absolute difference between the average number of object detections over a certain period of time in the past and the number of object detections in the current frame is calculated, and partial surfaces with the largest value are prioritized for processing.

[0005] PCT / JP2020 / 005812

[0006] Hiroyuki Uzawa, Shuhei Yoshida, Yuukou Iinuma, Saki Hatta, Daisuke Kobayashi, Yuya Omori, Ken Nakamura, Shuichi Takada, Hassan Toorabally, Kimikazu Sano, High-Definition Object Detection Technology Based on AI Inference Scheme and its Implementation, IEICE Electronics Express Pose Estimation Method for High-Resolution Images Using AI Inference, Shuhei Yoshida, Hiroyuki Uzawa, Yuukou Iinuma, Saki Hatta, Yuya Omori, Yusuke Horishita, Daisuke Kobayashi, Ken Nakamura, and Koji Yamazaki (NTT)

[0007] However, with the above method, the absolute value of the difference tends to be small when the number of new objects that have newly appeared on the partial surface and the number of discarded objects that are no longer reflected on the partial surface are close, which causes the problem that there are cases where the scene does not become the target of inference processing even though it actually has changed significantly.

[0008] The disclosed technology has been made in consideration of the above points, and aims to provide an inference processing device, an inference processing method, and an inference processing program that can reduce the amount of calculation and perform object detection or pose estimation with high accuracy.

[0009] A first aspect of the present disclosure is an inference processing device that performs inference processing using a moving image as input and outputs object position information, and includes a segmentation processing unit that segments each frame image of the moving image to perform object detection, a counting unit that counts the number of new objects and the number of discarded objects within a partial plane, which is a partial image obtained by segmentation, for each partial plane, an allocation control unit that determines the partial plane on which inference processing is performed based on the value counted by the counting unit for each frame image, an overall processing unit that reduces the entire frame image to perform object detection, and a synthesis processing unit that obtains a final object detection result by synthesizing the object detection result of the segmentation processing unit and the object detection result of the overall processing unit.

[0010] A second aspect of the present disclosure is an inference processing device that performs inference processing using a moving image as input and outputs object posture information, the inference processing device including: a segmentation processing unit that divides each frame image of the moving image to perform posture estimation; a frame generation unit that generates a detection frame within a partial plane that is a partial image obtained by the segmentation; a counting unit that counts the number of new objects and the number of discarded objects within the partial plane for each partial plane; an allocation control unit that determines the partial plane for performing inference processing based on the value counted by the counting unit for each frame image; an overall processing unit that reduces the entire frame image to perform posture estimation; and a synthesis processing unit that obtains a final posture estimation result by synthesizing the posture estimation result of the segmentation processing unit and the posture estimation result of the overall processing unit.

[0011] A third aspect of the present disclosure is an inference processing method in an inference processing device that performs inference processing using a moving image as input and outputs object position information, wherein a segmentation processing unit segments each frame image of the moving image to perform object detection, a counting unit counts the number of new objects and the number of discarded objects within a partial plane, which is a partial image obtained by segmentation, for each partial plane, an allocation control unit determines the partial plane on which inference processing is performed based on the value counted by the counting unit for each frame image, an overall processing unit reduces the entire frame image to perform object detection, and a synthesis processing unit obtains a final object detection result by synthesizing the object detection result of the segmentation processing unit and the object detection result of the overall processing unit.

[0012] A fourth aspect of the present disclosure is an inference processing method for an inference processing device that performs inference processing that takes a moving image as input and outputs object posture information, wherein a segmentation processing unit segments each frame image of the moving image to perform posture estimation, a frame generation unit generates a detection frame within a partial plane that is a partial image obtained by segmentation, a counting unit counts the number of new objects and the number of discarded objects within the partial plane for each partial plane, an allocation control unit determines the partial plane on which to perform inference processing for each frame image based on the value counted by the counting unit, an overall processing unit reduces the entire frame image to perform posture estimation, and a synthesis processing unit obtains a final posture estimation result by synthesizing the posture estimation result of the segmentation processing unit and the posture estimation result of the overall processing unit.

[0013] A fifth aspect of the present disclosure is an inference processing program for causing a computer to function as the inference processing device of the first or second aspect.

[0014] According to the disclosed technology, it is possible to reduce the amount of calculation and perform object detection or pose estimation with high accuracy.

[0015] FIG. 1 is a schematic diagram for explaining an overview of the first embodiment. FIG. 1 is a schematic diagram for explaining an overview of the first embodiment. FIG. 2 is a block diagram showing an input / output of an inference processing device of the first embodiment. FIG. 3 is a block diagram showing an example of a hardware configuration of an inference processing device of the first and second embodiments. FIG. 4 is a block diagram showing the configuration of the inference processing device of the first and second embodiments. FIG. 5 is a diagram for explaining the processing of a counting unit. FIG. 6 is a block diagram showing the configuration of a division processing unit of the inference processing device of the first embodiment. FIG. 7 is a block diagram showing the configuration of an overall processing unit of the inference processing device of the first embodiment. FIG. 8 is a flowchart showing the flow of inference processing of the inference processing device of the first embodiment. FIG. 9 is a flowchart showing the flow of processing of the counting unit and allocation control unit of the inference processing device of the first embodiment. FIG. 10 is a diagram for explaining the processing of the allocation control unit according to the first embodiment. FIG. 11 is a block diagram showing the input / output of an inference processing device of the second embodiment. FIG. 12 is a block diagram showing the configuration of a division processing unit of the inference processing device of the second embodiment. FIG. 13 is a block diagram showing the configuration of an overall processing unit of the inference processing device of the second embodiment.

[0016] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0017] [Summary of the Present Disclosure] In the technology of the present disclosure, in order to enable CNN inference processing such as object detection and pose estimation for high-resolution images, inference processing is performed on the entire surface obtained by reducing the input image to the above-mentioned constrained image size and on the divided surfaces obtained by dividing the input image, and the results are combined, as shown in Fig. 1. Here, the number of objects newly reflected in a certain partial surface (new objects) and the number of objects no longer reflected in the surface (discarded objects) are calculated, and the partial surface with the largest sum is given priority as the target of inference processing, thereby solving the problems of the conventional technology and suppressing accuracy degradation due to thinning (Fig. 2).

[0018] 2 shows an example in which the number of detected objects is 2, the number of new objects is 0, and the number of discarded objects is 0 in a certain partial surface of frame image #0. Also, an example in which the number of detected objects is 2, the number of new objects is 0, and the number of discarded objects is 0 in a certain partial surface of frame image #1. Also, an example in which the scene changes and the number of detected objects is 2, the number of new objects is 1, and the number of discarded objects is 1 in a certain partial surface of frame image #1. The object that has disappeared is the discarded object, and the newly appeared object is the new object. Looking at the number of detected objects alone, there appears to be no change, but by looking at the number of new objects and the number of discarded objects, the change in scene can be grasped.

[0019] The inference processing device of the present disclosure will be described in detail below.

[0020] [First embodiment] In the first embodiment, as shown in Fig. 3, an inference processing device 10 receives a moving image as input, performs object detection as inference processing, and outputs metadata representing the object detection result. Fig. 4 is a block diagram showing the hardware configuration of the inference processing device 10 of this embodiment.

[0021] 4, the inference processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0022] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, an inference processing program is stored in the ROM 12 or the storage 14. The inference processing program may be a single program or a group of programs composed of multiple programs or modules.

[0023] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0024] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0025] The display unit 16 is, for example, a liquid crystal display, and displays various information including the processing results. The display unit 16 may be a touch panel type and function as the input unit 15.

[0026] The communication interface 17 is an interface for communicating with other devices, such as a camera that captures input images and a database server that stores the input images captured by the camera, etc. Standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used for this communication.

[0027] Next, a description will be given of the functional configuration of the inference processing device 10. Fig. 5 is a block diagram showing an example of the functional configuration of the inference processing device 10.

[0028] As shown in FIG. 5, the inference processing device 10 functionally comprises a motion estimator 100, a division processor 110, a counter 120, an allocation controller 122, an overall processor 130, and a synthesis processor 140.

[0029] The motion estimation unit 100 calculates a motion vector for each frame image of the input moving image over the entire image.

[0030] The division processing unit 110 divides each frame image of the moving image and performs object detection.

[0031] The counting unit 120 counts the number of new objects and discarded objects in each partial plane, which is a partial image obtained by dividing.

[0032] Specifically, the degree of overlap between the detection frame of each object obtained from the estimation results by the coordinate correction unit 117 described later and the detection frame of each object included in the results of performing inference processing on the current frame image is calculated, and the number of new objects and discarded objects is counted.

[0033] FIG. 6 illustrates a specific method for counting new objects and discarded objects. This method determines new objects and discarded objects using the results of an actual object detection process performed on a certain partial surface of the current frame and a predicted detection result calculated by calculating a motion vector and sliding the coordinates of the detection result on a certain partial surface of a past frame based on the motion vector. For example, the overlapping area between the detection frames included in the actual detection result and the detection frames included in the predicted detection result is calculated, and a paired combination that maximizes the degree of overlap between all detection frames is determined. This type of problem is generally referred to as an assignment problem, and various solutions have been proposed. For example, the Hungarian method is known. The technology disclosed herein does not limit the solution method. Among the paired combinations, among the detection frames that do not have a partner, those included in the predicted result are determined to be discarded objects, and those not included are determined to be new objects. The number of new objects and discarded objects can then be determined by counting these.

[0034] The allocation control unit 122 determines the partial surfaces on which inference processing is to be performed for each frame image based on the value counted by the counting unit 120 .

[0035] Specifically, the allocation control unit 122 has a predetermined allocation period defined by the number of frames, and calculates, for each partial surface, the average value of the total or weighted sum of the number of new objects and discarded objects in the current allocation period, and determines, for each partial surface, the number of times to perform inference processing for the partial surface in the next allocation period based on the calculated average value.

[0036] More specifically, the average value of the sum of the number of new objects and discarded objects for each partial surface is calculated at regular intervals, and the partial surface for which inference processing is performed is determined so that the partial surface with a larger value has a higher number of inference executions. The formula for calculating the average value is shown below.

[0037]

[0038] Here, C ｉ is the sum of the number of new objects and discarded objects at the i-th inference execution time, C ｎｅｗ is the number of new objects, C ｌｏｓｔ represents the number of discarded objects. ａｖｇ is the average value of the total number of new objects and discarded objects, and n is the number of times inference is performed.

[0039] In this case, the comparison may be made using the average value of the weighted sum of the number of new objects and the number of discarded objects as shown in the following formula: For example, if emphasis is placed on detecting new objects, the weight on the number of new objects is increased.

[0040]

[0041] Here, α is a weighting factor for the number of new objects, and β is a weighting factor for the number of discarded objects. The other variables are the same as those described above.

[0042] The overall processing unit 130 reduces the entire image and performs object detection.

[0043] The synthesis processing unit 140 synthesizes the object detection result of the division processing unit 110 and the object detection result of the overall processing unit 130 to obtain a final object detection result.

[0044] As shown in FIG. 7, the division processing unit 110 includes a division unit 111 , a first selection unit 112 , an object detection unit 113 , a holding unit 114 , a second selection unit 116 , and a coordinate correction unit 117 .

[0045] The dividing unit 111 divides each frame image of the moving image and outputs it as partial planes, which are a plurality of partial images.

[0046] The first selection unit 112 selects a partial surface on which inference processing is to be performed for each frame image in accordance with the determination by the allocation control unit 122 .

[0047] The object detection unit 113 uses an object detection model, which is a convolutional neural network that takes an image as input and outputs object position information, to perform object detection to determine object position information for each of the selected partial surfaces, and outputs a detection frame representing the object.

[0048] Specifically, the object detection model is a trained model that has been trained in advance to output a detection frame surrounding each object included in an input image.

[0049] The object detection model may be stored in a storage medium such as the storage 14, or may be stored in an external cloud server. As a technique for performing object detection based on such images, known methods can be appropriately applied.

[0050] The storage unit 114 stores the object detection results for each partial surface by the object detection unit 113 for each frame image.

[0051] The second selection unit 116 selects, for each frame image, partial surfaces on which inference processing is not performed, in accordance with the determination by the allocation control unit 122 .

[0052] The coordinate correction unit 117 estimates the position information of the object in the current frame image from the position information of the object obtained from the result of the inference process on the previous frame image based on the motion vector.

[0053] For each partial plane for which inference processing is not performed, the position of the detection frame detected in that partial plane of the previous frame image is corrected based on the motion vector.

[0054] As shown in FIG. 8 , the overall processing unit 130 includes a reduction unit 131 and an object detection unit 132 .

[0055] The reduction unit 131 reduces the entire image for each frame image of the moving image.

[0056] The object detection unit 132 detects an object from the reduced image for each frame image of the video.

[0057] <Operation of the inference processing device according to this embodiment> Next, the operation of the inference processing device 10 according to this embodiment will be described.

[0058] 9 is a flowchart showing the flow of inference processing by the inference processing device 10. The inference processing is performed by the CPU 11 reading out an inference processing program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. It is also assumed that a moving image has been input to the inference processing device 10. The inference processing is an example of an inference processing method.

[0059] In step S100, the CPU 11 acquires frame images of the input moving image.

[0060] In step S102, the CPU 11 functions as the motion estimation unit 100 to calculate a motion vector for the entire image for the acquired frame image.

[0061] In step S104, the CPU 11 functions as the division processing unit 110 to divide the acquired frame image and perform object detection.

[0062] In step S108, the CPU 11 functions as the overall processing unit 130 to reduce the entire acquired frame image and perform object detection.

[0063] In step S110, the CPU 11 functions as the synthesis processing unit 140 to synthesize the object detection result of the division processing unit 110 and the object detection result of the overall processing unit 130 to obtain a final object detection result.

[0064] The step S104 is realized by a division processing routine shown in FIG.

[0065] In step S120, the CPU 11 functions as the division unit 111 to divide each frame image of the moving image and output it as partial planes, which are a plurality of partial images.

[0066] In step S122, the CPU 11 functions as the first selection unit 112 to select a partial surface on which inference processing is to be performed for each frame image in accordance with the determination made by the allocation control unit 122.

[0067] In step S124, the CPU 11 uses an object detection model, which is a convolutional neural network that receives an image as input and outputs object position information, as the object detection unit 113 to perform object detection to determine object position information for each of the selected partial surfaces, and outputs a detection frame representing the object.

[0068] In step S126, the CPU 11 causes the storage unit 114 to store the object detection results for each partial surface by the object detection unit 113 for each frame image.

[0069] In step S128, the CPU 11, as the second selection unit 116, selects a partial surface for which inference processing will not be performed for each frame image in accordance with the decision made by the allocation control unit 122, and obtains the result of the inference processing for the previous frame image for that partial surface.

[0070] In step S130, the CPU 11 functions as the coordinate correction unit 117 to estimate the position information of the object in the current frame image from the position information of the object obtained from the result of the inference process on the previous frame image based on the motion vector.

[0071] Each time the inference process shown in FIG. 9 is performed on each frame image, the processing routine shown in FIG. 11 is executed.

[0072] In step S140, the CPU 11, as the counting unit 120, performs inference processing on the partial surface to which the inference processing is to be performed that is assigned in the current cycle, calculates a motion vector, and predicts the position of the detection frame in the current frame based on the obtained motion vector and the position information of the detection frame of the same partial surface in the previous frame.

[0073] In step S142, the CPU 11, as the counting unit 120, compares the prediction result with the detection frame information obtained by actually executing the inference process, counts the number of new objects and discarded objects, and calculates the total or weighted sum of these.

[0074] In step S144, the CPU 11 determines whether or not the processes of steps S140 and S142 have been executed for all partial surfaces. If there are partial surfaces for which the processes of steps S140 and S142 have not been executed, the CPU 11 returns to step S140 and executes the processes of steps S140 and S142 for those partial surfaces.

[0075] In step S146, the CPU 11 determines whether the number of elapsed frames is less than the number of frames in a period. Here, the number of frames in a period is the number of frames processed per period, and is assumed to be externally provided. If the number of frames in a period is greater than the number of elapsed frames, the process proceeds to step S148, where the number of elapsed frames is incremented. On the other hand, if the number of elapsed frames has reached the number of frames in a period, the process proceeds to step S150.

[0076] In step S150, the CPU 11 functions as the allocation control unit 122 to change the allocation of partial surfaces to be subjected to inference processing in the next cycle.

[0077] Specifically, if multiple inference process executions are assigned to the same sub-surface within a period, the average value of the sum or weighted sum is calculated. In this case, the allocation may be performed so that each sub-surface is assigned at least one inference process execution. If this is not the case, for sub-surfaces to which no inference process executions have been assigned, a predetermined value may be used instead of calculating the average value.

[0078] As described above, the number of new objects and discarded objects in each partial surface is counted at regular intervals, and the setting of the inference processing allocation surface for the next period is changed so that inference processing is prioritized for surfaces with a larger average total or weighted sum of the counted objects. The allocation amount for each partial surface is determined by proportionally allocating the allocation amount to each partial surface. The allocation amount is a numerical value given by T × R, where T is the number of frames for which object detection can be performed per frame, and R is the number of frames in the period. Here, T can be given externally or dynamically determined internally based on some algorithm, but the technique disclosed herein does not limit the determination method. Furthermore, the initial allocation setting for the first period is given externally.

[0079] In the above allocation in this embodiment, the allocation amount is determined by proportional distribution based on the average value of the sum or weighted sum of the number of new objects and discarded objects, but this is not limited to this. A minimum amount of allocation (e.g., 1) may be set, and all partial surfaces may be executed at least the minimum number of times within a cycle, and the remaining number of object detection executions that can be executed within that cycle may be allocated as the proportional distribution. However, R is set to a value at least equal to or greater than the minimum amount x T.

[0080] In step S152, the CPU 11 resets the number of elapsed frames to zero.

[0081] As described above, the inference process shown in Fig. 9 and the process shown in Fig. 11 are repeatedly executed, and therefore, when the process is to be terminated, it is terminated by an interrupt process. In this case, termination process may be performed after all processes for the period at the time the interrupt is input are completed.

[0082] Finally, Figure 12 shows an example of periodic processing. This shows a case where inference can be performed on two surfaces per frame (i.e., T = 2). The number of frames processed within a period is set to 4 (i.e., R = 4). The image is divided into four sections (two sections vertically and horizontally). The allocation of inference execution targets in period 0 represents the initial setting. In this example, in period 0, inference processing is performed twice for each partial surface. That is, the number of new objects and discarded objects is also counted twice, and at the end of period 0, the average of the two values ​​is calculated for each partial surface, and the allocation is changed so that the number of allocations in the next period increases in descending order of the value. Here, an example is shown in which allocation is performed with the priority of surface #0 > surface #1 > surface #2 = surface #3. The above process is repeated.

[0083] As described above, the inference processing device according to the first embodiment performs object detection by dividing each frame image of a video, counting the number of new objects and the number of discarded objects in each partial plane, which is a partial image obtained by dividing, for each partial plane, and determining the partial plane on which inference processing is performed based on the counted values ​​for each frame image. This reduces the amount of calculation and enables highly accurate object detection. In particular, CNN inference processing can be performed with a low amount of calculation and high accuracy even for high-resolution images that exceed the constraint size.

[0084] Second Embodiment Next, an inference processing device according to a second embodiment will be described. Note that parts having the same configuration as those in the first embodiment will be given the same reference numerals and descriptions thereof will be omitted.

[0085] 13, an inference processing device 200 according to the second embodiment will be described taking as an example a case where a moving image is input, pose estimation is performed as inference processing, and metadata representing the pose estimation result is output. FIG. 4 above is a block diagram showing the hardware configuration of the inference processing device 200 according to this embodiment.

[0086] As shown in FIG. 5, the inference processing device 200 functionally comprises a motion estimator 100, a division processor 210, a counter 120, an allocation controller 122, an overall processor 230, and a synthesis processor 140.

[0087] The division processing unit 210 divides each frame image of the moving image to perform posture estimation.

[0088] The overall processing unit 230 reduces the entire image and performs pose estimation.

[0089] The synthesis processing unit 140 synthesizes the posture estimation result of the division processing unit 210 and the posture estimation result of the overall processing unit 230 to obtain a final posture estimation result.

[0090] As shown in FIG. 14 , the division processing unit 210 includes a division unit 111 , a first selection unit 112 , a posture estimation unit 213 , a frame generation unit 214 , a holding unit 114 , a second selection unit 116 , and a coordinate correction unit 117 .

[0091] The posture estimation unit 213 performs posture estimation to obtain the skeletal information of a person for each of the selected partial faces using a posture estimation model, which is a convolutional neural network that receives an image as input and outputs skeletal information of a person as posture information of an object.

[0092] Specifically, the posture estimation model is a trained model that has been trained in advance to output skeletal information including the positions of multiple feature points and the connections between the feature points for each person included in the input image.

[0093] The posture estimation model may be stored in a storage medium such as the storage 14, or may be stored in an external cloud server. As a technique for performing object detection based on such images, known methods may be appropriately applied.

[0094] Unlike object detection, pose estimation does not output detection frame information. Therefore, the frame generation unit 214 generates a detection frame to surround the skeletal information of a person, which is the output result of pose estimation. In this way, even for pose estimation, thinning control can be achieved using a method similar to that of the first embodiment.

[0095] The storage unit 114 stores the results of pose estimation for each partial surface by the pose estimation unit 213 for each frame image.

[0096] The second selection unit 116 selects, for each frame image, partial surfaces on which inference processing is not performed, in accordance with the determination by the allocation control unit 122 .

[0097] The coordinate correcting unit 117 estimates the position information of the detection frame in the current frame image from the detection frame for the immediately previous frame image based on the motion vector.

[0098] For each partial plane for which inference processing is not performed, the position of the detection frame detected in that partial plane of the previous frame image is corrected based on the motion vector.

[0099] As shown in FIG. 15 , the overall processing unit 230 includes a reduction unit 131 and a posture estimation unit 232 .

[0100] The reduction unit 131 reduces the entire image for each frame image of the moving image.

[0101] The posture estimation unit 232 estimates the posture of each frame image of the moving image from the reduced image.

[0102] The other configurations and operations of the inference processing device 200 according to the second embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0103] As described above, the inference processing device according to the second embodiment divides each frame image of a video sequence to perform pose estimation, counts the number of new objects and the number of discarded objects in each partial plane, which is the partial image obtained by dividing, and determines the partial plane to perform inference processing for each frame image based on the counted values. This reduces the amount of calculation and enables pose estimation with high accuracy.

[0104] In the above embodiments, the various processes executed by the CPU after reading the software (programs) may be executed by various processors other than the CPU. Examples of such processors include a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture, such as an FPGA (Field-Programmable Gate Array), and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute specific processes. Furthermore, the various processes may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0105] In addition, in each of the above embodiments, the inference processing program is pre-stored (installed) in the storage 14, but this is not limiting. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0106] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and modified examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, unnecessary parts may be deleted, new elements may be added, or new elements may be replaced with other elements from the above-described description and illustrations, within the scope of the gist of the technology of the present disclosure.

[0107] The following additional notes are provided regarding the above-described embodiments.

[0108] (Supplementary Item 1) An inference processing device that performs inference processing using a moving image as input and outputs object position information, comprising: a memory; and at least one processor connected to the memory, wherein the processor is configured to: divide each frame image of the moving image to perform object detection; count the number of new objects and the number of discarded objects in partial planes, which are partial images obtained by dividing, for each partial plane; determine the partial planes on which inference processing is performed based on the counted values ​​for each frame image; reduce the entire frame image to perform object detection; and obtain a final object detection result by combining the result of the object detection after dividing and the result of the object detection after reducing the entire frame image.

[0109] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to execute inference processing that takes a moving image as input and outputs object position information, wherein the inference processing: divides each frame image of the moving image to perform object detection; counts the number of new objects and the number of discarded objects in partial planes, which are partial images obtained by dividing, for each partial plane; determines the partial plane on which inference processing is performed based on the counted values ​​for each frame image; reduces the entire frame image to perform object detection; and obtains a final object detection result by combining the result of the object detection after dividing and the result of the object detection after reducing the entire frame image.

[0110] (Supplementary Item 3) An inference processing device that performs inference processing using a moving image as input and outputs pose information of an object, comprising: a memory; and at least one processor connected to the memory, wherein the processor is configured to: divide each frame image of the moving image to perform pose estimation; generate a detection frame within a partial plane that is a partial image obtained by dividing; count the number of new objects and the number of discarded objects within the partial plane for each partial plane; determine the partial plane on which to perform inference processing for each frame image based on the counted values; reduce the entire frame image to perform pose estimation; and obtain a final pose estimation result by combining the result of the pose estimation after dividing and the result of the pose estimation after reducing the entire frame image.

[0111] (Addendum 4) A non-transitory storage medium storing a program executable by a computer to execute inference processing that takes a moving image as input and outputs pose information of an object, wherein the inference processing: divides each frame image of the moving image to perform pose estimation; generates a detection frame within a partial plane that is a partial image obtained by dividing; counts the number of new objects and the number of discarded objects within the partial plane for each partial plane; determines the partial plane on which to perform inference processing for each frame image based on the counted values; reduces the entire frame image to perform pose estimation; and obtains a final pose estimation result by combining the result of the pose estimation after dividing and the result of the pose estimation after reducing the entire frame image.

[0112] 10, 200 Inference processing device 11 CPU 14 Storage 15 Input unit 16 Display unit 100 Motion estimation unit 110 Segmentation processing unit 111 Segmentation unit 112 First selection unit 113 Object detection unit 114 Storage unit 116 Second selection unit 117 Coordinate correction unit 120 Counting unit 122 Allocation control unit 130 Overall processing unit 131 Reduction unit 132 Object detection unit 140 Synthesis processing unit 210 Segmentation processing unit 213 Attitude estimation unit 214 Frame generation unit 230 Overall processing unit 232 Attitude estimation unit

Claims

1. An inference processing device that performs inference processing using a video as input and outputs object position information, comprising: a segmentation processing unit that divides each frame image of the video and performs object detection; a counting unit that counts the number of new objects and the number of discarded objects in partial planes, which are partial images obtained by segmentation, for each partial plane; an allocation control unit that determines the partial planes on which inference processing is performed based on the values ​​counted by the counting unit for each frame image; an overall processing unit that reduces the entire frame image and performs object detection; and a combination processing unit that obtains a final object detection result by combining the object detection result of the segmentation processing unit and the object detection result of the overall processing unit.

2. An inference processing device as described in claim 1, further comprising a motion estimation unit which calculates a motion vector for each frame image across the entire image, wherein the segmentation processing unit includes a coordinate correction unit which estimates position information of the object in the current frame image from position information of the object obtained from the result of the inference processing for the previous frame image based on the motion vector, and wherein the counting unit calculates the degree of overlap between the detection frame of each object obtained from the estimation result by the coordinate correction unit and the detection frame of each object included in the result of performing the inference processing on the current frame image, and counts the number of the new objects and the number of the discarded objects.

3. The allocation control unit of the inference processing device described in claim 1 has a predetermined allocation period defined by the number of frames, calculates, for each partial surface, an average value of the total or weighted sum of the number of new objects and discarded objects in the current allocation period, and determines, for each partial surface, the number of times to execute the inference processing for the partial surface in the next allocation period based on the calculated average value.

4. An inference processing device that performs inference processing using a moving image as input to output object posture information, comprising: a segmentation processing unit that divides each frame image of the moving image to perform posture estimation; a frame generation unit that generates a detection frame within a partial plane that is a partial image obtained by segmentation; a counting unit that counts the number of new objects and the number of discarded objects within the partial plane for each partial plane; an allocation control unit that determines the partial plane for performing inference processing based on the value counted by the counting unit for each frame image; an overall processing unit that reduces the entire frame image to perform posture estimation; and a synthesis processing unit that obtains a final posture estimation result by synthesizing the posture estimation result of the segmentation processing unit and the posture estimation result of the overall processing unit.

5. An inference processing method for an inference processing device that performs inference processing that takes a video image as input and outputs object position information, wherein a segmentation processing unit segments each frame image of the video image and performs object detection, a counting unit counts the number of new objects and the number of discarded objects in partial planes, which are partial images obtained by segmentation, for each partial plane, an allocation control unit determines the partial plane for which inference processing is performed based on the value counted by the counting unit for each frame image, an overall processing unit reduces the entire frame image and performs object detection, and a synthesis processing unit obtains a final object detection result by synthesizing the object detection result of the segmentation processing unit and the object detection result of the overall processing unit.

6. An inference processing method for an inference processing device that performs inference processing that takes a moving image as input and outputs object posture information, wherein a segmentation processing unit segments each frame image of the moving image and performs posture estimation, a frame generation unit generates a detection frame within a partial plane that is a partial image obtained by segmentation, a counting unit counts the number of new objects and the number of discarded objects within the partial plane for each partial plane, an allocation control unit determines the partial plane for performing inference processing based on the value counted by the counting unit for each frame image, an overall processing unit reduces the entire frame image and performs posture estimation, and a synthesis processing unit obtains a final posture estimation result by synthesizing the posture estimation result of the segmentation processing unit and the posture estimation result of the overall processing unit.

7. An inference processing program for causing a computer to function as an inference processing device according to any one of claims 1 to 5.

Images