Inference processing device and inference processing method

Abstract

This inference processing method assigns an input frame to one of either a first process which obtains a posture estimation result by performing inference, or a second process which, on the basis of motion vectors, corrects the position of each joint point of a posture estimation result of a previous frame. When the input frame is assigned to the first process, the inference processing method performs inference processing in relation to the input frame to obtain a posture estimation result. When the input frame is assigned to the second process, the inference processing method, with regard to the input frame, generates a border, calculates the size of an object, and determines the corresponding block count of each joint point. The inference processing method, with regard to the input frame, corrects the coordinates of joint points based on the corresponding block count of each joint point and calculated motion vectors.

Description

Inference processing device and inference processing method

[0001] The disclosed technology relates to an inference processing device and an inference processing method.

[0002] Edge AI systems that perform CNN inference processing such as object detection and pose estimation on edge devices such as surveillance cameras and drones are attracting attention. However, these CNN inferences are limited in the input image size that can be processed at one time due to constraints of the model itself and memory capacity constraints on the edge device.

[0003] For example, Patent Document 1 discloses a method for performing inference processing on the entire surface obtained by reducing an input image to a constrained image size and on the divided surfaces (divided images) obtained by dividing the input image, and then synthesizing the results, in order to enable object detection and pose estimation for high-resolution images.

[0004] In addition, the amount of calculation may become enormous depending on the number of image divisions in the division process. For example, if the input size is 608 × 608, in an ultra-high definition video such as 4K (3840 × 2160), the number of image divisions in the division process becomes approximately 28, which results in an enormous amount of calculation.

[0005] Regarding object detection, for example, Patent Document 2 discloses a method for reducing the amount of calculation by narrowing down (thinning out) partial planes on which object detection is performed for the same video frame. In the technology of Patent Document 2, when thinning out partial planes, in order to prioritize inference processing for partial planes with large scene changes, the absolute difference between the average number of object detections over a certain period of time in the past and the number of object detections in the current frame is calculated, and partial planes with the largest difference value are prioritized for processing. Furthermore, for partial planes in a video frame on which inference is not performed, the position coordinates of the detected object in the previous frame are corrected based on a separately calculated motion vector, and the result is used as the detection result.

[0006] International Publication No. WO 2021 / 161513 International Publication No. WO 2023 / 238248

[0007] Regarding posture estimation, thinning of the inference execution plane can be realized by correcting the position coordinates of each joint point in the posture estimation result of the previous video frame based on the motion vector, as in the above method. Specifically, each joint point is moved to a position where it is predicted to move, based on the motion vector of the coordinates where each joint point is located.

[0008] However, because objects vary in size, a fixed block size cannot accommodate all objects. Figures 1A and 1B are diagrams illustrating the relationship between block size and articulation points. Note that the dashed grid in the diagrams indicates the motion vector calculation block (N × N pixels), and the solid-line frame indicates the corresponding block for the articulation point. The arrows indicate motion vectors. Figure 1A shows an example in which the block size is too small compared to the actual articulation point size. Figure 1B shows an example in which the block size is too large compared to the actual articulation point size. When each articulation point moves, the motion vector is generally calculated in block units, with each block consisting of N × N pixels. However, the size of each person displayed varies in relation to the image size. Therefore, if the block size is small compared to the number of pixels occupied by the articulation point, a motion vector at a position other than the articulation point may be adopted if the articulation point moves rapidly ( Figure 1A ). On the other hand, if the block size is large compared to the number of pixels occupied by the joint points, multiple joint points may be corrected with the same motion vector, resulting in the joint points being corrected to positions different from their actual destinations (Fig. 1B).In either case, the accuracy of pose estimation deteriorates due to the joint points being corrected to positions different from their original destinations.

[0009] The disclosed technology has been made in consideration of the above points, and aims to provide an inference processing device and an inference processing method that can suppress a decrease in accuracy of pose estimation that can occur due to variations in object size.

[0010] A first aspect of the present disclosure is an inference processing device that receives a moving image as input and outputs a posture estimation result for an object having joints, the inference processing device including: a first storage unit that stores each frame of the moving image; a vector calculation unit that calculates a motion vector for an input frame of the moving image by taking a pixel difference from a previous frame; a processing control unit that assigns the input frame to either a first process that obtains a posture estimation result by performing inference or a second process that corrects positions of each joint point in the posture estimation result for the previous frame based on the motion vector; an inference processing unit that, when assigned to the first process, performs inference processing on the input frame and obtains a posture estimation result; and a a frame generation unit that, when assigned to the second process, generates a frame for the input frame based on the pose estimation result of the previous frame; a size calculation unit that calculates the size of the object for the input frame according to the length of the generated frame; a block number determination unit that determines the number of blocks corresponding to each joint point for the input frame according to the size of the object; a correction unit that corrects the coordinates of the joint points for the input frame based on the number of blocks corresponding to each joint point and the calculated motion vectors; and an output selection unit that outputs, for the input frame, either the pose estimation result obtained by performing inference or the pose estimation result obtained by correction using the motion vectors.

[0011] A second aspect of the present disclosure is an inference processing method in an inference processing device that receives a moving image as input and outputs a posture estimation result for an object having joints, wherein a first storage unit stores each frame of the moving image, and a motion vector is calculated for each input frame of the moving image by taking a pixel difference from a previous frame, and the input frame is assigned to either a first process that obtains a posture estimation result by performing inference, or a second process that corrects positions of each joint point in the posture estimation result of the previous frame based on the motion vector, and when the input frame is assigned to the first process, an inference process is performed on the input frame to obtain a posture estimation result, and the posture estimation result is stored in a second storage unit. The computer executes the following processes: a posture estimation result for an input frame is held, and when the input frame is assigned to the second process, the computer generates a frame for the input frame based on the posture estimation result of the previous frame, calculates the size of the object for the input frame according to the length of the generated frame, determines the number of blocks corresponding to each joint point according to the size of the object for the input frame, corrects the coordinates of the joint points based on the number of blocks corresponding to each joint point and the calculated motion vectors for the input frame, and outputs, for the input frame, either the posture estimation result obtained by performing inference or the posture estimation result obtained by correction using the motion vectors.

[0012] The disclosed technology can suppress a decrease in accuracy of pose estimation that may occur due to variations in object size.

[0013] FIG. 1A is a diagram illustrating the relationship between block size and articulation points. FIG. 1B is a diagram illustrating the relationship between block size and articulation points. FIG. 2A is a diagram illustrating a division mode of an input frame. FIG. 2B is a diagram illustrating a division mode of an input frame. FIG. 2C is a diagram illustrating a division mode of an input frame. FIG. 3 is a block diagram illustrating the hardware configuration of an inference processing device. FIG. 4 is a block diagram illustrating the configuration of an inference processing device according to a first embodiment. FIG. 5 is a diagram illustrating the relationship between block size and articulation points in this technique. FIG. 6A is an example of a posture estimation result. FIG. 6B is an example of setting a corresponding block size for each articulation point. FIG. 7 is a flowchart illustrating the flow of inference processing by the inference processing device according to the first embodiment. FIG. 8A illustrates a change mode of block size according to a second embodiment. FIG. 8B illustrates a change mode of block size according to the second embodiment. FIG. 9 is a block diagram illustrating the configuration of an inference processing device according to the second embodiment. FIG. 10 is a flowchart illustrating the flow of inference processing by the inference processing device according to the second embodiment.

[0014] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same reference numerals are used to designate identical or equivalent components and parts in each drawing. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0015] First, an overview of the technology disclosed herein will be described. The purpose of this technology is to provide a method for achieving pose estimation using CNN inference with a small amount of computation and high accuracy, even for high-resolution images (e.g., full HD, 4K) that exceed the above-mentioned constraint size. Pose estimation is a technology for detecting skeletal information (information on feature points such as eyes, nose, and joints, and the edges connecting them) of a person included in an input image.

[0016] For partial surfaces on which inference is not performed, the position coordinates of the detected object in the previous frame are corrected based on a separately calculated motion vector and the result is used as the detection result. In this case, the motion vector is calculated for each block of NxN pixels, which is the input image divided into a grid. The motion vector of the block in which the articulation point is located is then used to correct the coordinates of the articulation point to the predicted destination position of the articulation point. The technology disclosed herein solves the problem of possible accuracy degradation when the block size is small or large relative to the number of pixels occupied by the articulation point by changing the size of the block from which the motion vector corresponding to each articulation point is obtained depending on the size of the object.

[0017] The following describes the embodiments according to the division manner of the input frame. Figures 2A to 2C are diagrams illustrating the division manner of the input frame. Figure 2A shows an example in which the input frame is not divided (corresponding to the first and second embodiments). Figure 2B shows an example in which the input frame is divided (corresponding to the third embodiment). Figure 2C shows a case in which processing is performed on both the reduced whole frame and the divided frames (corresponding to the fourth embodiment).

[0018] FIG. 3 is a block diagram showing the hardware configuration of the inference processing device 100. The hardware configuration is common to all embodiments. As shown in FIG. 3, the inference processing device 100 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so that they can communicate with each other. The hardware configuration is common to all embodiments.

[0019] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads the programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, the programs are stored in the ROM 12 or the storage 14.

[0020] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0021] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0022] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

[0023] The communication interface 17 is an interface for communicating with other devices such as terminals, etc. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0024] First Embodiment Next, each functional configuration of the inference processing device 100 will be described. FIG. 4 is a block diagram showing the configuration of the inference processing device of the first embodiment. Each functional configuration is realized by the CPU 11 reading out a program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. The inference processing device 100 receives a moving image as input and outputs a posture estimation result for an object having joints. In this embodiment, a case where the object is a person will be described as an example. Note that the present invention is not limited to a person and can be applied to any object having joints, including animals and robots having joints.

[0025] As shown in Figure 4, the inference processing device 100 is configured to include a first holding unit 110, a vector calculation unit 112, a processing control unit 114, an inference processing unit 116, a second holding unit 118, a frame generation unit 120, a size calculation unit 122, a block number determination unit 124, a correction unit 126, and an output selection unit 128.

[0026] The inference processing device 100 receives each frame of a video as input and stores it in the first storage unit 110. The first storage unit 110 stores each frame of the video. Each frame of the video is processed sequentially as an input frame in the following processing.

[0027] As shown in FIG. 2A , each frame (MF) of an input video sequence is divided into a frame (α1) for which a posture estimation result is obtained by performing inference and a frame (α2) for which the position of each joint point is corrected. For the correction, a motion vector (vf) is calculated from the pixel difference with the previous frame, and the position of each joint point in the posture estimation result of the previous frame is corrected based on the calculated motion vector. Note that the previous frame is not limited to the frame immediately preceding, but may be any frame that can be compared in the time series.

[0028] For example, a rule-based method may be considered, such as performing inference on one frame out of every five frames to obtain a posture estimation result, and performing correction using a motion vector on the frames in between to obtain a posture estimation result. However, the method is not limited to such a static periodic determination method, and may also be a method in which whether to perform inference is determined dynamically in response to some trigger.

[0029] The vector calculation unit 112 calculates a motion vector for an input frame of a moving image by taking the pixel difference from the previous frame.

[0030] The processing control unit 114 assigns processing to either the first processing or the second processing for an input frame. The first processing is processing in which the inference processing unit 116 executes inference to obtain a posture estimation result. The second processing is processing in which the inference processing unit 116 corrects the positions of each joint point in the posture estimation result of the previous frame based on the motion vector to obtain a posture estimation result. Which processing is assigned to can be determined based on the above-mentioned rule base. The second processing corresponds to processing by the frame generation unit 120, size calculation unit 122, block number determination unit 124, and correction unit 126.

[0031] When an input frame is assigned to the first process, the inference processing unit 116 performs the inference process on the input frame to obtain a posture estimation result, which is stored in the second storage unit 118.

[0032] The second storage unit 118 stores the posture estimation results of the input frame that has undergone the inference processing.

[0033] When an input frame is assigned to the second process, the frame generating unit 120 generates a frame for the input frame based on the posture estimation result of the previous frame.

[0034] The size calculation unit 122 calculates the size of the object for the input frame based on the length of the generated frame. The block number determination unit 124 determines the number of blocks corresponding to each joint point for the input frame based on the size of the object. The correction unit 126 corrects the coordinates of the joint points for the input frame based on the number of blocks corresponding to each joint point and the calculated motion vector.

[0035] The output selection unit 128 outputs, for an input frame, either the posture estimation result obtained by performing inference or the posture estimation result obtained by correction using a motion vector.

[0036] FIG. 5 is a diagram showing the relationship between block size and articulation points in this method. As shown in FIG. 5 , when pose estimation results are obtained by correction using motion vectors, the block size for calculating motion vectors is fixed, and the number of blocks corresponding to articulation points is changed depending on the size of the object, thereby solving the aforementioned problem. Specifically, the number of corresponding blocks is reduced for people who appear small, and increased for people who appear large. In this case, if the number of corresponding blocks is two or more, the position of the corresponding articulation point is corrected using the average vector of the motion vectors corresponding to each block. Here, the average vector is a vector whose component values ​​are the average values ​​of the vector components. However, vectors whose components are all zero are not subject to averaging. Instead of averaging each component, the median of each component may be used.

[0037] FIG. 6A shows an example of a pose estimation result. FIG. 6B shows an example of setting the corresponding block size for each joint point. The size of an object is defined by the length of the long side of the frame surrounding the pose estimation result, as shown in (L) of FIG. 6A . However, the definition of the object size is not limited to this, and may be the length of the diagonal of the frame surrounding the pose estimation result, the area of ​​the frame, or the like. Here, the pose estimation result is assumed to consist of 14 joint points, but this is not intended to be limiting. For example, it is conceivable that finer feature points such as the eyes and nose of a face may be included as joint points. Then, as shown in the table in FIG. 6B , the number of corresponding blocks for each joint point is defined according to the range of object size. Here, the object size range is classified into three ranges, but any number of classifications may be possible. Here, the number of corresponding blocks is defined to be square, but may also be rectangular, cross, or the like. Here, when the object size range is the same, the same number of corresponding blocks is used for all joint points, but the number of corresponding blocks may vary depending on the joint point. For example, if eyes, noses, etc. are included as joint points, it is generally considered that their size is smaller than that of hands and feet, so it is conceivable to define the number of blocks corresponding to eyes and noses to be less than the number of blocks corresponding to hands and feet. In this way, a size range may be defined for each joint point according to the type of the joint point.

[0038] The block number determination unit 124 calculates the size of the object for all pose estimation results included in the frames of the input video, and determines the number of blocks corresponding to each joint point according to the calculated object size. The correction unit 126 calculates a motion vector for correcting the position of each joint point according to the number of blocks, and changes the coordinates of each joint point to a position to which the joint point is predicted to move based on the calculated motion vector.

[0039] Next, we will explain the operation of the inference processing device 100. Figure 7 is a flowchart showing the flow of inference processing by the inference processing device 100 of the first embodiment. The inference processing is performed by the CPU 11 reading a program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it.

[0040] In step S100 , the CPU 11 receives input frames of moving images from the first storage unit 110 .

[0041] In step S102, the CPU 11 determines whether the input frame is the first frame. If it is the first frame, the process proceeds to step S108 to execute inference processing. This is because there are no previous frames for the first frame, so motion vectors are not calculated. In other words, correction using motion vectors is not possible, so the posture estimation result is always obtained for the first frame by executing inference. If it is not the first frame, the process proceeds to step S104.

[0042] In step S104, the CPU 11 functions as the vector calculation unit 112 to calculate a motion vector for the input frame by taking the pixel difference from the previous frame.

[0043] In step S106, the CPU 11, functioning as the process control unit 114, determines whether to perform a first process for executing inference on the input frame or a second process for obtaining a posture estimation result by correction. If the first process is to be performed, the process proceeds to step S108. If the second process is to be performed, the process proceeds to step S110.

[0044] In step S108, the CPU 11 functions as the inference processing unit 116 to perform inference processing on the input frame to obtain a posture estimation result. That is, the input frame is a target surface for inference execution.

[0045] In step S110, the CPU 11 determines whether the input frame is to be corrected using a motion vector. If it is, the process proceeds to step S112. If it is not, the process returns to step S100, where the next input frame is accepted and processing continues. The input frame that is to be corrected is the surface to be corrected.

[0046] In step S112, the CPU 11 functions as the frame generating unit 120 to generate a frame for the input frame based on the posture estimation result of the previous frame.

[0047] In step S114, the CPU 11 functions as the size calculation unit 122 to calculate the size of the object in the input frame according to the length of the generated frame.

[0048] In step S116, the CPU 11 functions as the block number determination unit 124 to determine the number of blocks corresponding to each joint point in the input frame in accordance with the size of the object.

[0049] In step S118, the CPU 11, functioning as the correction unit 126, corrects the coordinates of the joint points of the input frame based on the number of blocks corresponding to each joint point and the calculated motion vector, thereby obtaining a posture estimation result.

[0050] In step S120, the CPU 11 functions as the output selection unit 128 to output, for the input frame, either the posture estimation result obtained by performing inference or the posture estimation result obtained by correction using a motion vector.

[0051] In step S122, the CPU 11 determines whether or not a termination condition is satisfied. If the termination condition is satisfied, the processing flow is terminated. If the termination condition is not satisfied, the processing returns to step S100 and the processing is repeated. Possible termination conditions include when there are no more input frames (i.e., when all frames of the moving image have been processed) or when an external interrupt instruction is issued, but the termination condition is not limited to these.

[0052] As described above, the inference processing device 100 of this embodiment can suppress a decrease in accuracy of pose estimation that may occur due to variations in object size.

[0053] [Second Embodiment] In the second embodiment, as in the first invention, each frame of a video is divided into frames for which a pose estimation result is obtained by performing inference and frames for which the positions of each joint point are corrected, as shown in Fig. 1A. The difference from the first embodiment is that the block size for calculating motion vectors is changed.

[0054] 8A and 8B show how block size is changed in the second embodiment. As shown in FIGS. 8A and 8B, the block size is changed according to the size of the smallest person in the image. This allows motion vectors to be calculated using the minimum necessary block size, thereby reducing the amount of calculation required for motion vector calculation. The block size according to the size of the smallest object is set in advance. However, for example, if the setting shown in FIG. 6B is created based on the minimum block size, the block size can be determined based on that setting. For example, FIG. 6B shows a setting where the block size is 8. Assume, for example, that the size of the smallest object is greater than 80 pixels and less than or equal to 400 pixels. In this case, the block size is changed so that 3×3 blocks are treated as one block, resulting in a block size of 24×24 pixels. Following the block size change, the block size corresponding to each joint point can be read as 1×1 (3×3) and 2×2 (5×5). 5 / 3 = 1.66, but this is rounded up to 2.

[0055] 9 is a block diagram showing the configuration of an inference processing device according to the second embodiment. It is basically the same as the first embodiment, but a block size modification unit 202 is added. The block size modification unit 202 modifies the block size for motion vector calculation in accordance with the calculated smallest object size based on the smallest object size in the input frame. Here, the smallest object size among the object sizes calculated by the size calculation unit 122 may be referenced. When the block size is modified, the vector calculation unit 112 may recalculate and correct the motion vector for the input frame using the modified block size.

[0056] 10 is a flowchart showing the flow of inference processing by the inference processing device 100 of the second embodiment. Basically, this is the same as in the first embodiment, but step S200 is executed after step S114. In step S200, the CPU 11 functions as the block size change unit 202 to change the block size for calculating motion vectors in accordance with the size of the smallest object. In step S202, the CPU 202 functions as the vector calculation unit 112 to calculate motion vectors for the input frame using the changed block size.

[0057] As described above, the inference processing device 100 of this embodiment can suppress a decrease in accuracy of pose estimation that may occur due to variations in object size, while taking into consideration the minimum size of the object.

[0058] In the third embodiment, as shown in FIG. 1B , an input video frame is divided into several partial planes (dMF). The processing control unit 114 in the third embodiment allocates processing to obtain a pose estimation result for each divided partial plane, either by performing inference or by correcting using a motion vector. Furthermore, two patterns can be applied, depending on whether the block size is fixed as in the first embodiment or variable as in the second embodiment.

[0059] The components of the third embodiment will now be described. The inference processing device 100 receives as input each frame of a moving image, divides it into partial planes, and stores them in the first storage unit 110.

[0060] The processing control unit 114 assigns each partial surface of the input frame to either the first processing or the second processing. When a partial surface is assigned to the first processing, the inference processing unit 116 performs inference processing on the partial surface to obtain a posture estimation result. The posture estimation result is stored in the second storage unit 118. The second storage unit 118 stores the posture estimation result of the partial surface of the input frame that has been inferred.

[0061] When a partial surface is assigned to the second process, the frame generation unit 120 generates a frame for the partial surface of the input frame based on the pose estimation result of the previous frame. The size calculation unit 122 calculates the size of the object for the partial surface of the input frame according to the length of the generated frame. The block number determination unit 124 determines the number of blocks corresponding to each articulation point for the partial surface of the input frame according to the size of the object. The correction unit 126 corrects the coordinates of the articulation points for the partial surface of the input frame based on the number of blocks corresponding to each articulation point and the calculated motion vector. The output selection unit 128 outputs either the pose estimation result obtained by inference execution or the pose estimation result obtained by correction using the motion vector for the partial surface of the input frame.

[0062] As described above, the inference processing device 100 of this embodiment can further divide and process frames of a moving image, thereby suppressing a decrease in accuracy of pose estimation that may occur due to variations in object size.

[0063] [Fourth Embodiment] In the fourth embodiment, as shown in FIG. 1C , processing is performed on both a frame (CF) obtained by reducing the entire frame of the input video and a frame (dMF) obtained by dividing the frame of the input video. In this case, inference by the inference processing unit 116 is always performed on the reduced-size frame. This embodiment applies the techniques disclosed in Patent Documents 1 and 2, and outputs a final pose estimation result by combining the pose estimation result obtained from the reduced-size frame and the pose estimation result obtained from the divided partial planes. The method of combination, etc., follows the methods described in Patent Documents 1 and 2. Furthermore, two patterns can be applied depending on whether the block size is fixed as in the first embodiment or variable as in the second embodiment.

[0064] As described above, according to the inference processing device 100 of this embodiment, video frames can be further divided into reduced frames and divided frames for processing, thereby suppressing the deterioration in accuracy of pose estimation that may occur due to variations in object size.

[0065] In addition, the inference processing executed by the CPU after reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include dedicated electrical circuits, such as programmable logic devices (PLDs) whose circuit configuration can be changed after manufacture, such as field-programmable gate arrays (FPGAs), graphics processing units (GPUs), and application-specific integrated circuits (ASICs), which are processors having a circuit configuration designed specifically to execute specific processing. The inference processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0066] In addition, in each of the above embodiments, the program is described as being pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0067] The following additional notes are provided regarding the above-described embodiments.

[0068] (Supplementary Item 1) A system including a memory and at least one processor connected to the memory, wherein the processor: a first storage unit stores each frame of a moving image; calculates a motion vector for an input frame of the moving image by taking a pixel difference from a previous frame; assigns the input frame to either a first process for obtaining a posture estimation result by performing inference or a second process for correcting the position of each joint point in the posture estimation result of the previous frame based on the motion vector; when the input frame is assigned to the first process, performs inference processing on the input frame to obtain a posture estimation result; holds the posture estimation result of the input frame that has been inferred in a second storage unit; when the input frame is assigned to the second process, generates a frame for the posture estimation result of the previous frame for the input frame; calculates the size of the object according to the length of the generated frame; determines the number of blocks corresponding to each joint point according to the size of the object for the input frame; corrects the coordinates of the joint points for the input frame based on the number of blocks corresponding to each joint point and the calculated motion vector; and outputting, for the input frame, either a posture estimation result obtained by performing inference or a posture estimation result obtained by correction using a motion vector.

[0069] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to execute inference processing, wherein a first storage unit stores each frame of a moving image, and for an input frame of the moving image, a motion vector is calculated by taking a pixel difference from a previous frame, and the input frame is assigned to either a first process for obtaining a posture estimation result by performing inference or a second process for correcting a position of each joint point in the posture estimation result of the previous frame based on the motion vector, and when the input frame is assigned to the first process, inference processing is performed on the input frame to obtain a posture estimation result, and a second storage unit stores the posture estimation result of the input frame that has been inferred, and when the input frame is assigned to the second process, a frame is generated for the input frame based on the posture estimation result of the previous frame, and for the input frame, a size of the object is calculated according to the length of the generated frame, and for the input frame, the number of blocks corresponding to each joint point is determined according to the size of the object, and for the input frame, coordinates of the joint points are corrected based on the number of blocks corresponding to each joint point and the calculated motion vector, a motion vector correction unit that corrects the motion vectors of the input frame and generates a motion vector correction result based on the motion vectors of the input frame, and outputs the motion vector correction result based on the motion vectors of the input frame and the motion vectors of the input frame.

[0070] REFERENCE SIGNS LIST 100 Inference processing device 110 First holding unit 112 Vector calculation unit 114 Processing control unit 116 Inference processing unit 118 Second holding unit 120 Frame generation unit 122 Size calculation unit 124 Number of blocks determination unit 126 Correction unit 128 Output selection unit 202 Block size change unit

Claims

1. An inference processing device that receives a video as input and outputs a posture estimation result for an object having joints, comprising: a first storage unit that stores each frame of the video; a vector calculation unit that calculates a motion vector for each input frame of the video by taking pixel differences from a previous frame; a process control unit that assigns the input frame to either a first process that performs inference to obtain a posture estimation result, or a second process that corrects the position of each joint point in the posture estimation result for the previous frame based on the motion vector; an inference processing unit that, if assigned to the first process, performs inference processing on the input frame to obtain a posture estimation result; a second storage unit that stores the posture estimation result for the input frame that has been inferred; a frame generation unit that, if assigned to the second process, generates a frame for the posture estimation result for the previous frame for the input frame; a size calculation unit that calculates the size of the object according to the length of the generated frame for the input frame; a block number determination unit that determines the number of blocks corresponding to each joint point according to the size of the object for the input frame; and a correction unit that corrects the coordinates of the joint points for the input frame based on the number of blocks corresponding to each joint point and the calculated motion vector. an output selection unit that outputs, for the input frame, either a posture estimation result obtained by performing inference or a posture estimation result obtained by correction using a motion vector.

2. The inference processing device according to claim 1, further comprising a block size change unit that changes the block size for calculating a motion vector in accordance with the size of the smallest object among the object sizes calculated by the size calculation unit, and the vector calculation unit calculates a motion vector for the input frame using the changed block size.

3. Dividing each frame of the moving image into partial surfaces, the processing control unit assigns the partial surface of the input frame to either a first process for obtaining a posture estimation result by performing inference or a second process for correcting the position of each joint point in the posture estimation result of the previous frame based on a motion vector, when the partial surface is assigned to the first process, the inference processing unit performs inference processing on the partial surface of the input frame to obtain a posture estimation result, the second holding unit holds the posture estimation result of the partial surface of the input frame that has been inferred, when the partial surface is assigned to the second process, the frame generation unit generates a frame for the input frame based on the posture estimation result of the previous frame, a size calculation unit calculates the size of the object according to the length of the generated frame for the partial surface of the input frame, the block number determination unit determines the number of blocks corresponding to each joint point according to the size of the object for the partial surface of the input frame, and the correction unit corrects the coordinates of the joint points for the partial surface of the input frame based on the number of blocks corresponding to each joint point and the calculated motion vector, The inference processing device according to claim 1 , wherein the output selection unit outputs, for a partial surface of the input frame, either a posture estimation result obtained by performing inference or a posture estimation result obtained by correction using a motion vector.

4. An inference processing method in an inference processing device that receives a video as input and outputs a posture estimation result for an object having joints, wherein a first storage unit stores each frame of the video, and calculates a motion vector for each input frame of the video by taking pixel differences from a previous frame, and assigns the input frame to either a first process that obtains a posture estimation result by performing inference or a second process that corrects the position of each joint point in the posture estimation result of the previous frame based on the motion vector, and when the input frame is assigned to the first process, performs inference processing on the input frame to obtain a posture estimation result, and a second storage unit stores the posture estimation result of the input frame that has been inferred, and when the input frame is assigned to the second process, generates a frame for the input frame based on the posture estimation result of the previous frame, calculates the size of the object according to the length of the generated frame, determines the number of blocks corresponding to each joint point according to the size of the object, and corrects the coordinates of the joint points for the input frame based on the number of blocks corresponding to each joint point and the calculated motion vector, and outputting, for the input frame, one of a posture estimation result obtained by performing inference and a posture estimation result obtained by correction using a motion vector.

Images