Posture estimation device and posture estimation method

Abstract

This posture estimation device executes an inference process for estimating the posture of an object having a joint from an image included in a moving image and comprises: an acquisition unit that acquires the image from the moving image; an allocation control unit that allocates, for each partial surface obtained by dividing the image, the frequency at which the inference process is to be executed; a division processing unit that executes the inference process on the partial surfaces in accordance with the allocated frequency; an overall processing unit that reduces the image to an entire surface which shows the entirety of the image and that executes the inference process on the entire surface; a motion estimation unit that calculates the motion vector of the object; a coordinate correction unit that corrects, on the basis of the motion vector, the result of the inference process in a previous frame with respect to a partial surface on which the inference process is not executed; a synthesis processing unit that synthesizes the result of the inference processes for each of the partial surfaces and the entire surface and that estimates the posture of the object from the image; and a correction deviation determination unit that determines, for each partial surface, the deviation between the result corrected by the coordinate correction unit and the actual posture of the object. For each partial surface, the allocation control unit allocates the frequency at which the inference process is to be executed in accordance with the determination result of the correction deviation determination unit.

Description

Attitude estimation device and attitude estimation method

[0001] The disclosed technology relates to a posture estimation device and a posture estimation method.

[0002] Edge AI (Artificial Intelligence) systems that perform convolutional neural network (CNN) inference processing such as object detection and pose estimation on edge devices such as surveillance cameras and drones have been attracting attention. However, these CNN inferences are limited in the input image size that can be processed at one time due to limitations in the model itself and memory capacity constraints on the edge devices.

[0003] Pose estimation is a technique for detecting skeletal information (information about feature points such as the eyes, nose, and joints, as well as the edges connecting them) of a person contained in an input image.

[0004] To enable pose estimation for high-resolution images, a method has been disclosed (Patent Document 1) in which an input image is reduced to the above-mentioned constrained image size, an inference process is performed on the entire image, and the divided surfaces are divided, and the results are then combined (Fig. 1). For example, if the input size is 608 x 608, for an ultra-high-resolution video such as 4K (3840 x 2160), the number of image divisions in the division process will be approximately 28, which will result in an enormous amount of calculation.

[0005] Regarding object detection, a method for reducing the amount of calculations by narrowing down (thinning out) partial planes on which object detection is performed in the same video frame is disclosed (Patent Document 2). When thinning out, inference processing is prioritized for partial planes with large scene changes by calculating the absolute difference between the average number of object detections over a certain period of time in the past and the number of object detections in the current frame, and partial planes with the largest difference are prioritized for processing. Furthermore, for partial planes in a certain video frame on which inference is not performed, the position coordinates of the detected object in the previous video frame are corrected based on a separately calculated motion vector, and the result is used as the detection result.

[0006] PCT / JP2020 / 005812PCT / JP2022 / 022986

[0007] Regarding posture estimation, it is possible to thin out the inference execution planes by correcting the position coordinates of each joint point in the posture estimation result of the previous video frame based on the motion vector, as in the method described in the above-mentioned Patent Document 2. However, as time passes, the deviation between the posture estimation result whose coordinates have been corrected based on the motion vector and the actual posture of the person increases, so in the method that prioritizes partial planes with large changes in the number of detected objects as described above, inference is not performed for planes with a constant number of people but large deviations, which may result in a decrease in accuracy.

[0008] The disclosed technology has been made in consideration of the above points, and aims to provide a posture estimation device and posture estimation method that can reduce the amount of calculation and perform posture estimation with high accuracy.

[0009] a motion estimation unit that calculates a motion vector of the object; a coordinate correction unit that corrects a result of the inference process in a previous frame based on the motion vector for a partial surface on which the inference process is not performed, based on the motion vector; a synthesis processing unit that combines the results of the inference process for each of the partial surfaces and the overall surface to estimate the posture of the object from the image; and a correction deviation determination unit that determines, for each partial surface, a deviation between the result of correction by the coordinate correction unit and the posture of the object; and a correction deviation determination unit that determines, for each partial surface, a deviation between the result of correction by the coordinate correction unit and the posture of the object.

[0010] A second aspect of the present disclosure is a posture estimation method for performing an inference process to estimate the posture of an object having joints from an image included in a moving image, wherein a computer acquires the images from the moving image, assigns a frequency for performing the inference process to each partial surface obtained by dividing the image, performs the inference process on the partial surface according to the assigned frequency, reduces the image to a full surface that represents the entire image, performs the inference process on the full surface, calculates a motion vector of the object, corrects a result of the inference process in the previous frame based on the motion vector for each partial surface on which the inference process is not performed, combines the results of the inference process for the partial surface and the full surface to estimate the posture of the object from the image, and determines, for each partial surface, a deviation between the corrected result and the actual posture of the object; and, by assigning the frequency, assigns a frequency for performing the inference process to each partial surface according to the result of the deviation determination.

[0011] According to the disclosed technology, it is possible to reduce the amount of calculation and perform posture estimation with high accuracy.

[0012] 1 is a schematic diagram illustrating an overview of the first embodiment. FIG. 1 is a diagram illustrating a first phenomenon that causes a deviation. FIG. 2 is a diagram illustrating a second phenomenon that causes a deviation. FIG. 3 is a diagram illustrating a third phenomenon that causes a deviation. A block diagram showing an example of the hardware configuration of the posture estimation device of the first, second, and third embodiments. A block diagram showing the configuration of the posture estimation device of the first embodiment. A block diagram showing the input / output of the posture estimation device of the first embodiment. A block diagram showing the configuration of a segmentation processing unit of the posture estimation device of the first embodiment. A block diagram showing the configuration of the overall processing unit of the posture estimation device of the first embodiment. A diagram showing a method for determining the first phenomenon. A diagram showing a method for determining the second phenomenon. A diagram showing frames that are input in determining the correction deviation of the first to third phenomena. A diagram showing frames that are input in determining the first mode and the second mode. A diagram showing an example of an allocation control method. A diagram showing an example of an allocation control method. A flowchart showing the flow of posture estimation processing of the posture estimation device of the first embodiment. A flowchart showing the flow of segmentation processing of the posture estimation device of the first embodiment. A diagram showing the method for determining the third phenomenon. A block diagram showing the input / output of the posture estimation device of the second embodiment. A block diagram showing the configuration of the posture estimation device of the third embodiment. A block diagram showing the flow of segmentation processing of the posture estimation device of the third embodiment.

[0013] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0014] [Summary of the Present Disclosure] In the technology of the present disclosure, in order to enable pose estimation for high-resolution images, a method is adopted in which an input image is reduced to the above-mentioned constrained image size, an inference process is performed on the entire surface, and on the divided partial surfaces, and the results are combined, as shown in Fig. 1. Then, the number of partial surfaces on which inference is performed per input video frame is thinned out to reduce the amount of calculation.

[0015] Furthermore, by periodically determining whether there is a discrepancy between the results corrected by the motion vector and the actual posture of the object, and changing the allocation of inference execution surfaces so that inference execution frequency increases for partial surfaces where a large number of objects are determined to have a discrepancy, the problem of inference not being executed for surfaces with a large discrepancy, which may result in a decrease in accuracy, is solved.

[0016] In the technology disclosed herein, phenomena that cause deviations are classified as follows (FIGS. 2A and 2B). In the first phenomenon, as shown in FIG. 2A, correction based on a partially erroneously calculated motion vector results in some joint point distances becoming extremely long. In the second phenomenon, as shown in FIG. 2B, the entire object slides due to the influence of the motion vectors of surrounding objects. In the third phenomenon, as shown in FIG. 2C, a deviation from the actual posture of an object occurs, although it does not fall under the first or second phenomenon.

[0017] In the first embodiment, only the first phenomenon and the second phenomenon are determined as correction deviations. In the second embodiment, only the second phenomenon and the third phenomenon are determined as correction deviations. In the third embodiment, when the number of objects is large, only the first phenomenon and the second phenomenon are determined as correction deviations, and when the number of objects is small, only the second phenomenon and the third phenomenon are determined as correction deviations.

[0018] The posture estimation device of the present disclosure will be described in detail below.

[0019] [First Embodiment] In the first embodiment, a case will be described in which a moving image is input to posture estimation apparatus 10 and a posture estimation result is output. Fig. 3 is a block diagram showing the hardware configuration of posture estimation apparatus 10 of this embodiment.

[0020] 3 , posture estimation device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0021] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, a posture estimation program is stored in the ROM 12 or the storage 14. The posture estimation program may be a single program, or may be a group of programs composed of multiple programs or modules.

[0022] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0023] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0024] The display unit 16 is, for example, a liquid crystal display, and displays various information including the processing results. The display unit 16 may be a touch panel type and function as the input unit 15.

[0025] The communication interface 17 is an interface for communicating with other devices, such as a camera that captures moving images and a database server that stores the moving images captured by the camera, etc. Standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used for this communication.

[0026] Next, a description will be given of the functional configuration of posture estimation apparatus 10. Fig. 4A is a block diagram showing an example of the functional configuration of posture estimation apparatus 10.

[0027] As shown in FIG. 4A , posture estimation device 10 functionally includes an acquisition unit 100, an allocation control unit 102, a division processing unit 104, an overall processing unit 106, a motion estimation unit 108, a result storage unit 110, a coordinate correction unit 112, a correction deviation determination unit 114, a selection unit 116, and a synthesis processing unit 118.

[0028] The acquisition unit 100 acquires each frame image of the input moving image.

[0029] The allocation control unit 102 allocates, for each frame image, a frequency of execution of an inference process for estimating the posture of an object having a joint to each partial surface obtained by dividing the frame image.

[0030] Specifically, the allocation control unit 102 allocates the frequency of executing the inference process for each partial surface according to the determination result by the correction deviation determination unit 114 (FIG. 4B).

[0031] The division processing unit 104 divides each frame image and performs inference processing on each partial plane according to the assigned frequency.

[0032] As shown in FIG. 5, the division processing unit 104 includes a division unit 120, a partial surface selection unit 122, and an inference processing unit .

[0033] The dividing unit 120 divides each frame image of the moving image and outputs it as partial planes, which are a plurality of partial images.

[0034] The partial surface selection unit 122 selects a partial surface for inference processing for each frame image in accordance with the allocation by the allocation control unit 102 .

[0035] The inference processing unit 124 uses a posture estimation model, which is a convolutional neural network that receives an image as input and outputs skeletal information of an object, to perform posture estimation to determine skeletal information of the object for each of the selected partial surfaces, and outputs the positions of each joint of each object included in the partial surface.

[0036] Specifically, the pose estimation model is a trained model that has been trained in advance to output, for each object included in an input image, skeletal information including the positions of each joint of the object.

[0037] The posture estimation model may be stored in a storage medium such as the storage 14, or may be stored in an external cloud server. As a technique for performing posture estimation based on such images, known methods can be appropriately applied.

[0038] The overall processing unit 106 reduces each frame image to an overall surface that shows the entire image, and performs inference processing on the overall surface.

[0039] As shown in FIG. 6, the overall processing unit 106 includes a reduction unit 130 and an inference processing unit 132 .

[0040] The reduction unit 131 reduces the entire image for each frame image of the moving image.

[0041] The inference processing unit 132 performs inference processing for each frame image of the moving image from the reduced image, in the same manner as the inference processing unit 124 .

[0042] The motion estimation unit 108 calculates the motion vector of an object for the entire frame image of each frame image of the video image. Specifically, the motion estimation unit 108 calculates the motion vector of the object for each position in the frame image.

[0043] The result storage unit 110 stores the pose estimation results for each partial surface by the inference processing unit 124 of the segmentation processing unit 104 for past frame images.

[0044] The coordinate corrector 112 corrects the result of the inference process in the previous frame for the partial surface on which the inference process is not executed, based on the motion vector calculated by the motion estimator 108 .

[0045] Specifically, the coordinate correction unit 112 performs coordinate correction on the posture estimation result in the previous frame image for the partial surface on which the inference process is not performed, based on the motion vector calculated by the motion estimation unit 108, and obtains the posture estimation result of the object in the current frame image.

[0046] More specifically, for each partial surface on which inference processing is not performed, the coordinate correction unit 112 corrects the position of each joint of each object included in the posture estimation result obtained for that partial surface in the previous frame image based on the motion vector calculated for the corresponding position.

[0047] The correction deviation determination unit 114 determines, for each partial surface, a correction deviation, which is a deviation between the result of correction by the coordinate correction unit 112 and the actual posture of the object.

[0048] Specifically, the correction deviation determination unit 114 determines, for each partial surface, the correction deviation which is the deviation between the result of correction by the coordinate correction unit 112 and the posture of the object determined in advance, and the correction deviation which is the deviation between the result of correction by the coordinate correction unit 112 and the posture of the object determined in the previous frame.

[0049] Here, a method for determining the first phenomenon of the corrected deviation, which is the deviation between the result of correction by the coordinate corrector 112 and the posture of the object obtained in advance, will be described (FIG. 7). If the distance between each joint point is longer than a certain ratio with respect to the standard distance estimated from the standard ratio to the overall size set as shown in Table 1, it is determined to correspond to the first phenomenon.

[0050]

[0051] Specifically, it is determined that the first phenomenon occurs when the following logical expression is satisfied:

[0052]

[0053] Here, B represents a set of pairs (i, j) of two articulation points that are in a connection relationship. ｉ，ｊ represents the distance between joint point i and joint point j in the posture estimation result. L represents the length of the long side of the frame surrounding the posture estimation result, and R ｉ，ｊ represents the standard ratio of the distance between joint point i and joint point j to L. The standard ratio is set in advance for each side, which is a pair of joint points, as shown in Table 1. α represents a magnification that determines how much longer the distance between joint points must be compared to the standard distance before it is determined to correspond to the first phenomenon. The magnification is set in advance.

[0054] Next, a method for determining the second phenomenon of correction deviation, which is a deviation between the correction result by the coordinate correction unit 112 and the object posture obtained in the previous frame, will be described ( FIG. 8 ). For each joint point of a person, a vector connecting that joint point in the posture estimation result for the previous frame and that joint point in the posture estimation result for the current frame is calculated. If the average of the dot products of the vectors is equal to or less than a certain value, it is determined that the person has slid overall compared to the previous frame, and the second phenomenon is determined to occur. Here, the dot product of the vectors becomes smaller the closer the angle between the vectors is to 0°. The posture estimation result for the previous frame may be obtained by performing inference or by correcting using a motion vector. In this method, since it is necessary to identify the same person between frames, a person ID is assigned to and managed in the posture estimation result. The assignment of the person ID does not change between frames when correcting using a motion vector, but it may change between frames when obtaining the posture estimation result by performing inference. Therefore, if the posture estimation result for the previous frame was obtained by executing inference, it is necessary to identify the same person between frames using another method; however, many such methods have been proposed in the field of object tracking, and the technology disclosed herein does not consider the means to be so.

[0055] 9A , the frames that are input when determining a correction error will be described. When determining the first phenomenon of the correction error for the current frame (Nth frame), only the posture estimation result for the current frame is input. On the other hand, when determining the second phenomenon of the correction error for the current frame, the posture estimation results for the previous frame (N−1th frame) and the posture estimation results for the current frame (Nth frame) are input.

[0056] Therefore, as shown in FIG. 9B , in the first mode for determining the first and second phenomena, the posture estimation results of the previous frame (the N-1th frame) and the posture estimation results of the current frame (the Nth frame) are input.

[0057] The selection unit 116 selects, for each partial surface of each frame image, either the result of inference performed on that partial surface or the result of coordinate correction based on the motion vector.

[0058] The synthesis processing unit 118 synthesizes the posture estimation result of the division processing unit 104 and the posture estimation result of the overall processing unit 106 for each frame image to obtain a final posture estimation result.

[0059] Specifically, the synthesis processing unit 118 synthesizes the results of pose estimation for the partial surface and the entire surface, and estimates the pose of the object from the frame images.

[0060] For example, for each object included in a frame image, the synthesis processing unit 118 estimates the posture of each object from the frame image by adopting the posture estimation result of the entire surface if the posture estimation result of the object is included in the posture estimation result of the entire surface, and adopting the posture estimation result of the partial surface if the posture estimation result of the object is not included in the posture estimation result of the entire surface.

[0061] Next, allocation control by allocation control unit 102 will be described in detail.

[0062] The technology disclosed herein can use one of two allocation control methods. The first is a method of allocating all partial planes together, and the second is a method of allocating each partial plane independently. In either case, the initial allocation is arbitrary.

[0063] First, the first method will be described (FIG. 10). A correction deviation determination is performed for each object for all partial surfaces at a fixed cycle, and the number of objects determined to have a correction deviation is counted for each partial surface. When counting for all partial surfaces is completed, an inference execution allocation for each partial surface in the next cycle is set so that the inference execution frequency for partial surfaces with a large number of objects determined to have a correction deviation increases. For example, allocation may be based on the ratio of the number of objects determined to have a correction deviation, but the specific allocation method is not limited.

[0064] Figure 10 shows an example in which a correction deviation determination is performed for all four partial surfaces every 10 frames, and an inference execution allocation is set so that the inference execution frequency for the 0th partial surface and the 1st partial surface that are determined to have a correction deviation is increased.

[0065] Next, the second allocation control method will be described ( FIG. 11 ). For each partial surface, after the frame in which pose estimation by inference execution was performed, pose estimation by correction using motion vectors is performed for a certain number of frames, and a correction deviation judgment is performed when a certain number of frames have passed since the inference execution. If the number of objects judged to have a correction deviation is equal to or greater than a certain number, it is judged that a correction deviation exists for that partial surface. If it is judged that a correction deviation exists, the partial surface is assigned to immediately execute inference in the next frame. If inference cannot be executed immediately, it is managed using a queue, etc., and the inference process is executed in order when it becomes possible to execute it. If the timing of judgments of the existence of a correction deviation overlaps, the one with the larger number of objects judged to have a correction deviation is assigned to execute inference first. Furthermore, even if it is not judged that a correction deviation exists, the inference process is assigned to be executed after a certain number of frames have passed.

[0066] 11 shows an example in which a correction deviation determination is performed for each partial surface, and immediately after it is determined that there is a correction deviation for the 0th partial surface, an inference process is performed for the 0th partial surface. Also, an example in which immediately after it is determined that there is a correction deviation for the 2nd and 3rd partial surfaces, an inference process is performed for the 2nd and 3rd partial surfaces in that order.

[0067] Also, for the first partial surface that is not determined to have a correction deviation, an example is shown in which the inference process is assigned to be executed after 22 frames have elapsed.

[0068] <Operation of Posture Estimation Device According to First Embodiment> Next, the operation of the posture estimation device 10 according to the first embodiment will be described.

[0069] 12 is a flowchart showing the flow of posture estimation processing by posture estimation device 10. The posture estimation processing is performed by CPU 11 reading a posture estimation program from ROM 12 or storage 14, expanding it into RAM 13, and executing it. It is also assumed that a moving image has been input to posture estimation device 10. The posture estimation processing is an example of a posture estimation method.

[0070] In step S100, the CPU 11 functions as the acquisition unit 100 to acquire frame images of the input moving image.

[0071] In step S102, the CPU 11 functions as the division processing unit 104 to divide the acquired frame image and perform posture estimation.

[0072] In step S104, the CPU 11 functions as the overall processing unit 106 to reduce the entire acquired frame image and perform posture estimation.

[0073] In step S106, the CPU 11 functions as the selection unit 116 to select, for each partial surface of the acquired frame image, either the result of inference for that partial surface or the result of coordinate correction based on the motion vector.

[0074] In step S108, the CPU 11 functions as the synthesis processing unit 118 to synthesize the posture estimation result of the division processing unit 104 and the posture estimation result of the overall processing unit 106 to obtain a final posture estimation result.

[0075] In step S110, the CPU 11 outputs the final posture estimation result obtained for the acquired frame image on the display unit 16.

[0076] In step S102, the CPU 11 functions as the division unit 120 and repeatedly executes the division processing routine shown in FIG. 13 for each partial plane, which is a plurality of partial images obtained by dividing the frame image.

[0077] In step S120, the CPU 11 acquires a partial surface to be processed.

[0078] In step S122, the CPU 11, functioning as the correction deviation determination unit 114, determines whether the partial surface to be processed is a target for correction deviation determination, in accordance with the allocation by the allocation control unit 102. If the partial surface to be processed is a target for correction deviation determination, the process proceeds to step S124, whereas if the partial surface to be processed is not a target for correction deviation determination, the process proceeds to step S128.

[0079] In step S124, the CPU 11, as the correction deviation determination unit 114, performs a correction deviation determination for each object on the partial surface to be processed to determine whether it is the first phenomenon or not, and a correction deviation determination for each object on the partial surface to be processed to determine whether it is the second phenomenon or not, and counts the number of objects that are the first phenomenon or the second phenomenon.

[0080] In step S126, the CPU 11 functions as the correction deviation determination unit 114 to output the number of objects that are the first phenomenon or the second phenomenon to the allocation control unit 102, and then the process proceeds to step S138.

[0081] In step S128, the CPU 11, functioning as the partial surface selection unit 122, determines whether the partial surface to be processed is a target for executing inference processing, in accordance with the allocation by the allocation control unit 102. If the partial surface to be processed is a target for executing inference processing, the process proceeds to step S130. On the other hand, if the partial surface to be processed is not a target for executing inference processing, the process proceeds to step S132.

[0082] In step S130, the CPU 11 functions as the inference processing unit 124 to execute inference processing on the partial surface to be processed, and then the process proceeds to step S138.

[0083] In step S132, the CPU 11, functioning as the coordinate correction unit 112, determines whether the partial surface to be processed is a target for coordinate correction based on a motion vector, in accordance with the allocation by the allocation control unit 102. If the partial surface to be processed is a target for coordinate correction based on a motion vector, the process proceeds to step S134; on the other hand, if the partial surface to be processed is not a target for coordinate correction based on a motion vector, the division processing routine is terminated.

[0084] In step S134, the CPU 11 functions as the motion estimation unit 108 to calculate the motion vector of the object for the entire frame image.

[0085] In step S136, the CPU 11, as the coordinate correction unit 112, performs coordinate correction on the posture estimation result in the previous frame image for the partial surface to be processed, based on the motion vector calculated by the motion estimation unit 108, obtains the posture estimation result of the object in the current frame image, and proceeds to step S138.

[0086] In step S138, the CPU 11 outputs the processing result of step S126, S130, or S136 to the selection unit 116 and the result storage unit 110.

[0087] In step S140, the CPU 11, functioning as the allocation control unit 102, determines whether or not it is time to change the allocation for the partial face to be processed. If it is time to change the allocation for the partial face to be processed, the process proceeds to step S142. On the other hand, if it is not time to change the allocation for the partial face to be processed, the division processing routine is terminated.

[0088] In step S142, the CPU 11 functions as the allocation control unit 102 to change the allocation of the frequency of execution of the inference process for the partial surface to be processed in accordance with the determination result by the correction deviation determination unit 114, and ends the division process routine.

[0089] As described above, the posture estimation process shown in Fig. 12 and the process shown in Fig. 13 are repeatedly executed, and therefore, when the process is to be terminated, it is terminated by an interrupt process. In this case, termination process may be performed such that the process is terminated after all processes in the cycle at the time the interrupt is entered are completed.

[0090] As described above, the posture estimation device according to the first embodiment determines, for each partial surface, a correction error, which is the error between the result of correction by the coordinate correction unit and the actual posture of the object, and assigns a frequency for executing the inference process according to the result of the correction error determination for each partial surface. This reduces the amount of calculation and enables accurate posture estimation. In particular, posture estimation using CNN inference can be achieved with a small amount of calculation and high accuracy even for high-definition images (e.g., full HD (High Definition), 4K) that exceed the constraint size.

[0091] Furthermore, when determining a correction error for the first phenomenon, only the posture estimation result for the current frame is input, so that the correction error can be determined with a small processing load. When determining a correction error for the second phenomenon, the posture estimation result for the previous frame (the N-1th frame) and the posture estimation result for the current frame (the Nth frame) are input. Because the correspondence between objects and the correspondence between joints between frames are known, the correction error can be determined with a relatively small processing load.

[0092] Second Embodiment Next, a posture estimation device according to a second embodiment will be described. Note that parts having the same configuration as those in the first embodiment will be assigned the same reference numerals and descriptions thereof will be omitted.

[0093] The second embodiment differs from the first embodiment in that a correction deviation determination is performed in a second mode for determining the second phenomenon and the third phenomenon.

[0094] The correction deviation determination unit 114 of the posture estimation device 10 according to the second embodiment determines, for each partial surface, a correction deviation that is the deviation between the result of correction by the coordinate correction unit 112 and the posture of the object determined in the previous frame, and a correction deviation that is the deviation between the result of correction by the coordinate correction unit 112 and the posture of the object obtained from the result of the inference processing in the current frame.

[0095] First, a method for determining the third phenomenon of correction deviation, which is a deviation between the result of correction by the coordinate correction unit 112 and the posture of the object obtained from the result of inference processing in the current frame, will be described (FIG. 14).

[0096] For the current frame, the posture estimation result is calculated by both executing the inference process and correcting the estimation result of the previous frame using a motion vector.Then, the difference sum of the position coordinates of each joint point in these results is calculated, and if it is equal to or greater than a certain value, it is determined that the third phenomenon applies.

[0097] As with the determination of the second phenomenon, this method requires identifying the same person between frames, so a person ID is assigned to the estimation result and managed. The assignment of person IDs does not change between frames when correcting with a motion vector, but may change between frames when performing inference processing to obtain pose estimation results. Therefore, it is necessary to identify the same person between frames using a different method, but it is sufficient to identify the same person between frames using a conventionally known method used in object tracking, and the technique disclosed herein does not matter how this method is used.

[0098] Furthermore, as shown in FIG. 9A above, when determining the third phenomenon of correction deviation for the current frame (Nth frame), the execution result of the inference process for the current frame and the correction result based on the motion vector for the posture estimation result of the previous frame (N-1th frame) are input.

[0099] Therefore, as shown in FIG. 9B above, in the second mode for determining the second and third phenomena, the posture estimation result of the previous frame (the N-1th frame) and the posture estimation result of the current frame (the Nth frame) (the execution result of the inference process for the current frame and the correction result based on the motion vector for the posture estimation result of the previous frame (the N-1th frame)) are input.

[0100] The configuration of the posture estimation device 10 according to the second embodiment is similar to that of the posture estimation device 10 according to the first embodiment, but as shown in FIG. 15 , the result of the inference processing is transmitted from the division processing unit 104 to the correction deviation determination unit 114.

[0101] <Operation of Posture Estimation Device According to Second Embodiment> Next, the operation of the posture estimation device 10 according to the second embodiment will be described.

[0102] When a moving image is input to posture estimation device 10, posture estimation device 10 performs the posture estimation process shown in FIG. 12, as in the first embodiment.

[0103] In step S102, as in the first embodiment, the CPU 11, as the dividing unit 120, repeatedly executes the division processing routine shown in Fig. 13 for each partial surface, which is a plurality of partial images obtained by dividing the frame image. At this time, in step S124, the CPU 11, as the correction deviation determination unit 114, performs correction deviation determination for each object on the partial surface to be processed, to determine whether or not it is the second phenomenon, and whether or not it is the third phenomenon, and counts the number of objects that are the second phenomenon or the third phenomenon.

[0104] The other configurations and operations of the posture estimation device according to the second embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0105] As described above, the posture estimation device according to the second embodiment determines, for each partial surface, the deviation between the result of correction by the coordinate correction unit and the actual posture of the object, and assigns the frequency of execution of the inference process to each partial surface according to the result of the deviation correction. This reduces the amount of calculation and enables accurate posture estimation.

[0106] Furthermore, in determining the correction error of the second phenomenon, the posture estimation result of the previous frame (the N-1th frame) and the posture estimation result of the current frame (the Nth frame) are input. Because the correspondence between objects and the correspondence between joints between frames are known, it is possible to determine the correction error with a relatively small processing load. Furthermore, in determining the correction error of the third phenomenon, it is possible to determine the correction error, which is the deviation between the result of correction by the coordinate correction unit 112 and the posture of the object obtained from the result of the inference process in the current frame, and therefore it is possible to determine the correction error with high accuracy.

[0107] Third Embodiment Next, a posture estimation device according to a third embodiment will be described. Note that parts having the same configuration as those in the first embodiment will be assigned the same reference numerals and descriptions thereof will be omitted.

[0108] The third embodiment differs from the first embodiment in that when the number of objects is large, the correction deviation judgment is performed using the first mode that judges the first phenomenon and the second phenomenon, and when the number of objects is small, the correction deviation judgment is performed using the second mode that judges the second phenomenon and the third phenomenon.

[0109] The functional configuration of posture estimation apparatus 200 according to the third embodiment will be described below. Fig. 16A is a block diagram showing an example of the functional configuration of posture estimation apparatus 200.

[0110] As shown in FIG. 16A , posture estimation device 200 functionally includes an acquisition unit 100, an allocation control unit 102, a division processing unit 104, an overall processing unit 106, a motion estimation unit 108, a result storage unit 110, a coordinate correction unit 112, a counting unit 213, a correction deviation determination unit 214, a selection unit 116, and a synthesis processing unit 118.

[0111] The counting unit 213 counts the number of specific objects for each partial surface. Specifically, the counting unit 213 performs object detection for each partial surface to detect people, and counts the number of people.

[0112] For each partial surface, depending on the number of specific objects, the correction deviation determination unit 214 switches between determining the deviation between the result of correction made by the coordinate correction unit 112 and the posture of the object determined in advance, and between the result of correction made by the coordinate correction unit 112 and the posture of the object determined in the previous frame, or determining the deviation between the result of correction made by the coordinate correction unit 112 and the posture of the object determined in the previous frame, and between the result of correction made by the coordinate correction unit 112 and the posture of the object obtained from the result of the inference processing in the current frame (Figure 16B).

[0113] Specifically, for each partial surface, when the number of specific objects counted on that partial surface is equal to or greater than a threshold value, the correction deviation judgment unit 214 performs a correction deviation judgment using a first mode for judging a first phenomenon and a second phenomenon, and when the number of specific objects counted on that partial surface is less than the threshold value, the correction deviation judgment is performed using a second mode for judging a second phenomenon and a third phenomenon.

[0114] <Operation of Posture Estimation Device According to Third Embodiment> Next, the operation of the posture estimation device 10 according to the third embodiment will be described.

[0115] When a moving image is input to posture estimation device 10, posture estimation device 10 performs the posture estimation process shown in FIG. 12, as in the first embodiment.

[0116] In step S102, the CPU 11 functions as the dividing section 120 to divide the frame image and repeatedly execute the division processing routine shown in FIG. 17 for each partial plane, which is a plurality of partial images.

[0117] In step S120, the CPU 11 acquires a partial surface to be processed.

[0118] In step S122, the CPU 11, functioning as the correction deviation determination unit 114, determines whether the partial surface to be processed is a target for correction deviation determination, in accordance with the allocation by the allocation control unit 102. If the partial surface to be processed is a target for correction deviation determination, the process proceeds to step S200, whereas if the partial surface to be processed is not a target for correction deviation determination, the process proceeds to step S128.

[0119] In step S200, the CPU 11, as the counting unit 213, performs object detection on the partial surface to be processed, counts the number of specific objects, and determines whether the number of specific objects is equal to or greater than a certain number. If the number of specific objects is equal to or greater than the certain number, the process proceeds to step S202, and if the number of specific objects is less than the certain number, the process proceeds to step S204.

[0120] In step S202, the CPU 11, as the correction deviation determination unit 214, performs a correction deviation determination for each object on the partial surface to be processed to determine whether it is the first phenomenon or not, and a correction deviation determination for each object to determine whether it is the second phenomenon or not, and counts the number of objects that are the first phenomenon or the second phenomenon.

[0121] In step S204, the CPU 11, as the correction deviation determination unit 214, performs a correction deviation determination for each object on the partial surface to be processed to determine whether it is the second phenomenon or not, and a correction deviation determination for each object on the partial surface to be processed to determine whether it is the third phenomenon or not, and counts the number of objects that are the second phenomenon or the third phenomenon.

[0122] In step S126, the CPU 11 functions as the correction deviation determination unit 114 and outputs the number of objects counted in step S202 or S204 to the allocation control unit 102, and then the process proceeds to step S138.

[0123] In step S128, the CPU 11, functioning as the partial surface selection unit 122, determines whether the partial surface to be processed is a target for executing inference processing, in accordance with the allocation by the allocation control unit 102. If the partial surface to be processed is a target for executing inference processing, the process proceeds to step S130. On the other hand, if the partial surface to be processed is not a target for executing inference processing, the process proceeds to step S132.

[0124] In step S130, the CPU 11 functions as the inference processing unit 124 to execute inference processing on the partial surface to be processed, and then the process proceeds to step S138.

[0125] In step S132, the CPU 11, functioning as the coordinate correction unit 112, determines whether the partial surface to be processed is a target for coordinate correction based on a motion vector, in accordance with the allocation by the allocation control unit 102. If the partial surface to be processed is a target for coordinate correction based on a motion vector, the process proceeds to step S134; on the other hand, if the partial surface to be processed is not a target for coordinate correction based on a motion vector, the division processing routine is terminated.

[0126] In step S134, the CPU 11 functions as the motion estimation unit 108 to calculate the motion vector of the object for the entire frame image.

[0127] In step S136, the CPU 11, as the coordinate correction unit 112, performs coordinate correction on the posture estimation result in the previous frame image for the partial surface to be processed, based on the motion vector calculated by the motion estimation unit 108, obtains the posture estimation result of the object in the current frame image, and proceeds to step S138.

[0128] In step S138, the CPU 11 outputs the processing result of step S126, S130, or S136 to the selection unit 116 and the result storage unit 110.

[0129] In step S140, the CPU 11, functioning as the allocation control unit 102, determines whether or not it is time to change the allocation for the partial face to be processed. If it is time to change the allocation for the partial face to be processed, the process proceeds to step S142. On the other hand, if it is not time to change the allocation for the partial face to be processed, the division processing routine is terminated.

[0130] In step S142, the CPU 11 functions as the allocation control unit 102 to change the allocation of the frequency of execution of the inference process for the partial surface to be processed in accordance with the determination result by the correction deviation determination unit 114, and ends the division process routine.

[0131] The other configurations and operations of posture estimation apparatus 200 according to the third embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0132] As described above, the posture estimation device according to the third embodiment determines, for each partial surface, the deviation between the result of correction by the coordinate correction unit and the actual posture of the object, and assigns the frequency of execution of the inference process to each partial surface according to the result of the deviation correction. This reduces the amount of calculation and enables accurate posture estimation.

[0133] Furthermore, when the number of objects is large, in order to reduce the processing load, a first mode is used in which a correction deviation determination as to whether or not the object is the first phenomenon and a correction deviation determination as to whether or not the object is the second phenomenon, which requires a low processing load. When the number of objects is small, a second mode is used in which a correction deviation determination as to whether or not the object is the second phenomenon and a correction deviation determination as to whether or not the object is the third phenomenon, which requires a relatively high processing load, in order to accurately determine the correction deviation. In this way, a correction deviation determination can be performed for each partial surface according to the number of objects.

[0134] In the third embodiment, the correction deviation determination mode is switched depending on the number of objects. However, the present invention is not limited to this. For example, the correction deviation determination mode may be switched between the first mode and the second mode by a user setting. For example, the user may set the first mode when it is desired to reduce the processing load, and may set the second mode when it is desired to prioritize the determination accuracy.

[0135] In the above embodiment, the absolute difference between the average number of detected objects over a certain period of time in the past and the number of detected objects in the current frame may be calculated for each partial surface, and the frequency of execution of the inference process may be assigned taking into consideration both the absolute difference in the number of detected objects and the determination result of the correction deviation. In this case, the frequency of execution of the inference process may be assigned by giving priority to partial surfaces with a larger absolute difference in the number of detected objects.

[0136] Although the example has been described in which a person is used as a specific object for estimating the posture of the person, the present invention is not limited to this, and posture estimation may be performed on any object other than a person as long as the object has joints. For example, posture estimation may be performed on objects such as animals and robots.

[0137] Furthermore, the various processes executed by the CPU after reading the software (program) in the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include dedicated electrical circuits, such as programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after manufacture, and application-specific integrated circuits (ASICs) that are processors having circuit configurations specifically designed to execute specific processes. Furthermore, the various processes may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0138] In addition, in each of the above embodiments, the posture estimation program is described as being pre-stored (installed) in storage 14, but this is not limiting. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0139] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and modified examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, unnecessary parts may be deleted, new elements may be added, or new elements may be replaced with other elements from the above-described description and illustrations, within the scope of the gist of the technology of the present disclosure.

[0140] The following additional notes are provided regarding the above-described embodiments.

[0141] (Supplementary Item 1) A posture estimation device that performs inference processing to estimate the posture of an object having joints from images included in a moving image, comprising: a memory; and at least one processor connected to the memory, wherein the processor is configured to: acquire the images from the moving image; assign a frequency of executing the inference processing to each partial surface obtained by dividing the image; perform the inference processing on the partial surface in accordance with the assigned frequency; reduce the image to a full surface that shows the entire image and perform the inference processing on the full surface; calculate a motion vector of the object; correct a result of the inference processing in the previous frame based on the motion vector for the partial surfaces on which the inference processing is not performed; combine the results of the inference processing for the partial surfaces and the full surface to estimate the posture of the object from the image; and determine a deviation between the corrected result and the posture of the object for each partial surface; and wherein the assigning the frequency assigns a frequency of executing the inference processing for each partial surface in accordance with the result of the deviation determination.

[0142] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to execute an inference process for estimating the posture of an object having joints from an image included in a moving image, wherein the posture estimation process includes: acquiring the image from the moving image; assigning a frequency of executing the inference process to each partial surface obtained by dividing the image; executing the inference process on the partial surface according to the assigned frequency; reducing the image to a full surface that shows the entire image and executing the inference process on the full surface; calculating a motion vector of the object; correcting a result of the inference process in a previous frame based on the motion vector for a partial surface on which the inference process is not executed; combining the results of the inference process for the partial surface and the full surface to estimate the posture of the object from the image; and determining a deviation between the corrected result and the posture of the object for each partial surface, wherein the assigning the frequency assigns a frequency of executing the inference process for each partial surface according to a result of determining the deviation.

[0143] 10, 200 Posture estimation device 11 CPU 14 Storage 15 Input unit 16 Display unit 100 Acquisition unit 102 Allocation control unit 104 Division processing unit 106 Overall processing unit 108 Motion estimation unit 110 Result storage unit 112 Coordinate correction unit 114, 214 Correction deviation determination unit 116 Selection unit 118 Synthesis processing unit 120 Division unit 122 Partial surface selection unit 124 Inference processing unit 130 Reduction unit 131 Reduction unit 132 Inference processing unit 213 Counting unit

Claims

1. A posture estimation device that performs inference processing to estimate the posture of an object having joints from images included in a video, comprising: an acquisition unit that acquires the images from the video; an allocation control unit that assigns a frequency of executing the inference processing to each partial surface obtained by dividing the image; a segmentation processing unit that performs the inference processing on the partial surfaces in accordance with the assigned frequency; an overall processing unit that reduces the image to an overall surface that represents the entire image and performs the inference processing on the overall surface; a motion estimation unit that calculates a motion vector of the object; a coordinate correction unit that corrects a result of the inference processing in a previous frame based on the motion vector for partial surfaces on which the inference processing is not performed; a synthesis processing unit that synthesizes the results of the inference processing for the partial surfaces and the overall surface to estimate the posture of the object from the image; and a correction deviation determination unit that determines, for each partial surface, a deviation between the result of correction by the coordinate correction unit and the actual posture of the object, wherein the allocation control unit assigns a frequency of executing the inference processing for each partial surface in accordance with the determination result by the correction deviation determination unit.

2. The posture estimation device of claim 1, wherein the correction deviation determination unit determines, for each partial surface, the deviation between the result of correction made by the coordinate correction unit and the posture of the object determined in advance, the deviation between the result of correction made by the coordinate correction unit and the posture of the object determined in the previous frame, or the deviation between the result of correction made by the coordinate correction unit and the posture of the object obtained from the result of the inference processing in the current frame.

3. The posture estimation device according to claim 2, further comprising a counting unit that counts the number of the objects for each of the partial surfaces, wherein the correction deviation determination unit switches, depending on the number of the objects for each of the partial surfaces, between determining the deviation between the result of correction by the coordinate correction unit and the posture of the object that was previously determined, and between the result of correction by the coordinate correction unit and the posture of the object that was determined in the previous frame, or determining the deviation between the result of correction by the coordinate correction unit and the posture of the object that was determined in the previous frame, and between the result of correction by the coordinate correction unit and the posture of the object obtained from the result of the inference processing in the current frame.

4. A posture estimation method for performing inference processing to estimate the posture of an object having joints from images included in a video, the method comprising: acquiring the images from the video; assigning a frequency for executing the inference processing to each partial surface obtained by dividing the image; performing the inference processing on the partial surface according to the assigned frequency; reducing the image to a full surface that shows the entire image and performing the inference processing on the full surface; calculating a motion vector of the object; correcting the result of the inference processing in the previous frame based on the motion vector for partial surfaces on which the inference processing is not performed; combining the results of the inference processing for the partial surfaces and the full surface to estimate the posture of the object from the image; and determining, for each partial surface, a deviation between the corrected result and the actual posture of the object; and the method for assigning the frequency allocates a frequency for executing the inference processing for each partial surface according to the result of determining the deviation.

Images