Video processing device, video processing method, and program

Abstract

A video processing device according to one aspect of the present invention comprises a two-dimensional pose information generation unit, a pose information processing unit, and a three-dimensionalizing processing unit. The two-dimensional pose information generation unit generates two-dimensional pose information of a photographic subject from video data of the photographic subject. The pose information processing unit corrects the two-dimensional pose information. The three-dimensionalizing processing unit generates three-dimensional pose information of the photographic subject from the corrected two-dimensional pose information. The pose information processing unit comprises a left / right flip determination unit that performs left / right flip determination processing. The left-right flip determination processing is processing in which a first pattern through a fourth pattern are set, said first pattern through fourth pattern corresponding to an upper body and a lower body being flipped or not being flipped in an image frame of the photographic subject in the video data, and the pattern with the lowest score among the first pattern through the fourth pattern is selected, said score being calculated using an evaluation formula in which the variance and the mean of distances of joint locations corresponding to the previous frame and the current frame are treated as variables.

Description

[Technical Field]

[0001] One aspect of the present invention relates to an image processing device for detecting, for example, a person's posture and creating a 3D (three-dimensional) wireframe model, an image processing method by a computer, and a program. [Background technology]

[0002] "Embodied knowledge" is expressed, for example, in a person's posture and movements as "skill." There has been active research into technology that uses video processing to quantify and record this kind of non-verbal, sensory-based information. If we can quantify human posture (hereafter referred to as "skill capture"), it will be possible to visualize it for analysis by instructors, for example, through post-mortem analysis and comparison. Furthermore, it is desirable to be able to estimate (acquire) human posture with high accuracy in order to effectively reproduce an instructor's techniques and efficiently teach them to students.

[0003] OpenPose is an open-source library that can detect skeletal coordinates from image data of a human subject and generate a wireframe model. A technique is known that uses this to estimate the position of each joint point (2D posture information) for each image frame and digitize the 2D posture (see, for example, Non-Patent Document 1). Techniques are also known for converting 2D (two-dimensional) posture information into three dimensions to obtain 3D (three-dimensional) posture information (see, for example, Non-Patent Documents 2 and 3). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh. "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields." In CVPR, 2017. [Non-patent document 2] Yuta Ohno et al., "Extending OpenPose for 3D Human Pose Estimation," Entertainment Computing Symposium (EC2017), September 2017 [Non-patent document 3] Kosuke Takahashi, Dan Mikami, Mariko Isogawa, Hideaki Kimata. "Human Pose As Calibration Pattern; 3D Human Pose Estimation With Multiple Unsynchronized and Uncalibrated Cameras." In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018, Salt Lake City, UT, USA, June 18-22, 2018. pages 1775-1782, IEEE Computer Society, 2018. [doi] Summary of the Invention [Problem to be solved by the invention]

[0005] In existing techniques for 3D reconstruction of subjects using posture estimation technology, left and right coordinates are sometimes inverted when estimating 2D posture (for example, skeletal coordinates related to the right arm are incorrectly estimated as skeletal coordinates related to the left arm). Using data with inverted left and right skeletal coordinates in 2D reduces the accuracy of estimating 3D posture information. This is undesirable because it can, for example, degrade the processing accuracy of skill capture. This invention was made in light of the above circumstances, and its purpose is to provide a technology that suppresses left-right inversion of skeletal coordinates in 2D posture information, thereby enabling accurate estimation of 3D posture information. [Means for solving the problem]

[0006] A video processing device according to one aspect of the present invention includes a 2D posture information generation unit, a posture information processing unit, and a 3D processing unit. The 2D posture information generation unit generates 2D posture information of a subject from video data of the subject. The posture information processing unit corrects the 2D posture information. The 3D processing unit generates 3D posture information of the subject from the corrected 2D posture information. The posture information processing unit includes a left-right flip determination unit that performs left-right flip determination processing. The left-right flip determination processing sets first to fourth patterns corresponding to whether or not the upper and lower bodies of the subject are flipped for image frames of the video data, and selects one of the first to fourth patterns that provides the smallest score calculated using an evaluation formula that uses the variance and average of the distances between corresponding joint positions in the previous frame and the current frame as variables. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to provide a technique that can suppress left-right reversal of skeletal coordinates in 2D posture information, thereby enabling accurate estimation of 3D posture information. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of a workflow for reproducing and imparting embodied knowledge. [Figure 2] FIG. 2 is a diagram showing an example of a wireframe model used in skill capture. [Figure 3] FIG. 3 is a diagram for explaining estimation of 3D posture information from 2D posture information. [Figure 4] FIG. 4 is a diagram for explaining left-right reversal. [Figure 5] FIG. 5 is a functional block diagram illustrating an example of a video processing device according to an embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a table stored in the 2D posture information 61. As shown in FIG. [Figure 7] FIG. 7 is a diagram showing an example of a table stored in the 3D posture information 62. As shown in FIG. [Figure 8] FIG. 8 is a diagram showing the flow of data between the functional blocks shown in FIG. [Figure 9] FIG. 9 is a diagram for explaining an outline of the [basic processing]. [Figure 10] FIG. 10 is a diagram for explaining the [Correction 2] process. [Figure 11] FIG. 11 is a diagram for explaining the process of [Correction 3]. [Figure 12] FIG. 12 is a diagram for explaining the details of the [Correction 3] process. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Figure 1 shows an example of a workflow for reproducing and imparting embodied knowledge (including simulated experiences). Embodied knowledge is effectively utilized by analyzing skills captured by cameras and sensors, comparing video data of an expert with that of a trainee, for example, to extract areas for improvement, and providing feedback in an appropriate manner.

[0010] Skill capture is a technology that uses multiple cameras installed without any special care to simultaneously acquire items such as camera parameters (position, orientation, field of view, distortion), tracking of each subject, and the 3D posture of each subject (3D coordinates of joints) without any camera calibration.

[0011] Skill capture involves a series of processes: 2D pose estimation from video data, digitization process, rotation angle fitting, and rotation angle denoising. In 2D posture estimation, each frame of the video is input and the posture of the feature points (2D skeletal coordinates, etc.) is estimated. It can also be used for multiple people. In the digitization process, each object is separated and tracked, and the 3D skeletal coordinates of the skeleton are estimated, and the camera parameters are also estimated at the same time. Rotation angle fitting and denoising involves converting from 3D coordinates to a 3D rotation angle model, which is then further denoised.

[0012] To more effectively reproduce and transmit embodied knowledge, it is desirable to refine each element shown in Figure 1. Here, the "skills" to be analyzed include physical movements (including fingertips), physiological information, physiological responses, psychological states, etc. The scope of the analysis can be further expanded.

[0013] Figure 2 shows an example of a wireframe model used in skill capture. In the wireframe model, a number (part ID) is assigned to each body part, such as a joint. By acquiring the coordinates of each part of the wireframe model, posture information can be expressed in two dimensions.

[0014] 3 is a diagram for explaining estimation of 3D pose information from 2D pose information. 2D pose information is acquired by multiple cameras (camera 1, camera 2) installed at different viewpoints. Here, the multiple cameras are synchronized with each other and each acquires video data of a subject (such as a person).

[0015] 2D posture information is 2D coordinates that represent the joint positions of the subject in the nth frame of each camera image. For example, it can be expressed as a set of four-vectors (camera ID, part ID, part x coordinate, part y coordinate). In FIG. 3, if the right wrist is shown in the frame of camera 2, the 2D posture information is expressed as (camera ID=2, part ID=4, part x coordinate=30, part y coordinate=100), for example.

[0016] The 3D posture information is 3D coordinates that represent joint positions estimated in 3D from the 2D posture information of multiple cameras in the nth frame of each camera image. For example, it can be expressed as a set of four-dimensional vectors (part ID, part x coordinate, part y coordinate, part z coordinate). In the example of FIG. 3, the 3D posture information is expressed as (part ID=4, part x coordinate=50, part y coordinate=300, part z coordinate=150). Note that although two cameras are shown in FIG. 3, the same applies to the case of three or more cameras.

[0017] FIG. 4 is a diagram for explaining left-right reversal. In FIG. 4(a), it can be seen that the left and right correspondences of the upper body parts of the subject are reversed. This state is called "left-right reversal." On the other hand, in FIG. 4(b), the right hand corresponds correctly to the right hand and the left hand corresponds correctly to the left hand, and left-right reversal has not occurred.

[0018] Left-right reversal occurs due to a recognition error during image processing, etc., and can occur in either the upper or lower body. In this embodiment, four combinations of cases are considered: (upper body, no reversal), (upper body, reversal), (lower body, no reversal), and (lower body, reversal).

[0019] In this way, in the skill capture process, the posture of feature points (such as 2D skeletal coordinates) is estimated using each frame of video as input. At this time, the 2D skeletal coordinates may be inverted left and right. This is undesirable because it reduces the accuracy of the 3D skeletal coordinates estimated from the 2D skeletal coordinates. When estimating the posture of 3D skeletal coordinates, existing technologies do not include a determination algorithm that solves the left-right inversion problem for the input 2D posture information.

[0020] Therefore, in this embodiment, when estimating 3D skeletal coordinates from 2D skeletal coordinates, the presence or absence of left-right flipping is determined for each frame / subject ID, and correction processing is performed based on the result. The following explanation will explain [basic processing] that determines the presence or absence of left-right flipping, and multiple correction processing distinguished as [correction 1], [correction 2], and [correction 3].

[0021] Here, in the first frame of video data, past 3D skeletal coordinates cannot be referenced. However, from the next frame onwards, past 3D skeletal coordinates can be referenced (Case 1). In other words, it is possible to track the subject between frames (a predicted subject for the current (n)th frame can be generated from the past (n-1)th frame). Therefore, in this embodiment, (Case 1) will be described in detail.

[0022] [Basic processing] Assuming that the predicted subject is correct (however, this will be corrected in a post-process), a pattern with the smallest variance and average of joint position distances from the current frame (four patterns, with or without inversion of the upper / lower body) is selected.

[0023] (intra-frame correction) (1-1) [Correction 1] Determine and correct the inversion / non-inversion pattern based on the number of cameras. (1-2) [Correction 2] In addition to the basic processing, correction 1 is performed twice (the predicted subject for the second time is generated from the result of the first time. This is called two-stage left-right inversion determination).

[0024] (Inter-frame correction) (1-3) [Correction 3] Determine whether or not there is left-right reversal between the current frame and the previous frame (inter-frame left-right determination), and perform correction.

[0025] 5 is a functional block diagram showing an example of a video processing device according to an embodiment. The video processing device 10 is an information processing device (computer) including a processor 20 and a memory 90. In addition, the video processing device 10 includes a storage 60 and an interface unit 70 connected to a plurality of cameras 1-1 to 1-n. The units of the video processing device 10 are connected via a bus 80.

[0026] The storage 60 stores 2D posture information 61, 3D posture information 62, and a program 63. The program 63 is loaded into the memory 90 by the OS (Operating System) of the video processing device 10 and executed by the processor 20. The program 63 causes the processor 20 to function as a 2D posture information generation unit 30, a posture information processing unit 40, and a 3D processing unit 50. The posture information processing unit 40 includes a left-right inversion determination unit 41 and a left-right inversion correction processing unit 42 as processing routines.

[0027] The 2D posture information generating unit 30 generates 2D posture information of a subject from video data of the subject. The posture information processing unit 40 corrects the 2D posture information generated by the 2D posture information generating unit 30. The 3D processing unit 50 generates 3D posture information of the subject from the 2D posture information corrected by the posture information processing unit 40.

[0028] The left-right inversion determination unit 41 sets four patterns for the image frames of the subject in the video data of the subject: [As detected], [Upper body left-right inversion], [Lower body left-right inversion], and [Full body left-right inversion]. These patterns are combinations of four cases: (Upper body, no inversion), (Upper body, inversion), (Lower body, no inversion), and (Lower body, inversion). In other words, these patterns correspond to whether the upper body and lower body are inverted or not.

[0029] Then, the left-right reversal determining unit 41 selects, from these four patterns, the pattern that provides the smallest score calculated using an evaluation formula that uses the variance and average of the distances between the corresponding joint positions in the previous frame and the current frame as variables.

[0030] The left-right inversion correction processing unit 42 performs left-right inversion determination processing within a frame. Alternatively, the left-right reversal correction processing unit 42 performs two-stage left-right reversal determination processing within a frame. Alternatively, the left-right reversal correction processing unit 42 performs left-right reversal determination processing between successive frames. Here, the left-right reversal correction processing performed by the left-right reversal correction processing unit 42 is similar to the processing performed by the left-right reversal determination processing unit 41 .

[0031] 6 is a diagram showing an example of a table stored in the 2D posture information 61. The 2D posture information 61 is a table having columns of subject ID, camera ID, part ID, part x coordinate, and part y coordinate.

[0032] 7 is a diagram showing an example of a table stored in the 3D posture information 62. The 3D posture information 62 is a table with columns of subject ID, part ID, part x coordinate, part y coordinate, and part z coordinate.

[0033] Fig. 8 is a diagram showing the flow of data between the functional blocks shown in Fig. 6. Fig. 8 shows the case of camera parameter pre-calibration. In Fig. 8, multi-viewpoint video data from cameras 1-1 to 1-n is passed to a current frame extraction unit 21 together with the camera ID.

[0034] The current frame extraction unit 21 extracts the n-th frame image for each camera ID from the video data and passes it to the 2D posture information generation unit 30 . The 2D orientation information generation unit 30 calculates 2D orientation information for the n-th frame image and passes it to a left-right reversal determination unit 41 of the orientation information processing unit 40. The 2D orientation information is also stored in a 2D orientation information storage unit of the orientation information processing unit 40.

[0035] In the posture information processing unit 40, a left-right inversion determination unit 41 performs "basic processing," and a left-right inversion correction processing unit 42 performs "correction 1," "correction 2," and "correction 3" to calculate 3D posture information. The 3D posture information is output to the outside and stored in a 3D posture information storage unit.

[0036] [Basic processing] Fig. 9 is a diagram for explaining the outline of the [basic processing]. The [basic processing] is based on the idea that "in the current frame, a predicted subject is projected for each camera, and if the distance between this predicted subject and the 2D posture information of the current frame is large, left-right flip processing is performed."

[0037] To deal with left-right reversal, the basic process involves projecting a predicted 3D object onto each camera and matching it with the detected object for each camera. In other words, matching is performed on a 2D plane on each camera image. Assume that the 3D object (predicted 3D object) generated from the past (n-1) frame is left-right correct (no flipping). Then, for one of the 2D objects (detected object) detected from the current (n) frame, four patterns are generated: [As detected], [Upper body flipped left-right], [Lower body flipped left-right], and [Full body flipped left-right]. Then, select the pattern with the smallest variance or average of the joint position distances (errors) from the predicted object (projected from the predicted 3D object onto each camera). This is because if the left and right are correct, the average distances for all joints should be similar and the variance should be small. It is particularly effective to focus on the variance.

[0038] For example, the evaluation value can be calculated using equation (1).

number

[0039] In equation (1), α is a parameter, which is set to, for example, 1. Using equation (1), the evaluation value of each of the four patterns is calculated, and the pattern with the smallest evaluation value (score of each pattern) is selected. For example, if the upper body is reversed left to right, the score as detected will be large, and the score for the upper body being reversed left to right will be smaller than this.

[0040] [Regarding Amendment 1] [Correction 1] is essentially a process for determining left-right inversion within a frame. The predicted subject may be incorrect. However, if the frequency of left-right inversion of 2D skeletal information is low, when there are many inverted patterns (per camera / per subject), there is a high possibility that the selection of the left-right inversion pattern has failed. Therefore, in [Correction 1], if there are many cameras with inversion, the result is not adopted (no correction is made) as it is an incorrect determination.

[0041] If there are many inverted cameras, it is determined that the prediction is incorrect and the detection is used as is. In other words, the detection is used as is = n1 cameras, upper body left-right inversion = n2 cameras, lower body left-right inversion = n3 cameras, whole body left-right inversion = n4 cameras, and the total number of cameras N is N = n1 + n2 + n3 + n4. In this case, if n2 + n3 + n4 > N / 2, a correction is made to return all cameras to "as detected."

[0042] [Regarding Amendment 2] 10 is a diagram for explaining the process of [Correction 2]. [Correction 2] is, so to speak, a two-stage left-right inversion determination process within a frame. If the frequency of left-right inversion of 2D skeleton coordinates is low and there are few errors in the left-right inversion judgment in [Correction 1], the 3D posture information (estimated 3D object) estimated from the predicted 3D posture information (predicted 3D object = 3D object one frame before) is likely to be close to the true coordinates. Therefore, in [Correction 2], the basic process + [Correction 1] is repeated once more to get closer to the true coordinates.

[0043] (a) The estimated 3D object obtained in the first stage is input to the second stage as a predicted 3D object. (b) In the second stage, left-right inversion is determined using the estimated 3D coordinates and the detected subject, and a second estimated value of the 3D coordinates is obtained.

[0044] [Regarding Amendment 3] FIG. 11 is a diagram for explaining the process of Correction 3. In FIG. 11, for simplicity of explanation, the following explanation focuses on left-right reversal of only the upper body. In reality, the same process is performed on the lower body. If (1) in FIG. 11 there is no reversal in the past frame and (2) there is reversal between frames, there is a high possibility that there is reversal in the current frame (nth frame).

[0045] [Correction 3] is, in other words, a process for determining left-right inversion between frames. By introducing logic to detect left-right flipping using 2D posture information between frames, it is possible to predict left-right flipping in the current frame. For example, if there is no flipping in the previous frame but there is flipping between frames, there is a high possibility that there will be flipping in the current frame. Therefore, we estimate (1) and (2) by comparing the following: (1) The left-right inversion status of a previous frame can be estimated by comparing the 3D posture information with the 2D posture information of the previous frame. (2) By comparing the 2D posture information of the previous frame with the 2D posture information of the current frame (inter-frame comparison), it is possible to estimate whether the 2D posture information is horizontally reversed between the current frame and the previous frame.

[0046] FIG. 12 is a diagram for explaining the details of the process of Correction 3. The inter-frame left-right inversion determination process will be explained in detail with reference to FIG. 12. Below, the inter-frame left-right inversion determination process for the current frame using inter-frame comparison will be explained. For simplicity, two patterns will be explained, one with inversion of the upper body and one without inversion. In reality, the same process is also performed for the lower body.

[0047] By comparing Figure 12(a) and Figure 12(b), it is possible to predict whether the image will be flipped horizontally in the current frame. In other words, if the "prediction of whether the image will be flipped horizontally between the previous frame and the current frame (=d=a+b)" matches the "presence of whether the image will be flipped horizontally in the current frame (=c)," the result of c is considered reliable. In this case, a reliable pattern is more likely to be adopted by lowering its score.

[0048] [1] Left-right inversion of past frames (Fig. 12(a)) The 3D object estimated in the previous frame is projected onto the screen and compared with the 2D pose information from the previous frame. This comparison determines whether the target camera and object in the previous frame are left-right reversed. To determine left-right reversal, left-right reversal patterns are generated and the pattern with the smallest positional distance between the projected object and the joints is selected. The left-right reversal estimation at this point is not a prediction based on movement from previous frames, but rather a 3D coordinate estimated from the observation points of multiple cameras, so it is thought to be more accurate than one that uses predictions.

[0049] [2] Frame-to-frame left-right flip determination (Fig. 12(a)) The 2D posture information of the previous frame is compared with the 2D posture information of the current frame to determine whether left-right flipping has occurred between frames for the target camera and subject. The difference in joint position between the previous frame and the current frame is calculated, and if the difference in joint position between the right and left halves of the body exceeds the respective thresholds, it is determined that left-right flipping has occurred between frames. This is based on the assumption that if there is no left-right flipping, the joint positions will not change significantly. The joint position difference is scaled according to the size of the subject displayed on the screen, as shown in the formula below. Threshold d th is set to 0.8, and when the condition of equation (2) is satisfied for both the right and left halves of the body, the left and right sides are flipped between frames.

[0050]

number

[0051] In equation (2), length indicates the two-dimensional length on the screen when the three-dimensional length between the hip joints of the target subject is displayed on the screen. pprevious indicates the joint position in the previous frame. pcurrent indicates the joint position in the current frame. avg() is a function that calculates the average.

[0052] [3] Calculate the left-right inversion pattern score for the current frame (Fig. 13(c)) The current frame left-right inversion determination is the same as the left-right inversion pattern comparison method, using the predicted 3D subject and 2D posture information of the current frame, and calculating the score for each inversion pattern using equation (1).

[0053] [4] Current frame horizontal flip determination (Fig. 13(d)) Predict the horizontal flip of the current frame from the results of [1] and [2]. If either [1] or [2] determines that flipping occurs, predict that there is horizontal flipping. If both [1] and [2] are not flipped or are flipped, predict that there is no horizontal flipping. If both are flipped, something that was flipped horizontally in the past is flipped between frames. In other words, it is a flip of a flip, so the result is no flipping. The reliability of the prediction obtained here is not enough to use alone, so it is treated as a correction to the score calculated in [3].

[0054] The score calculated in [3] is corrected so that the score of the predicted left-right reversed pattern is more likely to be selected. By adding this correction to the score in equation (1), we obtain equation (3).

[0055]

number

[0056] In equation (3), S represents each left-right flip pattern. Spred represents the left-right flip pattern of the current frame predicted from the results of [1] and [2]. Length represents the two-dimensional length on the screen when the three-dimensional length between the hip joints of the target subject is displayed on the screen. β is the coefficient (parameter) of the correction term.

[0057] As described above, according to the embodiment, a pattern with small variance and small mean distances between corresponding joint positions in the previous frame and the current frame is selected from the four patterns of upper body, lower body, with inversion, and without inversion. Furthermore, intra-frame correction and inter-frame correction are performed based on the selected pattern. In this way, errors due to left-right inversion can be suppressed, and therefore, 2D posture information and 3D posture information can be accurately estimated.

[0058] That is, according to the embodiment, an algorithm for determining the problem of left-right inversion of 2D posture information is introduced as post-processing. This improves the robustness of the process and increases the accuracy of estimating 3D posture information. As a result, according to the embodiment, left-right inversion of skeleton coordinates in 2D posture information is suppressed, thereby enabling accurate estimation of 3D posture information.

[0059] In addition, in the implementation stage, the components of the present invention can be modified to a specific extent without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0060] 1-1~1-n...Camera 10...Video processing device 20...Processor 21...Current frame extraction unit 30...D posture information generation section 40... Posture information processing unit 41...Right-left inversion determination unit 42...Right-left inversion correction processing section 50...3D processing section 60…Storage 61…2D posture information 62…3D posture information 63...Program 70...Interface section 80...Bus 90...memory.

Claims

1. a 2D posture information generation unit that generates 2D posture information of a subject from video data of the subject; a posture information processing unit that corrects the 2D posture information; a 3D processing unit that generates 3D posture information of the subject from the corrected 2D posture information, the posture information processing unit includes a left-right inversion determination unit that performs left-right inversion determination processing, The left-right inversion determination process is a process of setting first to fourth patterns corresponding to whether the upper and lower body are inverted or not for the image frames of the subject in the video data, and selecting from the first to fourth patterns the pattern that has the smallest score calculated using an evaluation formula whose variables are the variance and average of the distances between corresponding joint positions in the previous frame and the current frame.

2. The image processing device according to claim 1 , wherein the posture information processing unit further comprises a left-right inversion correction processing unit that performs the left-right inversion determination processing within a frame.

3. The image processing device according to claim 1 , wherein the posture information processing section further comprises a left-right inversion correction processing section that performs the left-right inversion determination process in two stages within a frame.

4. The image processing device according to claim 1 , wherein the orientation information processing unit further comprises a left-right inversion correction processing unit that performs the left-right inversion determination processing between successive frames.

5. A video processing method executed by a processor of a video processing device including a processor and a storage, generating 2D pose information of the subject from video data of the subject by the processor; the processor correcting the 2D pose information; and generating 3D pose information of the object from the corrected 2D pose information by the processor; The step of correcting the 2D posture information includes: the processor setting first to fourth patterns corresponding to whether or not an upper body and a lower body are inverted for the image frames of the subject of the video data; a step of the processor calculating a score using an evaluation formula in which the variance and the average of the distances between the corresponding joint positions in the previous frame and the current frame are used as variables, from among the first to fourth patterns; and a step by the processor selecting the pattern that minimizes the score.

6. A program that causes a computer to function as each of the units of the video processing device according to claim 1 .

Images