Joint position misrecognition determination device and joint position misrecognition determination method

Abstract

To detect misrecognition of joint positions caused by left-right inversion.SOLUTION: A joint position misrecognition determination device comprises first 2D joint position estimation means for estimating a first 2D coordinate value as a joint position, and second 2D joint position estimation means for estimating a second 2D coordinate value as a joint position. The first and second 2D coordinate values are obtained in synchronization as values on a common XY plane. The device also comprises means for calculating a distance D between the first and second 2D coordinate values for the same frame and the same joint ID, and determination means for determining that the joint position is left-right inverted when the distance D exceeds a predetermined value d. The predetermined value d is determined on the basis of the distance between the 2D coordinates of a predetermined pair of left-right joints in the same frame.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to motion capture, and more particularly to a joint position misidentification determination device and method. [Background technology]

[0002] In the fields of sports and rehabilitation, 3D motion analysis using motion capture is used to record and analyze the movements of subjects. For example, in rehabilitation medicine, treatment plans and the degree of disability have traditionally been assessed based on experience, but in recent years the importance of evidence-based medicine (EBM) has been emphasized. By acquiring 3D joint positions, human movements can be quantitatively measured, and treatment plans and quantitative assessments of disabilities can be made, making it possible to realize evidence-based medicine (EBM).

[0003] Until now, quantitative evaluation has been performed using an infrared reflection type three-dimensional motion analysis device, but attaching the markers and processing the measurements afterwards takes time, making it difficult to use in clinical settings in terms of measurement time.

[0004] In recent years, so-called markerless motion capture, which does not use markers or inertial sensors, has emerged. Known markerless joint position acquisition techniques include a technique in which a distance image (depth image) is acquired and joint positions are estimated from the acquired depth image (Non-Patent Document 1), and a technique in which a human figure is found in a single RGB image and joint positions are estimated by generating a heat map that indicates the likelihood of joint positions. For the former, relatively inexpensive motion capture using distance imaging, such as Kinect (Non-Patent Document 2), has appeared on the market. For the latter, OpenPose (https: / / github.com / CMU-Perceptual-Computing-Lab / open) and VisionPose (https: / / www.next-system.com / visionpose) are well-known. The present applicant has proposed a joint position acquisition device and method that combines these techniques (Patent Document 1).

[0005] However, these technologies often result in misidentification of joint positions. One characteristic of misidentification of joint positions is that the left and right joint points are output in an inverted manner due to the bilateral symmetry of the human body (see Figure 1). "Bilateral symmetry of the human body" refers to the fact that the human body is symmetrical with respect to the sagittal plane (the plane formed by the vertical and anteroposterior axes). In a static image, if the frontal plane (the plane formed by the vertical and lateral axes) is facing in the same direction as the plane of the camera's elements and the face is not captured, it can be difficult for a machine to determine whether the person is facing towards or away from the camera.

[0006] Previously, measurements were taken directly, or the output 3D joint positions of the target were detected by human eyes after left-right mirroring, the IDs of the joints in that area were removed, and the joint positions were interpolated.

[0007] The easiest way to solve this left-right inversion problem is to use deep learning or other AI technologies for additional learning. However, unless you are a developer, it is difficult to obtain the learning data. In addition, obtaining the learning data independently and performing data preprocessing and learning is extremely costly, making it difficult for users to actually implement this. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent No. 7287686 [Non-patent literature]

[0009] [Non-Patent Document 1] J. Shotton et al., “Real-Time Human Pose Recognition in Parts from a Single depth Image,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), IEEE CS Press, 2011, pp. 1297-1304. [Non-patent document 2] Z. Zhang. Microsoft kinect sensor and its effect. IEEE Multimedia, 19(2):4-10, Feb 2012. DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0010] Detecting misidentification of joint positions due to left-right reversal as described above and appropriately correcting the joint positions is important in ensuring the accuracy of joint positions, and it is desirable to achieve this at a lower cost. [Means for solving the problem]

[0011] The joint position misidentification determination device adopted in the present invention is a first- and second-dimensional joint position estimation means for estimating first- and second-dimensional coordinate values ​​as joint positions; a second-2D joint position estimation means for estimating second-2D coordinate values ​​as joint positions; Equipped with The 1st and 2nd 2D coordinate values ​​are obtained synchronously as values ​​on a common XY plane, A means for calculating a distance D between a first 2D coordinate value and a second 2D coordinate value for the same frame and the same joint ID; a determination means for determining that the position of the joint is reversed when the distance D is greater than a predetermined value d; Equipped with The predetermined value d is determined based on the distance between the 2D coordinates of a predetermined pair of left and right joints in the same frame. In one embodiment, the predetermined value d is calculated using either the first 2D coordinate value or the second 2D coordinate value.

[0012] In one aspect, the pair of predetermined left and right joints corresponds to the joints to be judged by the judging means. In one embodiment, the predetermined pair of left and right joints is one or more representative joints selected from a plurality of pairs of left and right joints.

[0013] In one embodiment, the predetermined value d is determined from a value between 1 / 3 and 1 / 2 of the distance between the 2D coordinates of the pair of left and right joints.

[0014] In one embodiment, the present invention includes a 3D joint position estimation means for estimating 3D coordinate values ​​of joints, The 3D coordinate value assigned to the joint ID determined by the determination means to be left-right inverted is canceled and replaced with a 3D coordinate value obtained by interpolation using multiple 3D coordinate values ​​assigned to the joint ID in multiple frames excluding the frame in question. In one aspect, the 3D joint position estimation means uses a distance image acquired by a distance image sensor, One of the first 2D joint position estimation means and the second 2D joint position estimation means estimates 2D coordinate values ​​based on the 3D joint positions estimated by the 3D joint position estimation means.

[0015] In a more specific aspect, a range image sensor; At least one camera; a 3D joint position estimation means for estimating 3D joint positions of an object based on a distance image of the object acquired by the distance image sensor; a first- and second-dimensional joint position estimation means for estimating first- and second-dimensional joint positions of an object based on an RGB image of the object acquired by the camera; and second 2D joint position estimation means for estimating second 2D joint positions based on the estimated 3D joint positions (2D joint positions on the camera image corresponding to each 3D joint position acquired by the 3D joint position estimation means are obtained). In one embodiment, the range image sensor and the camera are integrated as a combined sensor.

[0016] In one embodiment, the present invention includes a 3D joint position estimation means for estimating 3D coordinate values ​​of joints, the 3D joint position estimation means reconstructs a plurality of 2D coordinate values ​​of a joint to obtain a 3D coordinate value of the joint; a means for correcting (by exchanging or interpolating) the 2D coordinate values ​​corresponding to the joint IDs determined by the determining means to be left-right reversed, The 3D coordinate values ​​corresponding to the joint IDs determined by the determining means to be left-right reversed are cancelled and replaced with 3D coordinate values ​​reconstructed using the corrected 2D coordinate values. In one aspect, the 3D joint position estimation means acquires either a first-2D coordinate value or a second-2D coordinate value as the 3D coordinate value of the joint. In one embodiment, the means for modifying the 2D coordinate values ​​modifies either the first·2D coordinate values ​​or the second·2D coordinate values, and the 3D coordinate values ​​are recalculated using the modified 2D coordinate values. In one embodiment, the means for correcting the 2D coordinate values ​​swaps the coordinate values ​​of a pair of joint IDs that are determined to be mirror-inverted. In one aspect, the means for correcting the 2D coordinate values ​​interpolates the coordinate values ​​of a joint ID determined to be mirror-inverted using multiple 2D coordinate values ​​assigned to the joint ID in multiple frames excluding the frame in question.

[0017] The joint position misidentification determination method adopted in the present invention is as follows: estimating 1D / 2D coordinate values ​​as joint positions by 1D / 2D joint position estimation means; estimating 2D coordinate values ​​as joint positions by a 2D joint position estimation means; Calculating the distance D between the 1st and 2nd 2nd coordinate values ​​for the same frame and joint ID; When the distance D is greater than a predetermined value d, it is determined that the position of the joint is reversed left and right. Equipped with The 1st and 2nd 2D coordinate values ​​are obtained synchronously as values ​​on a common XY plane, The predetermined value d is determined based on the distance between the 2D coordinates of a predetermined pair of left and right joints in the same frame.

[0018] In one aspect, the pair of predetermined left and right joints corresponds to the joints to be judged by the judging means. In one embodiment, the predetermined pair of left and right joints is one or more representative joints selected from a plurality of pairs of left and right joints.

[0019] In one embodiment, the predetermined value d is determined from a value between 1 / 3 and 1 / 2 of the distance between the 2D coordinates of the pair of left and right joints.

[0020] In one aspect, the method comprises estimating 3D coordinate values ​​of the joints by a 3D joint position estimation means, The 3D coordinate values ​​assigned to joint IDs that are determined to be mirror-inverted are canceled and replaced with 3D coordinate values ​​obtained by interpolation. In one aspect, the 3D joint position estimation means uses a distance image acquired by a distance image sensor, One of the first 2D joint position estimation means and the second 2D joint position estimation means estimates 2D coordinate values ​​based on the 3D joint positions estimated by the 3D joint position estimation means.

[0021] In one aspect, the method comprises estimating 3D coordinate values ​​of the joints by a 3D joint position estimation means, the 3D joint position estimation means reconstructs a plurality of 2D coordinate values ​​of a joint to obtain a 3D coordinate value of the joint; Correct the 2D coordinate values ​​corresponding to the joint IDs that are determined to be inverted left and right, The 3D coordinate values ​​corresponding to the joint IDs determined to be left-right inverted are canceled and replaced with 3D coordinate values ​​reconstructed using the corrected 2D coordinate values. [Effects of the Invention]

[0022] In the present invention, the size (distance) on the camera image decreases in inverse proportion to the distance the subject moves away from the camera, and this can be detected by a simple method of comparing two 2D coordinate values ​​estimated by different 2D joint position estimation means. Furthermore, although the present invention is typically used to obtain more accurate 3D joint positions, it can also be applied to assessing the accuracy of joint positions in two-dimensional images. [Brief explanation of the drawings]

[0023] [Figure 1] This is a diagram showing a case where the left and right joint points are output in an inverted manner, with the left diagram showing the correct joint position and the right diagram showing the incorrect joint position. [Figure 2] 10A and 10B are diagrams illustrating joint position misidentification determination according to the present embodiment. [Figure 3] 10 is a diagram illustrating a predetermined value d used in joint position misidentification determination according to the present embodiment. FIG. [Figure 4] FIG. 10 is a diagram showing joint position misidentification determination according to the first embodiment. [Figure 5] FIG. 11 is a diagram showing joint position misidentification determination according to the second embodiment. [Figure 6] 5A to 5C are diagrams illustrating in detail joint position error determination according to the first embodiment. [Figure 7] 10A and 10B are diagrams illustrating a method for correcting a 3D joint position that has been determined to be an erroneous joint position; DETAILED DESCRIPTION OF THE INVENTION

[0024] [A] Overview of the joint position misrecognition detection device The joint position misidentification determination device according to this embodiment typically functions as part of a 3D joint position estimation device. An object (human body) has multiple joints, and the 3D pose of the object in each frame is determined by the multiple 3D joint positions estimated in that frame. Each joint is identified by being assigned an ID, and each joint ID is associated with each 3D coordinate value. Here, the 3D joint position may be output with its left and right sides flipped. For example, when two coordinate values ​​A and B are detected, the left knee joint ID should be assigned to coordinate value A and the right knee joint ID should be assigned to coordinate value B, but the right knee joint ID is assigned to coordinate value A and the left knee joint ID is assigned to coordinate value B.

[0025] Here, in many cases, the 3D joint position estimation means obtains (including cases where it is possible to calculate) a two-dimensional image that defines a 2D joint position corresponding to the 3D joint position. For example, in the aspects shown in Fig. 4 and Fig. 6, a 2D joint position (XY plane) corresponding to the 3D joint position estimated by the 3D joint position estimation means (using a range image acquired by a range image sensor) is obtained. Furthermore, in the 3D joint position estimation means that reconstructs a 3D joint position from multiple 2D joint positions, the 2D joint positions used when calculating the 3D joint position exist.

[0026] If a 3D joint position is misidentified (left-right inverted), it can be estimated that the corresponding 2D joint position is also misidentified (left-right inverted). In this embodiment, this feature is utilized to detect left-right inversion of joint positions using two types of 2D joint positions in the same frame.

[0027] As shown in FIG. 2 , the joint position error determination device according to this embodiment includes a first-2D joint position estimation means for estimating 1·2D coordinate values ​​as joint positions of a target (human body), and a second-2D joint position estimation means for estimating 2·2D coordinate values ​​as joint positions of the target (human body). The first-2D coordinate values ​​and the second-2D coordinate values ​​are obtained synchronously and as values ​​on a common XY plane. It will be understood by those skilled in the art that if the 2D coordinate values ​​primarily estimated by the first-2D joint position estimation means and the 2D coordinate values ​​primarily estimated by the second-2D joint position estimation means are values ​​on different planes, one of the 2D coordinate values ​​can be converted so that the first-2D coordinate values ​​and the second-2D coordinate values ​​become values ​​on a common XY plane.

[0028] Each joint ID is assigned a 1D and 2D coordinate value, respectively, and it is possible to compare the 1D and 2D coordinate values ​​for the same frame and joint ID. If the positions obtained using the two 2D joint estimation methods are far apart, it can be determined that the ID assignment is incorrect. For the same frame and joint ID, the distance D between the 1D and 2D coordinate values ​​(X1, Y1) and the 2D and 2D coordinate values ​​(X2, Y2) is calculated. If the distance D is greater than a predetermined value d, it is determined that the joint position is mirrored. In other words, if the distance D between the coordinate values ​​obtained using the first and second 2D estimation methods for each joint in each frame is greater than a certain value d, the ID assignment for the joint being processed is canceled.

[0029] If the positions obtained by the two 2D joint estimation methods are far apart, it can be determined that the ID assignment is incorrect. However, the distance between them must be set appropriately; in other words, how to determine the predetermined value d is important. The size of an object captured in a camera image (distance D) is inversely proportional to the spatial distance from the camera. For example, when the distance between two points in space is the same, the distance between the camera's pixels decreases inversely proportional to the distance from the camera to a feature point (marker or joint point). Therefore, since the distance D varies depending on the distance between the camera and the object, the predetermined value d must also be variable.

[0030] In this embodiment, the predetermined value d is a variable threshold determined based on the distance between the 2D coordinates of a pair of predetermined left and right joints in the same frame. By calculating the predetermined value d based on the distance between the predetermined joint positions in the same frame, the condition of the distance from the camera to the object can be reflected in the predetermined value d. In this embodiment, each joint has a 1·2D coordinate value and a 2·2D coordinate value, so that the predetermined value d can be calculated in each frame using either the 1·2D coordinate value or the 2·2D coordinate value.

[0031] In one embodiment, the pair of predetermined left and right joints corresponds to the joints to be determined by the determining means. a) When the joint to be calculated is the knee, the distance d1 is set to half the distance between the left and right knee joints in that frame. b) When the joint to be calculated is the ankle joint, d is set to d2, which is half the distance between the left and right ankle joints in that frame. c) If the joint to be calculated is the toe, let d be the distance d3, which is half the distance between the left and right toes in that frame.

[0032] In one embodiment, the predetermined pair of left and right joints is one or more representative joints selected from a plurality of pairs of left and right joints. For example, as shown in FIG. 3, the left and right hip joints are used as representative joints, d) If the joint to be calculated is a joint other than those mentioned above, the distance d4, which is half the distance between the left and right hip joints in that frame, is set to d.

[0033] The above method of calculating the predetermined value d is an example, and a person skilled in the art can appropriately determine how to select a pair of left and right joints for calculating the predetermined value d. In d), the left and right hip joints are used as representative joints, but multiple representative joints may be set depending on the joints to be judged.

[0034] In the above, the predetermined value d is set to 1 / 2 of the distance between the 2D coordinates of the selected pair of left and right joints, but "1 / 2" is an example and is not limited to this. For example, the predetermined value d is determined from a value between 1 / 3 and 1 / 2 of the distance between the 2D coordinates of the selected pair of left and right joints. Furthermore, the value by which the predetermined pair of joints is multiplied (1 / 2 in the above example) does not have to be uniformly applied to all joints, and the value by which the predetermined joint distance is multiplied may be different depending on the joint to be determined.

[0035] As described above, the joint position misrecognition determination device according to this embodiment includes a 1·2D joint position estimation unit that estimates 1·2D coordinate values ​​as joint positions, and a 2·2D joint position estimation unit that estimates 2·2D coordinate values ​​as joint positions, where the 1·2D coordinate values ​​and the 2·2D coordinate values ​​are synchronously obtained as values ​​on a common XY plane. The device also includes a means for calculating a distance D between the 1·2D coordinate values ​​and the 2·2D coordinate values ​​for the same frame and the same joint ID, and a determination unit that determines that the joint position is mirror-reversed when the distance D is greater than a predetermined value d, where the predetermined value d is determined based on the distance between the 2D coordinates of a pair of predetermined left and right joints in the same frame. The joint position misrecognition determination device according to this embodiment is part of a 3D joint position estimation device, and the 3D joint position estimation unit and the joint position misrecognition determination device can be configured by a computer equipped with a memory and a processor.

[0036] The joint position misidentification detection device of this embodiment consists of 1) joint position misidentification detection (a process of removing IDs from 3D joint positions by comparing two types of 2D images) and 2) a correction process (interpolation or recalculation) for the 3D joint positions from which IDs have been removed.

[0037] [B] First embodiment of joint position misidentification determination 4 and 6, the joint position misidentification determination device according to the first embodiment functions as part of a 3D joint position estimation device, which is a distance image-based joint position estimation device that includes a distance image camera that acquires a depth image (distance image) of an object, a video camera (RGB camera) that acquires a color image (RGB image) of the object, a 3D joint position estimation unit that acquires a 3D joint position of the object based on the depth image of the object acquired by the distance image sensor, and a 2D joint position estimation unit that acquires a 2D joint position of the object based on the color image (RGB image) of the object acquired by the video camera. In this embodiment, the distance image sensor and the video camera are integrated into a composite sensor, but the distance image sensor and the video camera may also be provided as separate entities.

[0038] In one embodiment, the distance image-based joint position estimation device is a device that outputs human body skeletal coordinates and color images, such as Kinect (registered trademark). Kinect is configured as a composite sensor consisting of an infrared projector, a distance image camera, and a color image camera, and the distance image sensor consists of the infrared projector and the infrared camera. Kinect is equipped with a 3D joint position estimation means that acquires a person area based on the acquired distance image and inputs the distance information of the person area into a classifier to estimate joint positions. A program that estimates the 3D joint positions of an object based on depth data of the object is stored in the computer's memory, and the program is executed by a processor to estimate and output the 3D joint positions of the object. The estimated 3D joint positions are stored in the memory as time-series data.

[0039] Kinect is equipped with a video camera (RGB camera) that can capture color images of objects. The RGB image captured by the color camera contains position information (pixel position information) on a 2D image and color information (RGB values) for each pixel. In the combined sensor, the distance image captured by the distance image sensor and the color image captured by the video camera are synchronized, and a distance image, color image, and 3D joint coordinates are obtained at the same time t. The computer's memory stores the depth image (distance image) captured by the depth camera (distance image sensor), the RGB image captured by the RGB camera, and the 3D joint positions. The depth image data, RGB image data, and 3D joint positions are time-series data and are stored synchronized.

[0040] The composite sensor (range image sensor, RGB camera) according to this embodiment has a coordinate system centered on the range image sensor and a coordinate system centered on the RGB camera. The range image acquired by the range image sensor is defined by the 2D coordinate system of the range image sensor, and the color image acquired by the RGB camera is defined by the 2D coordinate system of the RGB camera. The 3D joint positions estimated based on the range image are defined by the 3D coordinate system of the range image sensor. The conversion information between the 2D coordinate system of the range image sensor and the 2D coordinate system of the RGB camera is obtained in advance, making the conversion possible. In other words, the coordinates (XY) of any point (XY, depth value) on the range image and the coordinates (u, v) of any point (u, v, RGB value) on the color image can be converted.

[0041] The depth image and the RGB image are acquired synchronously, and by calculating projection points based on the 3D joint positions estimated by the 3D joint position estimation means, it is possible to estimate 2D joint positions based on the 3D joint positions acquired by the 3D joint position estimation means (constituting the first 2D joint position estimation means).

[0042] This section describes a 2D joint position estimation method (second 2D joint position estimation method) based on RGB images captured by a color camera. The 2D joint position estimation method uses deep learning to find a human figure from a single RGB image, generate a heat map representing the likelihood of joint positions, and estimate joint positions based on the heat map. A program that estimates the pose (2D joint positions) of an object based on the object's RGB image is stored in the computer's memory. The program is executed by the processor to estimate and output the object's 2D joint positions. The estimated 2D joint positions are stored in the computer's memory. Estimating an object's pose (2D joint coordinate values) based on a single color image is well known. One representative research method is OpenPose (Openpose. https: / / github.com / CMU-Perceptual-Computing-Lab / open). OpenPose can estimate multiple human feature points (keypoints), such as wrists and shoulders, in real time from a single RGB image. Various methods have been proposed to estimate and output joint coordinate values ​​based on heat maps (PCM in OpenPose) of each joint position (feature point). Pose estimation, which estimates joint positions from a single RGB image, is commercially available (e.g., https: / / www.next-system.com / visionpose).

[0043] The depth image and the RGB image are acquired synchronously. Therefore, the 3D joint positions acquired by the 3D joint position estimation means and the 2D joint positions (second-2D joint positions) acquired by the 2D joint position estimation means are synchronized. By calculating projection points based on the 3D joint positions estimated by the 3D joint position estimation means, the 2D joint positions (first-2D joint positions) based on the 3D joint positions acquired by the 3D joint position estimation means can be estimated. In one embodiment, the markerless motion capture using the range image method is a Kinect, which is a composite sensor. The first-2D joint position estimation means is implemented as a function of the Kinect. The first-2D coordinate values ​​are 2D joint coordinate values ​​of the Kinect. The second-2D joint position estimation means is VisionPose. The second-2D coordinate values ​​are calculated using color images output from the Kinect. The first-2D coordinate values ​​tend to be less accurate than the second-2D coordinate values.

[0044] The 1·2D coordinate values ​​and 2·2D coordinate values ​​are obtained synchronously as values ​​on a common XY plane. 1·2D coordinate values ​​and 2·2D coordinate values ​​are assigned to each joint ID. For the same frame and joint ID, the distance D between the 1·2D coordinate value (X1, Y1) and the 2·2D coordinate value (X2, Y2) is calculated. If the distance D is greater than a predetermined value d, the joint position is mistakenly determined to be left-right reversed. Joint IDs corresponding to joint positions that are mistakenly determined to be incorrect are canceled, and joint IDs corresponding to joint positions that are not mistakenly determined to be incorrect are treated as valid.

[0045] [C] Second embodiment of joint position misidentification judgment The joint position misidentification judgment device shown in Figure 5 functions as part of a 3D joint position estimation device, which estimates 3D joint positions by reconstructing multiple 2D joint positions estimated from multiple camera images in each frame.The 3D joint position estimation device is equipped with a 2D joint position estimation means based on RGB images acquired by a color camera (human body skeleton coordinate value estimation using color camera images), and by using two or more color cameras, it estimates 3D joint positions by reconstructing 2D joint positions estimated from each camera image.

[0046] The joint position error determination device according to the second embodiment includes two types of 2D joint position estimation means. Based on color camera images, the first-2D joint position estimation means estimates 1D coordinate values, and the second-2D joint position estimation means estimates 2D coordinate values. The first-2D coordinate values ​​and the 2D coordinate values ​​are synchronously obtained as values ​​on a common XY plane. Each joint ID is assigned a 1D coordinate value and a 2D coordinate value. For the same frame and joint ID, the distance D between the 1D coordinate value (X1, Y1) and the 2D coordinate value (X2, Y2) is calculated. If the distance D is greater than a predetermined value d, the joint position is determined to be reversed. The joint ID corresponding to the joint position determined to be incorrect is canceled, and the joint ID corresponding to the joint position not determined to be incorrect is treated as valid.

[0047] [D] Correction process for 3D joint positions that were identified as misidentified The 3D joint positions determined to be misidentified are cancelled and replaced with corrected 3D joint positions. The following approaches, for example, can be used to correct the 3D joint positions determined to be misidentified. 1) Replace the canceled 3D joint positions with values ​​interpolated using the 3D joint positions of multiple frames (Figure 7). 2) When two or more color cameras are used, the 2D joint positions corresponding to the joint IDs determined to be inverted left and right are corrected, and the 3D joint positions are recalculated using the corrected 2D joint positions to replace the canceled 3D joint positions.

[0048] [D-1] Method 1 (3D joint position interpolation) In the first method, the 3D coordinate value assigned to a joint ID determined by the determination means to be left-right inverted is canceled and replaced with a 3D coordinate value obtained by interpolation using multiple 3D coordinate values ​​assigned to the joint ID in multiple frames excluding the frame in question. (Figure 7, Figure 6).

[0049] The 3D joint positions estimated by the 3D joint position estimation device are time-series data, and in multiple consecutive frames, the ID assignment of 3D joint positions that were incorrectly determined is canceled, and valid 3D joint positions are obtained in multiple frames excluding the frames with the canceled 3D joint positions. By performing interpolation processing using multiple valid 3D joint positions, the canceled 3D joint positions are replaced with interpolated 3D joint positions and assigned IDs. An example of the interpolation processing is spline interpolation. As an optional process, the projected coordinate values ​​of the replaced 3D joint positions onto the XY plane are adopted as new 2D coordinate values ​​(first and second D coordinate values).

[0050] [D-2] Second method (3D reconstruction using corrected 2D joint positions) A 3D joint position estimation device to which the second technique is applied uses a means for reconstructing 3D joint positions using multiple cameras. In the second technique, 3D coordinate values ​​assigned to joint IDs determined by the determination means to be mirror-inverted are canceled, 2D coordinate values ​​assigned to a pair of left and right joint IDs determined by the determination means to be mirror-inverted are corrected, 3D coordinate values ​​are recalculated using the corrected 2D coordinate values ​​to perform 3D reconstruction, and the canceled 3D joint positions are replaced with the obtained 3D coordinate values. The means for correcting the 2D coordinate values ​​assigned to a pair of left and right joint IDs determined to be mirror-inverted may, for example, use 2D coordinate values ​​obtained by interpolation from valid 2D coordinate values ​​assigned to the joint IDs in multiple consecutive frames. Alternatively, when it is guaranteed that the second·2D coordinate values ​​of one of the two 2D joint position estimation means used in the joint position error determination are highly reliable, and there is also a third 2D joint position estimation means that was not used in the joint position error determination, the 3D coordinate values ​​may be three-dimensionally reconstructed from the second·2D coordinate values ​​and the third·2D coordinate values ​​output by the third 2D joint position estimation means.

Claims

1. a first and second-dimensional joint position estimation means for estimating first and second-dimensional coordinate values ​​as joint positions; a second 2D joint position estimation means for estimating second 2D coordinate values ​​as joint positions; Equipped with The 1st and 2nd 2D coordinate values ​​and the 2nd and 2nd 2D coordinate values ​​are obtained synchronously as values ​​on a common XY plane, A means for calculating a distance D between a first 2D coordinate value and a second 2D coordinate value for the same frame and the same joint ID; a determining means for determining that the position of the joint is reversed left and right when the distance D is greater than a predetermined value d; Equipped with The predetermined value d is determined based on the distance between the 2D coordinates of a predetermined pair of left and right joints in the same frame. Joint position misidentification detection device.

2. The predetermined pair of left and right joints corresponds to the joints to be determined by the determination means. The joint position misidentification determination device according to claim 1 .

3. The predetermined pair of left and right joints is one or more representative joints selected from a plurality of pairs of left and right joints. The joint position misidentification determination device according to claim 1 .

4. The predetermined value d is determined from a value between 1 / 3 and 1 / 2 of the distance between the 2D coordinates of the pair of left and right joints. The joint position misidentification determination device according to any one of claims 1 to 3.

5. a 3D joint position estimation means for estimating 3D coordinate values ​​of joints; The 3D coordinate value assigned to the joint ID determined by the determination means to be left-right inverted is canceled and replaced with a 3D coordinate value obtained by interpolation using multiple 3D coordinate values ​​assigned to the joint ID in multiple frames excluding the frame in question. The joint position misidentification determination device according to claim 1 .

6. the 3D joint position estimation means uses a distance image acquired by a distance image sensor; one of the first 2D joint position estimation means and the second 2D joint position estimation means estimates 2D coordinate values ​​based on the 3D joint positions estimated by the 3D joint position estimation means; The joint position misidentification determination device according to claim 5 .

7. a 3D joint position estimation means for estimating 3D coordinate values ​​of joints; the 3D joint position estimation means reconstructs a plurality of 2D coordinate values ​​of a joint to obtain a 3D coordinate value of the joint; a means for correcting a 2D coordinate value corresponding to the joint ID determined by the determining means to be left-right reversed, The 3D coordinate values ​​corresponding to the joint IDs determined by the determination means to be left-right reversed are canceled and replaced with 3D coordinate values ​​reconstructed using the corrected 2D coordinate values. The joint position misidentification determination device according to claim 1 .

8. estimating first and second-dimensional coordinate values ​​as joint positions by a first and second-dimensional joint position estimation means; estimating second 2D coordinate values ​​as joint positions by a second 2D joint position estimation means; Calculating the distance D between the first 2D coordinate value and the second 2D coordinate value for the same frame and the same joint ID; When the distance D is greater than a predetermined value d, it is determined that the position of the joint is reversed left and right. Equipped with The 1st and 2nd 2D coordinate values ​​and the 2nd and 2nd 2D coordinate values ​​are obtained synchronously as values ​​on a common XY plane, The predetermined value d is determined based on the distance between the 2D coordinates of a predetermined pair of left and right joints in the same frame.

9. The predetermined pair of left and right joints corresponds to the joints to be determined by the determination means. The joint position misidentification determination method according to claim 8 .

10. The predetermined pair of left and right joints is one or more representative joints selected from a plurality of pairs of left and right joints. The joint position misidentification determination method according to claim 8 .

11. The predetermined value d is determined from a value between 1 / 3 and 1 / 2 of the distance between the 2D coordinates of the pair of left and right joints. The joint position misidentification determination method according to any one of claims 8 to 10.

12. estimating 3D coordinate values ​​of the joints by a 3D joint position estimation means; The 3D coordinate value assigned to the joint ID determined to be left-right inverted is canceled and replaced with a 3D coordinate value obtained by interpolation using multiple 3D coordinate values ​​assigned to the joint ID in multiple frames excluding the frame in question. The joint position misidentification determination method according to claim 8 .

13. the 3D joint position estimation means uses a distance image acquired by a distance image sensor, one of the first 2D joint position estimation means and the second 2D joint position estimation means estimates 2D coordinate values ​​based on the 3D joint positions estimated by the 3D joint position estimation means; The joint position misidentification determination method according to claim 12.

14. estimating 3D coordinate values ​​of the joints by a 3D joint position estimation means; the 3D joint position estimation means reconstructs a plurality of 2D coordinate values ​​of a joint to obtain a 3D coordinate value of the joint; Correct the 2D coordinate values ​​corresponding to the joint IDs that are determined to be inverted left and right, The 3D coordinate values ​​corresponding to the joint IDs determined to be inverted are canceled and replaced with 3D coordinate values ​​reconstructed using the corrected 2D coordinate values. The joint position misidentification determination method according to claim 8 .

Images