Feature point location estimation device, feature point location estimation method, and program

Abstract

To estimate a position of a body feature point.SOLUTION: A feature-point position estimation device for estimating three-dimensional coordinates of a prescribed feature point of a body located at a target segment of an object, includes: a position input part for inputting three-dimensional coordinate values of a first body feature point for identifying a position in a first side of the target segment of the object and of a second body feature point for identifying a position in a second side, and three-dimensional coordinate values of a plurality of body surface points of the object; a plane calculation part for calculating a plane in a three-dimensional space passing through the first body feature point, the plane perpendicular to a straight line connecting the first body feature point and the second body feature point; an extraction part for extracting a plurality of body surface points on the plane, a distance of which from the plane is a prescribed range out of the plurality of body surface points; an approximation part for approximating the plurality of body surface points on the plane to a space ellipse on a three-dimensional space; a condition input part for inputting conditions representing at which point on the space ellipse the prescribed body feature point is located; and a position estimation part for estimating the three-dimensional coordinate values of the prescribed body feature point on the basis of, the space ellipse and conditions.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a technique for estimating the positions of body feature points.

Background Art

[0002] Conventionally, in rehabilitation medicine, orthopedic treatment, etc., three-dimensional motion analysis using motion capture has been performed to measure and analyze the movements of subjects. As one method for evaluating motion in three-dimensional motion analysis, a motion analysis method using markerless motion capture is known. As markerless motion capture, for example, a method using the TOF (Time of Flight) method that measures the distance to an object by irradiating an infrared laser with a distance image sensor (depth sensor) to obtain a depth image (distance image) having information on the distance to the object, and deriving (estimating) the joint positions, etc. of the object from the obtained depth image is known (Non-Patent Document 1). Also, a method for estimating the joint positions, etc. of an object from a depth image using Kinect (registered trademark) as a motion capture device is also described in Non-Patent Document 2.

[0003] In markerless motion capture, unlike optical motion capture or inertial motion capture, it is not necessary to attach markers or sensors to objects such as people and objects. Therefore, it is not necessary to perform complicated measurement preparations, and it is possible to measure the natural movements of subjects. Thus, markerless motion capture is used as a highly useful method in various scenarios. For example, a method for estimating skeletal information based on a distance image by using markerless motion capture has been proposed (Patent Document 1), and the utilization of markerless motion capture in clinical settings is also expected (Non-Patent Document 3).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] [Non-Patent Document 1] Takashi Nishibayashi, "The Mechanism of Kinect and Natural User Interface," Journal of the Institute of Image Information and Television Engineers, Vol. 66 No. 9, 2012, pp. 755-759. [Non-Patent Document 2] "Azure Kinect DK Documentation", [online], Microsoft, [Retrieved October 27, 2021], Internet<URL:https: / / docs.microsoft.com / ja-jp / azure / Kinect-dk / > [Non-Patent Document 3] Koichi Haruna, et al., "Applications of Three-Dimensional Motion Analysis Using Markerless Motion Capture," Journal of the Japanese Society for Prosthetics and Orthotics, Vol. 35, No. 1, 2019, pp. 17-23. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] As mentioned above, in motion analysis methods using markerless motion capture, depth images are acquired by distance image sensors, making it possible to obtain the positions (3D position coordinates) of physical feature points such as joints of the object (subject).

[0007] However, while distance image sensors can acquire (output) the position coordinates of predetermined body feature points such as joints and bones within the sensor, it is difficult to acquire the position of body feature points that are not predetermined within the sensor. Therefore, there is a problem in that it may not be possible to calculate the desired joint angle using only the 3D position coordinates of body feature points predetermined within the sensor.

[0008] For example, in the lower limbs, the 3D position coordinates of predetermined body feature points—the knee joint, ankle joint, and toes—are typically output from the depth image sensor. While it is possible to calculate the plantarflexion / dorsiflexion angles and adduction / abduction angles of the ankle joint using projection angles based on these 3D position coordinates of body feature points, it is difficult to correctly calculate the inversion / eversion angles of the ankle joint. To calculate the inversion / eversion angles of the ankle joint, it is possible to use, for example, the Euler angles between body segments (between body segment coordinate systems). However, in body segments where only the position coordinates of two body feature points are output, such as the foot where only the 3D position coordinates of the ankle joint and toes are output, it is not possible to generate (specify) a coordinate system, making it difficult to calculate the Euler angles (inversion / eversion angles).

[0009] In light of the above-mentioned problems, this disclosure aims to estimate the location of physical characteristic points. [Means for solving the problem]

[0010] An example of this disclosure is, A feature point position estimation device that estimates the three-dimensional coordinate values ​​of predetermined physical feature points located at target body segments of an object, A position input means for inputting the three-dimensional coordinate values ​​of a first physical feature point that identifies the position of the first side of a body segment of the object and a second physical feature point that identifies the position of the second side, and the three-dimensional coordinate values ​​of multiple body surface points of the object. A plane calculation means that calculates a plane in three-dimensional space passing through the first physical feature point, and perpendicular to the line connecting the first physical feature point and the second physical feature point, using the three-dimensional coordinate values ​​of the first and second physical feature points. An extraction means for extracting a plurality of plane surface points from the plurality of body surface points whose distance from the plane is within a predetermined range, An approximation means for approximating the plurality of plane surface points of the body using the three-dimensional coordinate values ​​of the plurality of plane surface points of the body as a spatial ellipse which is an ellipse in three-dimensional space, A condition input means for inputting conditions indicating where the predetermined physical feature point is located on the spatial ellipse, A position estimation means for estimating the three-dimensional coordinate values ​​of a predetermined physical feature point based on the spatial ellipse and the conditions, This is a feature point position estimation device equipped with [a specific feature]. Since the body surface of human limbs and other body parts can be approximated by an ellipse, in this disclosure, the body surface (a collection of body surface points) is approximated by an ellipse.

[0011] In one embodiment, the approximation means finds an equation from which the three-dimensional coordinate values ​​of a point on the spatial ellipse can be obtained.

[0012] In one embodiment, the approximation means obtains, as equations from which the three-dimensional coordinate values ​​of a point on the spatial ellipse can be obtained, the equation of a two-dimensional ellipse that coincides with the spatial ellipse when moved in three-dimensional space, and a transformation equation that transforms a point on the two-dimensional ellipse into a point on the spatial ellipse.

[0013] In one embodiment, the position estimation means estimates the three-dimensional coordinate values ​​of the predetermined body feature point by inputting the coordinate values ​​on the ellipse in the two-dimensional space corresponding to the input conditions into the transformation equation.

[0014] In one embodiment, the condition is determined based on the position of the third physical feature point on an ellipse, assuming that the shape of the body surface of the target body segment in a plane passing through the first physical feature point and perpendicular to the line connecting the first physical feature point and the second physical feature point is approximated by an ellipse.

[0015] In one embodiment, the condition is a condition indicating that the predetermined physical feature point is one of the vertices of the spatial ellipse.

[0016] In one embodiment, the three-dimensional coordinate values ​​of the first physical feature point and the second physical feature point are coordinate values ​​obtained by a three-dimensional measuring device that measures the three-dimensional coordinate values ​​of predetermined physical feature points of an object.

[0017] In one embodiment, the system further comprises coordinate system generation means for generating an orthogonal coordinate system of the target body segment using the three-dimensional coordinate values ​​of the first and second physical feature points and the estimated three-dimensional coordinate values ​​of the predetermined physical feature points.

[0018] In one embodiment, the system further includes angle calculation means for calculating joint angles associated with the movement of the target body segment by calculating the Euler angles between the Cartesian coordinate system of the target body segment and another Cartesian coordinate system.

[0019] In one embodiment, the target segment is one of several target segments. The position input means, the plane calculation means, the approximation means, the condition input means, the feature point position estimation means, and the coordinate system generation means each process not only the target segment but also other segments from among the multiple target segments that are different from the target segment. The angle calculation means calculates the joint angle associated with the movement of the target body segment by calculating the Euler angle between the Cartesian coordinate system of the target body segment and the Cartesian coordinate system of the other body segments.

[0020] In one embodiment, the body segment is a foot, The first physical characteristic point mentioned above is the toes, The second physical feature mentioned above is the ankle joint, The aforementioned predetermined physical characteristic point is the widthwise end of the toe side of the foot.

[0021] In one embodiment, the predetermined physical feature is the fifth metatarsal bone, The aforementioned condition is that the fifth metatarsal bone, which is the predetermined physical characteristic point, is the vertex on the lateral side of the major axis of the spatial ellipse. The position estimation means estimates the three-dimensional coordinate values ​​of the fifth metatarsal bone, which is the predetermined physical characteristic point.

[0022] The present invention A computer that estimates the 3D coordinate values ​​of predetermined physical feature points located on the target body segment of an object, A position input step includes inputting the three-dimensional coordinate values ​​of a first physical feature point that identifies the position of the first side of the body segment of the object, and a second physical feature point that identifies the position of the second side, and the three-dimensional coordinate values ​​of multiple body surface points of the object. A plane calculation step in which a plane in three-dimensional space passing through the first physical feature point, and perpendicular to the line connecting the first physical feature point and the second physical feature point, is calculated using the three-dimensional coordinate values ​​of the first physical feature point and the second physical feature point; An extraction step of extracting a plurality of plane-based body surface points from the plurality of body surface points whose distance from the plane is within a predetermined range, An approximation step in which the plurality of plane surface points are approximated by a spatial ellipse, which is an ellipse in three-dimensional space, using the three-dimensional coordinate values ​​of the plurality of plane surface points, A condition input step involves inputting a condition indicating the location on the spatial ellipse where the predetermined physical feature point is located, A position estimation step in which the three-dimensional coordinate values ​​of the predetermined physical feature points are estimated based on the spatial ellipse and the conditions, It can be defined as a feature point location estimation method that performs the following:

[0023] The present invention A computer that estimates the 3D coordinate values ​​of predetermined physical feature points located at specific body segments of an object. A position input means for inputting the three-dimensional coordinate values ​​of a first physical feature point that identifies the position of the first side of a body segment of the object and a second physical feature point that identifies the position of the second side, and the three-dimensional coordinate values ​​of multiple body surface points of the object. A plane calculation means that calculates a plane in three-dimensional space passing through the first physical feature point, and perpendicular to the line connecting the first physical feature point and the second physical feature point, using the three-dimensional coordinate values ​​of the first and second physical feature points. An extraction means for extracting a plurality of plane surface points from the plurality of body surface points whose distance from the plane is within a predetermined range, An approximation means for approximating the plurality of plane surface points of the body using the three-dimensional coordinate values ​​of the plurality of plane surface points of the body as a spatial ellipse which is an ellipse in three-dimensional space, A condition input means for inputting conditions indicating where the predetermined physical feature point is located on the spatial ellipse, A position estimation means for estimating the three-dimensional coordinate values ​​of a predetermined physical feature point based on the spatial ellipse and the conditions, It can be defined as a program designed to function as such.

[0024] This disclosure can be understood as a device, system, method performed by a computer, or a program to be performed by a computer. Furthermore, this disclosure can be understood as such a program recorded on a recording medium readable by a computer, other device, machine, etc. Here, a recording medium readable by a computer, etc., means a recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical means and can be read by a computer, etc. [Effects of the Invention]

[0025] According to this disclosure, it becomes possible to estimate the location of physical characteristic points. [Brief explanation of the drawing]

[0026] [Figure 1] This is a schematic diagram showing the configuration of the system according to the embodiment. [Figure 2] This figure shows a schematic representation of the functional configuration of the feature point position estimation device according to the embodiment. [Figure 3] This figure shows an example of a foot according to the embodiment. [Figure 4] This figure shows an example of joint angle calculation according to the embodiment. [Figure 5] This is a flowchart illustrating the overview of the feature point location estimation process according to the embodiment. [Figure 6] This is a flowchart outlining the angle calculation process according to the embodiment. [Modes for carrying out the invention]

[0027] Hereinafter, embodiments of the apparatus, method, and program relating to this disclosure will be described with reference to the drawings. However, the embodiments described below are illustrative and not limited to the specific configurations of the apparatus, method, and program relating to this disclosure. In implementation, specific configurations may be adopted as appropriate depending on the manner of implementation, and various improvements and modifications may be made.

[0028] This embodiment describes an embodiment in which the apparatus, method, and program according to this disclosure are implemented in a feature point location estimation device for estimating the location of physical feature points on a subject's foot. However, the apparatus, method, and program according to this disclosure can be broadly used for techniques for estimating the location of arbitrary physical feature points of an object, and the scope of application of this disclosure is not limited to the examples shown in the embodiment.

[0029] <System Configuration> Figure 1 is a schematic diagram showing the configuration of the system according to this embodiment. As shown in Figure 1, the system 9 according to this embodiment consists of a three-dimensional measuring device 3 that acquires (measures) coordinate data (two-dimensional image coordinates and depth information) of surface points (observation points) of an object and coordinate data (three-dimensional coordinates) of the positions of physical feature points such as joints, and a feature point position estimation device 1 that records the coordinate data acquired by the three-dimensional measuring device 3 and estimates the position of the target physical feature point (reference point) using the coordinate data. Here, the object is not limited to a person such as a subject, but may be any animal or robot, etc., as long as it has physical feature points such as bones and joints. In this embodiment, the case in which the object is a subject (person) is given as an example. In this embodiment, physical feature points refer to feature points related to the skeleton of the object, such as bones and joints, and are exemplified by the head, shoulders, trunk, waist (pelvis), hands, toes, and joints of the limbs.

[0030] The three-dimensional measurement device 3 includes a distance image sensor (depth sensor) that acquires a distance image (depth image), which is an image containing distance information to the object (subject) (an image in which each pixel has distance information in the depth direction), and a position acquisition unit that acquires the positions (3D coordinate values) of predetermined physical feature points (joints, etc.) of the object (subject). The three-dimensional measurement device 3 may also include a video camera (color camera (RGB camera)) that acquires a color image (RGB image) of the object (subject). For example, the three-dimensional measurement device 3 is exemplified by Kinect®, a motion capture device equipped with a distance image sensor and a video camera. However, the three-dimensional measurement device 3 is not limited to Kinect, and may be any device that acquires the positions of physical feature points such as joints. Furthermore, the specific hardware configuration of the three-dimensional measurement device 3 can be omitted, replaced, or added as appropriate depending on the manner of implementation.

[0031] A distance image sensor is a sensor capable of measuring the distance (depth) to an observation point on an object, and mainly comprises a light projector that emits infrared light and a camera (infrared camera) that receives the reflected infrared light. In this embodiment, the observation point on the object is a point on the surface (body surface) of the object whose distance is measured by the distance image sensor, and is a point corresponding to each pixel of the distance image acquired by the distance image sensor. In the distance image sensor, the distance to the object (observation point) is mainly measured by the TOF method or the pattern irradiation method. The TOF method is a method of measuring the distance to the object (observation point) based on the time it takes for the emitted (projected) infrared laser to travel to and from the object (observation point) (the time from irradiation to receiving the reflected light). The pattern irradiation method is a method of measuring the distance to the object (observation point) by irradiating with an infrared laser having a specific pattern and analyzing the distortion of the pattern of the reflected light. In this embodiment, any method may be used for the distance image sensor to measure the distance to the object.

[0032] A depth image sensor measures the distance to an object and acquires (captures) a depth image in which each pixel has information about the distance (depth value) from the depth image sensor (camera) to the point (observation point) corresponding to that pixel as its pixel value. The depth image contains the two-dimensional image coordinate values ​​(position coordinates in the depth image) (u,v) in the uv-orthogonal coordinate system for each observation point, and the distance (depth value) d from the depth image sensor corresponding to those two-dimensional image coordinate values ​​(u,v). Distance d is the distance between the depth image sensor and the object. Hereinafter, (u,v,d) will be referred to as the "two-dimensional coordinate values" for each observation point (pixel).

[0033] The position acquisition unit uses the distance image acquired by the distance image sensor to determine the position (3D coordinate value (X) of predetermined body feature points (joints, bones, etc.) in the three-dimensional measurement device (distance image sensor). c ,Y c ,Z c The position acquisition unit acquires the 3D coordinate values ​​of multiple body feature points (for example, 25 body feature points) that are predetermined body feature points, such as the head, shoulders, trunk, waist (pelvis), fingertips, toes, and joints of the limbs. In this embodiment, in order to estimate the body feature points in the foot, the position acquisition unit acquires the 3D coordinate values ​​of at least the ankle joint and the toes (near the second metatarsal bone). In this embodiment, the 3D measurement device 3 (position acquisition unit) shows an example in which the 3D coordinate values ​​of the toes are acquired by defining the area near the head of the second metatarsal bone as the toes, but the 3D coordinate values ​​of the toes may also be acquired by defining a part other than the area near the second metatarsal bone (for example, near the third metatarsal bone) as the toes.

[0034] Furthermore, various arbitrary methods may be used to acquire the positions of predetermined body feature points in the three-dimensional measurement device 3 based on the distance image. For example, the method disclosed in Patent Document 1 or a method of estimation by inputting the distance information of the object into a trained model (classifier) ​​may be used. In addition, the 3D coordinate values ​​(X) acquired by the position acquisition unit c ,Y c ,Z c) are coordinate values in a three-dimensional orthogonal coordinate system (the three-dimensional coordinate system of the distance image sensor) with the distance image sensor (infrared camera) as the origin.

[0035] When the three-dimensional measurement device 3 includes a video camera that acquires the RGB image of the object, the positions of the body feature points acquired by the position acquisition unit may be drawn on the RGB image. However, the distance image sensor (infrared camera) and the video camera are each associated with an independent two-dimensional coordinate system. In other words, the distance image is defined by the two-dimensional coordinate system of the distance image (distance image sensor), and the RGB image is defined by the two-dimensional coordinate system of the RGB image (video camera). The same applies to the three-dimensional coordinate system. Therefore, when drawing the three-dimensional coordinate values acquired by the position acquisition unit on the RGB image, it is necessary to convert the three-dimensional coordinate values of the positions of the body feature points into the coordinate values on the RGB image by using a conversion function capable of converting coordinates between coordinate systems. Note that the conversion of coordinate systems is also described in Non-Patent Document 2, and any coordinate conversion may be performed.

[0036] From this, the system 9 according to the present embodiment includes the three-dimensional measurement device 3 described above, so that the two-dimensional coordinate values (u, v, d) for a plurality of observation points on the surface (body surface) of the object (subject), and the three-dimensional coordinate values (X c , Y c , Z c ) of the positions of the predetermined body feature points of the object (subject) are acquired.

[0037] The various calculations performed in the three-dimensional measuring device 3 are carried out by a computer equipped with a processor and memory. For example, the three-dimensional measuring device 3 includes an input unit for acquiring information obtained by cameras, sensors, etc., a storage unit for storing the information acquired by the input unit and the information calculated by the processing unit, and a processing unit for performing various processes on the information acquired by the input unit. Each of these functional units (input unit, storage unit, processing unit) is executed by a general-purpose processor, but some or all of these functions may be executed by one or more dedicated processors. Furthermore, some or all of these functions may be executed by devices installed in remote locations or by multiple distributed devices using cloud technology, etc.

[0038] The feature point location estimation device 1 is a computer equipped with a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, storage devices 14 such as EEPROM (Electrically Erasable and Programmable Read Only Memory) and HDD (Hard Disk Drive), communication devices 15 such as a NIC (Network Interface Card), input devices 16 such as a keyboard, and output devices 17 such as a display. However, the specific hardware configuration of the feature point location estimation device 1 can be omitted, replaced, or added as appropriate depending on the implementation. Furthermore, the feature point location estimation device 1 is not limited to a device consisting of a single enclosure. The feature point location estimation device 1 may be implemented by multiple devices using so-called cloud or distributed computing technologies.

[0039] The feature point position estimation device 1 is a device that estimates the position of a desired (predetermined) physical feature point based on the coordinate positions of observation points on the surface (body surface) of an object (subject) and the coordinate positions of predetermined physical feature points of the object, which are acquired by the three-dimensional measurement device 3. In this embodiment, an example is given in which the feature point position estimation device 1 estimates the position of the fifth metatarsal bone in the foot based on the coordinate positions of the ankle joint and toes of the subject and the coordinate positions of multiple body surface points (observation points) on the foot. In this embodiment, the feature point position estimation device 1 and the three-dimensional measurement device 3 are shown as separate devices (separate housings), but the system 9 is not limited to this example, and the system 9 may be equipped with a single device (housing) that has the functions of the feature point position estimation device 1 and the functions of the three-dimensional measurement device 3.

[0040] Figure 2 is a diagram illustrating the schematic functional configuration of the feature point location estimation device according to this embodiment. The feature point location estimation device 1 functions as a device comprising a location input unit 21, a storage unit 22, a condition input unit 23, a plane calculation unit 24, an extraction unit 25, an approximation unit 26, a location estimation unit 27, a coordinate system generation unit 28, and an angle calculation unit 29, by having a program recorded in the storage device 14 read into the RAM 13 and executed by the CPU 11, thereby controlling each hardware component of the feature point location estimation device 1. In this embodiment and other embodiments described later, each function of the feature point location estimation device 1 is executed by the CPU 11, which is a general-purpose processor, but some or all of these functions may be executed by one or more dedicated processors. Furthermore, each functional component of the feature point location estimation device 1 is not limited to being implemented in a device consisting of a single enclosure (one device), but may be implemented remotely and / or in a distributed manner (for example, on the cloud).

[0041] The position input unit 21 receives coordinate data of multiple observation points on the subject's body surface and of physical feature points predetermined (set) in the subject's three-dimensional measurement device 3 (distance image sensor). Specifically, the position input unit 21 receives the three-dimensional coordinate values ​​of multiple body surface points (observation points) of the subject, and the three-dimensional coordinate values ​​of the first physical feature point, which identifies the position of the first side of the subject's target body segment, and the second physical feature point, which identifies the position of the second side, from among the physical feature points predetermined in the three-dimensional measurement device 3. Here, the human body is mainly classified into multiple body segments (parts) such as the head, trunk (upper trunk, middle trunk, lower trunk), upper arm, forearm, hand, thigh, lower leg, and foot, and there are joints that connect the body segments. Hereinafter, the body segment in which the physical feature point whose position you want to estimate (hereinafter referred to as the "third physical feature point") is located will be referred to as the "target body segment".

[0042] The position input unit 21 first acquires two-dimensional coordinate values ​​(u,v,d) for multiple body surface points of the subject in at least one body segment from the three-dimensional measurement device 3. In this embodiment, the position input unit 21 acquires two-dimensional coordinate values ​​for multiple body surface points of the subject in the foot (target body segment) where the fifth metatarsal bone (third physical feature point) is located. Then, the position input unit 21 uses a conversion function stored in the memory unit 22, which will be described later, to convert the two-dimensional coordinate values ​​(u,v,d) of the subject's body surface points to the coordinate values ​​(3D coordinate values ​​(X) of the distance image sensor's three-dimensional coordinate system. c ,Y c ,Z c The system converts to ).)) Then, the position input unit 21 acquires (inputs) 3D coordinate values ​​for at least multiple body surface points in the target body segment among the body surface points of the subject. Any method may be used to convert from 2D coordinate values ​​to 3D coordinate values, for example, the DLT method (Direct Linear Transformation method) may be used.

[0043] Furthermore, the position input unit 21 receives 3D coordinate values ​​(X) from the 3D measurement device 3 for the positions of the first physical feature point, which identifies the position of the first side of the target body segment of the object (subject), and the second physical feature point, which identifies the position of the second side.c ,Y c ,Z c The input is obtained by acquiring the following. In this embodiment, the position input 21 is exemplified as inputting the 3D coordinate values ​​of the toe, which identifies the position of the first side of the foot (target body segment) where the fifth metatarsal bone (third body feature point) is located, and the ankle joint, which identifies the position of the second side of the foot. The body feature point that is close to the third body feature point is designated as the first body feature point, and the body feature point that is not close is designated as the second body feature point. Therefore, in this embodiment, the first body feature point is designated as the toe that is close to the fifth metatarsal bone, and the second body feature point is designated as the ankle joint. The following is an example of the 3D coordinate values ​​of the first body feature point input by the position input unit 21, P1((P1=(p 1x ,p 1y ,p 1z Let )) be the 3D coordinates of the second physical feature point be P2((P2=(p 2x ,p 2y ,p 2z ))

[0044] In this embodiment, the position input unit 21 inputs the 3D coordinate values ​​of each point by acquiring coordinate data from the 3D measuring device 3. However, the method of acquiring coordinate data is not limited to the example described above. The coordinate data may also be acquired from a recording device 14 or an external recording medium (not shown) that has previously stored measured (acquired) coordinate data. Furthermore, the position input unit 21 may acquire only the 3D coordinate values ​​of the first and second body feature points, rather than acquiring the 3D coordinate values ​​of all predetermined body feature points within the distance image sensor.

[0045] The memory unit 22 stores the conversion function. The conversion function converts the 2D coordinate values ​​(u,v,d) acquired by the distance image sensor into 3D coordinate values ​​(X) in a 3D Cartesian coordinate system with the distance image sensor (infrared camera) as the origin. c ,Y c ,Z cThis is a function for converting to ). The memory unit 22 stores a conversion function used, for example, when acquiring the 3D coordinate values ​​of the positions of predetermined body feature points such as joints in a three-dimensional measuring device 3 (distance image sensor).

[0046] The condition input unit 23 inputs feature point determination conditions used to estimate (determine) the position of the third body feature point. The feature point determination conditions indicate (specify) where the third body feature point located in the target segment is located on a spatial ellipse (an approximate ellipse in 3D space) that approximates the surface points of the target segment. For example, the feature point determination condition specifies that the third body feature point is one of the four vertices of the spatial ellipse (two vertices on the major axis and two vertices on the minor axis). However, the position of the third body feature point on the spatial ellipse specified in the feature point determination condition is not limited to the vertices of the spatial ellipse, but may be any point on the spatial ellipse other than the vertices. As described above, since the surface of human limbs, etc., can be approximated by an ellipse, in this embodiment, the surface points are approximated by an ellipse. Details of the spatial ellipse that approximates the surface points of the target segment will be described later.

[0047] The feature point determination condition is determined by the shape of the body segment (shape of the body surface) of the target where the third body feature point is located. For example, the feature point determination condition is determined by assuming that the shape of the body surface (cross-sectional shape) of the target body segment in a plane perpendicular to the line line connecting the first and second body feature points, passing through the first body feature point, is approximated by an ellipse. The position of the third body feature point on this ellipse is then determined by a user, such as an operator performing motion analysis. The determined feature point determination condition is then input by the user to the feature point position estimation device 1, allowing the condition input unit 23 to acquire the feature point determination condition.

[0048] For example, if the third physical feature point is the fifth metatarsal bone, the shape of the body surface of the target segment (foot) in a plane perpendicular to the line passing through the first physical feature point (toes) and connecting the first physical feature point (toes) and the second physical feature point (ankle joint) can be determined (discriminated) to approximate the shape of a transversely elongated ellipse (transverse arch shape) when the foot is viewed from the front (toe side). Therefore, the feature point determination condition for the fifth metatarsal bone is determined (set) as the vertex of the major axis of the spatial ellipse (the vertex in the lateral direction of the body). Specifically, the feature point determination condition for the fifth metatarsal bone of the right foot is set to the vertex of the major axis of the spatial ellipse that is in the + direction of the left-right axis (Y-axis) in the 3D Cartesian coordinate system of the target segment (see Figure 4), which will be described later. On the other hand, the feature point determination condition for the fifth metatarsal bone of the left foot is set to the vertex in the - direction of the same left-right axis.

[0049] The method by which the condition input unit 23 acquires the feature point determination conditions is not limited to the example described above. It may also acquire feature point determination conditions that have been previously determined (set) and saved by the user from the recording device 14 or an external recording medium (not shown). Furthermore, although the above example shows the feature point determination conditions being determined by the user, the position of the third body feature point on the ellipse may be automatically determined based on the name and other attributes of the third body feature point, and the feature point determination conditions may be determined accordingly.

[0050] The plane calculation unit 24 calculates (generates) a plane in three-dimensional space that passes through the first physical feature point and is perpendicular to the line connecting the first and second physical feature points (plane A). In this embodiment, first the plane calculation unit 24 calculates the three-dimensional coordinate value P1((P1=(p 1x ,p 1y ,p 1z )) and the 3D coordinate value P2((P2=(p 2x ,p 2y ,p 2z )) by which a unit direction vector V(V=(v) is drawn from the second body feature point to the first body feature point. x ,v y ,v zThe plane calculation unit 24 then calculates the equation of a plane A (a plane whose normal vector is the unit direction vector V) that passes through the 3D coordinate value P1 of the first physical feature point and is perpendicular to the unit direction vector V, using the following equation (1).

[0051]

number

[0052] Figure 3 shows an example of a foot according to this embodiment. As shown in Figure 3, in this embodiment, the equation of plane A perpendicular to the vector V that passes through the toes (first physical feature point) and points from the ankle joint (second physical feature point) to the toes (first physical feature point) is calculated using equation (1). Note that the normal vector used when determining plane A is not limited to a unit vector, as long as it is a vector pointing from the second physical feature point to the first physical feature point (or from the first physical feature point to the second physical feature point).

[0053] The extraction unit 25 extracts, from among multiple surface points (observation points) on the target body segment, surface points whose distance from plane A calculated (generated) by the plane calculation unit 24 is within a predetermined range, as surface points on plane A (hereinafter referred to as "surface points on the plane"). Specifically, the extraction unit 25 uses the equation of plane A (equation (1)) and the three-dimensional coordinate values ​​(X) of each surface point of the subject input by the position input unit 21. c ,Y c ,Z c The distance between plane A and each body surface point is calculated using (x). Then, the extraction unit 25 considers body surface points (body surface points close to plane A) for which the calculated distance is within a predetermined range (for example, a few millimeters) as body surface points on plane A and extracts them as body surface points on the plane. The coordinate values ​​of each point of the body surface points on the plane extracted in this way are calculated using (x j ,y j ,z j Let j be a natural number from 2 to N, and N be the number of points on the surface of the body in the plane, and the points on the surface of the body in the plane (x j ,y j ,z j The set of (x) will be referred to as "point group A" below. j ,y j ,zj ) is the 3D coordinate value (X) obtained by the position input unit 21 for a body surface point considered to be a body surface point on plane A. c ,Y c ,Z c )

[0054] The approximation unit 26 calculates the 3D coordinate values ​​(x) of the multiple plane surface points (point cloud A) extracted by the extraction unit 25. j ,y j ,z j The approximation unit 26 approximates the point group A (the body surface on which the point group A is located) with a spatial ellipse, which is an ellipse in three-dimensional space. By approximating the point group A with a spatial ellipse, the approximation unit 26 makes it possible to obtain the three-dimensional coordinate values ​​of the points on the spatial ellipse that approximates the point group A. In this embodiment, the approximation unit 26 finds an equation that allows obtaining the three-dimensional coordinate values ​​of the points on the spatial ellipse that approximates the point group A.

[0055] Here, an ellipse in three-dimensional space (spatial ellipse) can be represented as an ellipse obtained by performing rotational and translational translations on a flat ellipse with its center at the origin in two-dimensional space (XY plane) (hereinafter referred to as a "planar ellipse"). In other words, by performing rotational and translational translations on a point on a planar ellipse, it is possible to move (transform coordinates) a point on a spatial ellipse to a point on a spatial ellipse. Therefore, in this embodiment, the spatial ellipse approximating the point group A is assumed to be a planar ellipse moved into three-dimensional space, and the approximation unit 26 obtains the equation of the planar ellipse that coincides with the spatial ellipse when moved into three-dimensional space, and the equation that transforms a point on a planar ellipse to a point on a spatial ellipse (hereinafter referred to as the "transformation equation"), as equations that can obtain the three-dimensional coordinate values ​​of a point on the spatial ellipse.

[0056] In this embodiment, the equation of the planar ellipse is the equation shown in equation (2) below. In equation (2), a and b are either the semi-major axis and the semi-minor axis of the ellipse, respectively, and S is the scaling factor (including 1) of the spatial ellipse relative to an ellipse with area 1, with abπ=1 and z=0.

[0057]

number

[0058] Furthermore, if a point Q' on a planar ellipse is subjected to rotation by a rotation matrix M and translation by a translation vector T, resulting in point Q' being moved (transformed) to point Q on a spatial ellipse, then the 3D coordinate values ​​(x,y,z) of point Q on the spatial ellipse will be the same as the coordinate values ​​(x) of point Q'. ’’ ,y ’’ ,z ’’ ), rotation matrix M and translation vector T(T=(x ’’’ ,y ’’’ ,z ’’’ Using ), it is calculated by the following equation (3). Note that equation (3) is the equation (transformation equation) for transforming (moving) a point on a planar ellipse to a point on a spatial ellipse, as described above.

[0059]

number

[0060] In this embodiment, the rotation matrix M is set to an arbitrary axis (unit vector n = (n x ,n y ,n z This is the rotation matrix for rotating by α around the given element, and is shown by equation (4) below.

[0061]

number

[0062] In this embodiment, in order to obtain the equation of the planar ellipse that coincides with the spatial ellipse by moving in three-dimensional space as described above (equation (2)) and the transformation equation (equation (3)), the point group A' (the point group before rotation and translation) that moves to point group A by rotation by rotation matrix M and translation by translation vector T is optimized to approximate with a planar ellipse. In other words, it is optimized to minimize the error between point group A' and the equation of the planar ellipse. The approximation unit 26 first takes the coordinate values ​​(x) of each point in point group A into equation (5) below. j ,y j ,z jBy inputting (x), the coordinate values ​​of each point in point cloud A are obtained, and the coordinate values ​​of each point in point cloud A' are obtained. j ’’ ,y j ’’ ,z j ’’ Convert to (x). Note that in equation (5), M is a rotation matrix that represents the amount that point group A' rotated to move to point group A, and (x ’’’ ,y ’’’ ,z ’’’ ) is a translation vector that represents the amount of translation (parallel movement) that point group A' undergoes to move to point group A.

[0063]

number

[0064] Then, the approximation part 26 is the coordinate value (x) of the point group A' shown by equation (5). j ’’ ,y j ’’ ,z j ’’ The equation of the plane ellipse and the transformation equation are obtained by optimizing to minimize the error between the point group A' and the plane ellipse using the equation of the plane ellipse shown by equation (2). In this embodiment, the sum of the absolute values ​​of the errors between each point of point group A' and the plane ellipse, shown in equation (6) below, is used as the error (error function) between the point group A' and the plane ellipse. The approximation unit 26 optimizes each parameter (determination of optimal parameters) by optimizing to minimize the error function shown in equation (6). The parameters optimized to minimize the error function shown in equation (6) are each component (x) of the translation vector. ’’’ ,y ’’’ ,z ’’’ ), each component of the unit vector n (n x ,n y ,n z ), α, a, b, S.

[0065]

number

[0066] Thus, the approximation part 26 optimizes to minimize the error function shown in Equation (6), so that each parameter (x ’’’ , y ’’’ , z ’’’ , n x , n y , n z , α, a, b, S) of the plane ellipse that best approximates the point group A’ can be calculated (determined). From this, the equation of the plane ellipse (Equation (2)) and the transformation equation (Equation (3)) can be calculated, and the point group A can be approximated by a spatial ellipse. Note that any optimization method may be used to minimize the error function shown in Equation (6). For example, a method using a solver, which is a kind of function of spreadsheet software, may be used. Further, the error function to be minimized is not limited to the error function shown in Equation (6), and any error indicating the error between the point group A and the spatial ellipse or the error between the point group A’ and the plane ellipse may be used. For example, it may be the sum of squares of the errors (sum of squared errors) between the point group A’ and the plane ellipse.

[0067] Also, in the present embodiment, by obtaining the equation of the plane ellipse and the transformation equation, the three-dimensional coordinate values of the points on the spatial ellipse approximating the point group A can be obtained. However, any method such as directly obtaining the equation of the spatial ellipse may be used as long as it is a method capable of obtaining the three-dimensional coordinate values on the spatial ellipse. Further, the point group A may be approximated by a spatial ellipse under the constraint condition that the spatial ellipse exists on the plane A.

[0068] The position estimation unit 27 estimates the position (3D coordinate values) of the third body feature point based on the spatial ellipse approximating the point cloud A and the feature point determination conditions (calculation of estimated coordinate values). In this embodiment, the position estimation unit 27 calculates coordinate values ​​on the spatial ellipse by inputting the coordinate values ​​on the plane ellipse corresponding to the feature point determination conditions into a transformation equation (equation (3)) that converts points on the plane ellipse to points on the spatial ellipse, and determines the calculated coordinate values ​​on the spatial ellipse as the estimated coordinate values ​​(3D coordinate values) of the third body feature point. Specifically, the position estimation unit 27 calculates (determines) the coordinate values ​​(p,q,r) on the plane ellipse of the third body feature point corresponding to the feature point determination conditions. Then, the position estimation unit 27 uses the calculated (p,q,r) as the coordinate values ​​(x) on the plane ellipse in equation (3). ’’ ,y ’’ ,z ’’ By inputting the values ​​into the input field, the coordinate values ​​(x,y,z) on the spatial ellipse are calculated, and these calculated coordinate values ​​(x,y,z) on the spatial ellipse are determined as the 3D coordinate values ​​(estimated coordinate values) of the third body feature point.

[0069] As described above, when the fifth metatarsal bone in the foot is the third body feature point, the feature point determination condition is set to the "vertex of the major axis (vertex in the direction of the outside of the body)" of a spatial ellipse that approximates the body surface points of the foot (see Figure 3). Therefore, for example, if the x-axis in 2D space is the major axis direction of the planar ellipse (a>b), and the positive direction of the x-axis is the right horizontal direction as viewed from the distance image sensor, the position estimation unit 27 calculates (determines) the coordinate value (p,q,r) of the fifth metatarsal bone of the right foot on the planar ellipse using (p,q,r)=(-aS,0,0), and calculates the coordinate value (p,q,r) of the fifth metatarsal bone of the left foot on the planar ellipse using (p,q,r)=(aS,0,0). Here, S is the magnification ratio based on an ellipse with an area of ​​1, as described above. Then, the position estimation unit 27 calculates the coordinate (x) on the planar ellipse in equation (3). ’’ ,y ’’ ,z ’’ By inputting the coordinate values ​​(aS,0,0) and (-aS,0,0) on a planar ellipse corresponding to the characteristic point determination conditions of the fifth metatarsal bone, respectively, the 3D coordinate values ​​(x,y,z) of the fifth metatarsal bones of the right and left feet are calculated.

[0070] For example, if the feature point determination condition is that the third body feature point is at the vertex of the minor axis of a spatial ellipse, and the x-axis in 2D space is the major axis direction of the planar ellipse (a>b), then the coordinate values ​​(p,q,r) of the third body feature point on the planar ellipse are calculated (determined) by (p,q,r)=(0,bS,0) or (0,-bS,0). Also, if the equation of the spatial ellipse is directly calculated by the approximation unit 26, the 3D coordinate values ​​of the third body feature point may be calculated (estimated) using the spatial ellipse equation and the feature point determination condition. In this embodiment, by obtaining the equation of the planar ellipse and the transformation equation as equations from which the 3D coordinate values ​​of a point on the spatial ellipse can be obtained, it becomes possible to specify the position of the desired third body feature point as a coordinate value on the planar ellipse (for example, (aS,0,0)).

[0071] The coordinate system generation unit 28 generates a coordinate system (orthogonal coordinate system) for a body segment by determining the three axes of that segment. The coordinate system generation unit 28 generates the coordinate system for the target body segment using the three-dimensional coordinate values ​​of the first and second body feature points and the estimated three-dimensional coordinate value of the third body feature point. For example, the coordinate system generation unit 28 determines the unit vector from the second body feature point to the first body feature point as the first axis, the unit normal vector of the plane passing through the first body feature point, the second body feature point, and the estimated third body feature point as the second axis, and the unit vector perpendicular to the first and second axes as the third axis. For example, if the third body feature point is the fifth metatarsal bone, the unit vector from the ankle joint to the toes is determined as the first axis, the unit normal vector of the plane passing through the ankle joint, the toes, and the estimated fifth metatarsal point is determined as the second axis, and the unit vector perpendicular to the first and second axes is determined as the third axis.

[0072] The coordinate system generation unit 28 then determines (generates) a coordinate system consisting of the determined first axis, second axis, and third axis as the coordinate system for the target body segment. Note that the three axes in the coordinate system for the target body segment are not limited to the three axes shown above, but may be any three axes based on the first, second, and third body feature points. In addition, the coordinate system generation unit 28 may generate coordinate systems for other body segments besides the target body segment.

[0073] The angle calculation unit 29 calculates the joint angles associated with the movement of the target body segment by calculating the Euler angles between the coordinate system of the target body segment generated by the coordinate system generation unit 28 and another coordinate system (Cartesian coordinate system). Here, the movement of the target body segment refers to movements of the body segment such as varus, eversion, adduction, abduction, flexion, extension, internal rotation, external rotation, pronation, supination, plantarflexion, and dorsiflexion, and the joint angles associated with these movements are angles such as varus angle, eversion angle, adduction angle, abduction angle, flexion angle, extension angle, internal rotation angle, external rotation angle, pronation angle, supination angle, plantarflexion angle, and dorsiflexion angle. The desired joint angles can be calculated by calculating the Euler angles between the coordinate system of the target body segment generated by the coordinate system generation unit 28 and another appropriate coordinate system (reference coordinate system) for calculating the desired (predetermined) joint angles. The other coordinate system may be any coordinate system, such as the coordinate system of another body segment adjacent to the target body segment or the global coordinate system. Furthermore, for other coordinate systems, other coordinate systems stored in the memory device 14 may be used, or the coordinate systems of other body segments other than the target body segment, generated by the coordinate system generation unit 28, may be used.

[0074] The following example illustrates how to calculate the inversion and eversion angle of the foot (ankle joint) using the estimated 3D coordinate values ​​of the fifth metatarsal bone, where the third physical feature point is the fifth metatarsal bone. In this embodiment, the inversion and eversion angle of the ankle joint refers to the rotation angle around the axis of the second metatarsal bone from the heel.

[0075] Figure 4 shows an example of joint angle calculation according to this embodiment. Figure 4 shows the coordinate system (X-axis, Y-axis, Z-axis) of the target body segment (foot) and another coordinate system (reference coordinate system) consisting of three axes (X'-axis, Y'-axis, Z'-axis) determined by the three points of the hip joint, knee joint, and ankle joint. As shown in Figure 4, in this embodiment, the coordinate system of the target body segment is a coordinate system in which the plane passing through the first, second, and third physical feature points is the horizontal plane, and the axis from the second physical feature point toward the first physical feature point is the anterior-posterior axis (X-axis). Furthermore, in this embodiment, the reference coordinate system is a coordinate system of a virtual body segment (rigid body) in which the lower limb other than the foot is treated as a single body segment (rigid body). Specifically, the plane passing through the hip joint, knee joint, and ankle joint is the sagittal plane, and the direction from the ankle joint toward the knee joint is the vertical axis (Z'-axis). Furthermore, since the knee joint is a uniaxial joint, it is possible to treat the plane formed by the hip, knee, and ankle joints as a single virtual rigid body.

[0076] The angle calculation unit 29 calculates the Euler angle (rotation angle) between the coordinate system of the target body segment (foot) and the reference coordinate system, as shown in Figure 4. Specifically, the angle calculation unit 29 calculates the Euler angle required for the coordinate system of the target body segment to coincide with the coordinate system of the reference body segment by rotating around its own coordinate axes (X-axis (anterior-posterior axis), Y-axis (lateral axis), Z-axis (up-down axis)). In this embodiment, the angle calculation unit 29 calculates the Euler angle required to coincide with the coordinate system of the reference body segment by calculating a rotation matrix Q around three axes, where the rotation angle around the X-axis is φ, the rotation angle around the Y-axis is θ, and the rotation angle around the Z-axis is Ψ. From this, the angle calculation unit 29 can determine (calculate) the obtained rotation angle φ around the X-axis (anterior-posterior axis) as the inversion and eversion angle of the ankle joint. Note that while the rotation order described above was used for calculating Euler angles in this case, this is not the only rotation order that is permitted.

[0077] In this embodiment, the reference coordinate system used to calculate the Euler angles with respect to the target segment's coordinate system in order to calculate the inversion / eversion angles is determined by the three points of the hip, knee, and ankle joints. However, the method for determining the reference coordinate system is not limited to this method. For example, using the method for estimating the position of physical feature points described above, the coordinate position of the third physical feature point in a reference segment can be estimated based on the first physical feature point that identifies the position of the first side and the second physical feature point that identifies the position of the second side of another segment different from the target segment (hereinafter referred to as the "reference segment"). The coordinate system determined by the three points of the first, second, and third physical feature points of the reference segment may then be used as the reference coordinate system.

[0078] For example, by setting the first body feature point to the knee joint and the second body feature point to the ankle joint, and setting the feature point determination condition to the vertex of the minor axis (the vertex in the direction of the outside of the body), the position estimation unit 27 calculates the 3D coordinate values ​​of the lateral epicondyle of the knee joint. Then, the coordinate system generation unit 28 generates a reference coordinate system (the coordinate system of the lower leg, which is the reference body segment) using the three points of the knee joint, ankle joint, and lateral epicondyle of the knee joint, in addition to the coordinate system of the target body segment. Specifically, a coordinate system is generated in which the plane passing through the knee joint, ankle joint, and lateral epicondyle of the knee joint is the frontal plane, and the direction from the ankle joint to the knee joint is the vertical axis. Then, the angle calculation unit 29 calculates the Euler angle between the coordinate system of the target body segment generated by the coordinate system generation unit 28 and the coordinate system of the reference body segment, making it possible to calculate the inversion and eversion angles of the foot.

[0079] The angle calculation unit 29 may calculate the calculated Euler angle itself as the desired joint angle, or it may reverse the polarity (sign of positive or negative) of the calculated Euler angle and calculate the reversed angle as the desired joint angle. For example, the inversion and eversion angles of the ankle joint calculated by the method described above are calculated (output) with the anterior-posterior axis of the foot as positive for rotation from the ankle joint towards the toes, and positive for rotation in the right-hand screw direction. In this case, for the right foot, when everping, the rotation direction around the anterior-posterior axis of the foot (ankle joint) is negative, and when inverting, the rotation direction around the anterior-posterior axis of the foot is positive. On the other hand, for the left foot, when everping, the rotation direction around the anterior-posterior axis of the foot (ankle joint) is positive, and when inverting, the rotation direction around the anterior-posterior axis of the foot is negative. Therefore, if it is desired to calculate (output) an eversion angle with positive eversion, the angle calculation unit 29 calculates the eversion angle of the right foot by multiplying the calculated Euler angle of the right foot around the anterior-posterior axis by -1, and calculates the eversion angle of the left foot by multiplying the calculated Euler angle of the left foot around the anterior-posterior axis by +1. In this way, the angle calculation unit 29 may perform various processes to calculate the desired joint angle by matching the polarity of the desired joint angle with the polarity of the Euler angle between coordinate systems.

[0080] As described above, this embodiment illustrates a method for estimating the position (3D coordinate value) of the fifth metatarsal bone using the 3D coordinate values ​​of the ankle joint and toes acquired by the 3D measurement device 3. However, the object to be estimated (third body feature point) is not limited to the fifth metatarsal bone, but may be any other body feature point. For example, in this embodiment, by setting the feature point determination condition to the vertex of the major axis (vertex in the medial direction of the body) instead of the vertex of the major axis (vertex in the lateral direction of the body), the position of the first metatarsal bone can be estimated. Therefore, for example, if the coordinate value (p,q,r) on the planar ellipse corresponding to the feature point determination condition of the fifth metatarsal bone of the right foot is (p,q,r)=(-aS,0,0), then the coordinate value (p,q,r) on the planar ellipse corresponding to the feature point determination condition of the first metatarsal bone of the right foot will be (p,q,r)=(aS,0,0). In this way, by defining the target body segment as the foot, the first physical feature point as the toes, and the second physical feature point as the ankle joint, it becomes possible to estimate the position (3D coordinate value) of the widthwise end of the toe side of the foot.

[0081] Furthermore, by designating the ankle joint as the first physical feature point and the knee joint as the second physical feature point, the positions of the medial and lateral malleoli can be estimated. Similarly, by designating the wrist joint as the first physical feature point and the elbow joint as the second physical feature point, the positions of the radial and ulnar styloid processes can be estimated. Furthermore, by designating the knee joint as the first physical feature point and the hip joint as the second physical feature point, the positions of the medial and lateral femoral epicondyles can be estimated. Finally, by designating the elbow joint as the first physical feature point and the shoulder joint as the second physical feature point, the positions of the medial and lateral humeral epicondyles can be estimated. In each case, the feature point determination conditions can be appropriately set by those skilled in the art.

[0082] <Processing flow> Next, the flow of the feature point location estimation process performed by the feature point location estimation device 1 according to this embodiment will be described. Note that the specific content and processing order of the process described below is an example for implementing this disclosure. The specific content and processing order may be appropriately selected depending on the manner in which this disclosure is implemented.

[0083] Figure 5 is a flowchart illustrating the overview of the feature point position estimation process according to this embodiment. The process shown in this flowchart is executed in the feature point position estimation device 1 when it receives instructions from a user or other party to acquire coordinate data of an object (subject) or to estimate physical feature points. In the following example, the third physical feature point is the fifth metatarsal bone located in the foot, and the process for estimating the position of the fifth metatarsal bone is given as an example.

[0084] In step S101, the 3D coordinate values ​​of predetermined physical feature points (first physical feature point, second physical feature point) are input to the 3D measurement device 3. The position input unit 21 obtains and inputs the 3D coordinate values ​​of the toes (first physical feature point) that specify the position of the first side of the foot (target body segment) where the subject's fifth metatarsal bone is located, and the ankle joint (second physical feature point) that specifies the position of the second side, from the 3D measurement device 3. After that, the process proceeds to step S102.

[0085] In step S102, the 3D coordinate values ​​of the surface points (observation points) of the object (subject) are input. The position input unit 21 obtains 2D coordinate values ​​for multiple surface points on the subject's feet (target body segments) from the 3D measurement device 3, and uses a conversion function stored in the memory unit 22 to convert the 2D coordinate values ​​for those surface points into 3D coordinate values, thereby obtaining (inputting) the 3D coordinate values ​​of multiple surface points on the subject's feet. After that, the process proceeds to step S103.

[0086] In step S103, the feature point determination conditions are entered. The condition input unit 23 inputs the feature point determination conditions for the fifth metatarsal bone (third body feature point), indicating that the fifth metatarsal bone is located at the vertex of the major axis (vertex in the lateral direction of the body) of a spatial ellipse that approximates the surface points of the foot. After that, the process proceeds to step S104. Note that steps S101 to S103 can be performed in any order.

[0087] In step S104, a plane (plane A) is calculated that passes through the first physical feature point and is perpendicular to the line connecting the first and second physical feature points. The plane calculation unit 24 uses the 3D coordinate values ​​of the toe (first physical feature point) and ankle joint (second physical feature point) input in step S101 to calculate the equation of plane A, which passes through the toe and is perpendicular to the vector V that points from the ankle joint to the toe. The process then proceeds to step S105.

[0088] In step S105, body surface points (point cloud A) whose distance from plane A is within a predetermined range are extracted. The extraction unit 25 uses the equation of plane A (equation (1)) calculated in step S104 and the 3D coordinate values ​​of each body surface point input in step S102 to extract body surface points whose distance from plane A is within a predetermined range as plane surface points (point cloud A), which are body surface points on plane A. The process then proceeds to step S106.

[0089] In step S106, the point group A is approximated by a spatial ellipse. The approximation unit 26 approximates the point group A with a spatial ellipse and obtains equations that allow the acquisition of 3D coordinate values ​​of points on the spatial ellipse approximating the point group A. These equations include the equation of a planar ellipse that coincides with the spatial ellipse when moved in 3D space, and a transformation equation that converts points on the planar ellipse to points on the spatial ellipse. The process then proceeds to step S107.

[0090] In step S107, the position of the third body feature point is estimated based on the feature point determination conditions and the spatial ellipse. The position estimation unit 27 obtains the coordinate values ​​on the planar ellipse corresponding to the feature point determination conditions of the fifth metatarsal bone input in step S103, and calculates the 3D coordinate values ​​of the fifth metatarsal bone by inputting the obtained coordinate values ​​into the transformation equation obtained in step S106. The position estimation unit 27 then determines the calculated 3D coordinate values ​​as the estimated position of the fifth metatarsal bone. After that, the process shown in this flowchart is completed.

[0091] Figure 6 is a flowchart illustrating the overview of the angle calculation process according to this embodiment. The process shown in this flowchart is executed by the feature point position estimation device 1 when the position of the third body feature point in the target body segment is estimated, etc. In the following, the third body feature point is assumed to be the fifth metatarsal bone in the foot, and the process for calculating the inversion and eversion angles of the ankle joint is given as an example.

[0092] In step S201, a Cartesian coordinate system is generated for the target body segment. The coordinate system generation unit 27 determines the unit vector from the ankle joint to the toes as the first axis, the unit normal vector of the plane passing through the ankle joint, toes, and estimated fifth metatarsal point as the second axis, and the unit vector perpendicular to the first and second axes as the third axis. The coordinate system consisting of the first, second, and third axes is then generated as the coordinate system for the target body segment. The process then proceeds to step S202.

[0093] In step S202, the joint angle is calculated. The angle calculation unit 29 calculates the Euler angle between the coordinate system of the target body segment (foot) generated in step S201 and the reference coordinate system (see Figure 4), and determines (calculates) the rotation angle φ around the X-axis (anterior-posterior axis) from the calculated Euler angle as the inversion and eversion angle of the ankle joint. After that, the process shown in this flowchart is completed.

[0094] According to the system shown in this embodiment, by using the three-dimensional coordinate values ​​of a first physical feature point that identifies the position of the first side of a target segment of an object, a second physical feature point that identifies the position of the second side, the three-dimensional coordinate values ​​of multiple surface points on the target segment, and feature point determination conditions, it becomes possible to estimate the three-dimensional coordinate position of a third physical feature point located on the target segment.

[0095] As mentioned above, distance image sensors (three-dimensional measuring devices) can acquire the position coordinates of predetermined body feature points. For example, in the lower limbs, the 3D coordinate values ​​of the knee joint, ankle joint, and toes, which are typically set as body feature points within the distance image sensor, are output from the sensor. While it is possible to calculate the plantarflexion / dorsiflexion angles and adduction / abduction angles of the ankle joint using projection angles based on these 3D coordinate values ​​of body feature points, it is difficult to correctly calculate the inversion / eversion angles of the ankle joint. In order to calculate these ankle inversion / eversion angles, as mentioned above, it is necessary to determine the Euler angles between body segments (between the coordinate systems of body segments), and therefore it is necessary to acquire the 3D coordinate value of another point (body feature point) in the body segment (foot). Thus, in order to calculate joint angles around three axes, it is necessary to estimate the position of another body feature point (virtual point) in a body segment that only outputs the coordinate values ​​of two body feature points (for example, the foot, which only outputs the coordinate values ​​of the ankle joint and toes).

[0096] As described above, the system shown in this embodiment makes it possible to estimate the 3D coordinate position of a third body feature point located on a target body segment from the 3D coordinate values ​​of a first body feature point that identifies the position of the first side of the target body segment and a second body feature point that identifies the position of the second side of the target body segment. Therefore, in a body segment where only the 3D coordinate values ​​of two body feature points are output from the distance image sensor, it becomes possible to estimate the position of another body feature point (a body feature point not predetermined by the distance image sensor). From this, it becomes possible to generate a coordinate system for the body segment, and by calculating the Euler angle between the coordinate system of the body segment and other coordinate systems, it becomes possible to calculate joint angles that are difficult to calculate using projection angles. In this way, by making it possible to calculate joint angles that are normally difficult to calculate, it becomes possible to deepen our understanding of living organisms and contribute to rehabilitation medicine, orthopedic treatment, etc. [Explanation of Symbols]

[0097] 1. Feature Point Position Estimation Device 3. Three-dimensional measuring device 9 Systems

Claims

1. A feature point position estimation device that estimates the three-dimensional coordinate values ​​of a predetermined body feature point located in a body segment, taking as input the three-dimensional coordinate values ​​of a first and second body feature point of a target body segment acquired using a distance image sensor, and the three-dimensional coordinate values ​​of a plurality of body surface points of the target, The aforementioned segment is the foot, The first physical characteristic point mentioned above is the toes, The second physical feature mentioned above is the ankle joint, The aforementioned predetermined physical characteristic point is the widthwise end of the toe side of the foot, A position input means for inputting the three-dimensional coordinate values ​​of the first and second body feature points acquired by the distance image sensor, and the three-dimensional coordinate values ​​of the plurality of body surface points acquired by the distance image sensor, A plane calculation means that calculates a plane in three-dimensional space passing through the first physical feature point, and perpendicular to the line connecting the first physical feature point and the second physical feature point, using the three-dimensional coordinate values ​​of the first and second physical feature points, An extraction means for extracting a plurality of plane surface points from the plurality of body surface points whose distance from the plane is within a predetermined range, An approximation means for approximating the plurality of plane surface points of the body using the three-dimensional coordinate values ​​of the plurality of plane surface points of the body as a spatial ellipse which is an ellipse in three-dimensional space, A condition input means for inputting conditions indicating where the predetermined physical feature point is located on the spatial ellipse, A position estimation means for estimating the three-dimensional coordinate values ​​of the predetermined physical feature points based on the spatial ellipse and the conditions, A coordinate system generation means generates an orthogonal coordinate system of the target body segment using the three-dimensional coordinate values ​​of the first and second physical feature points and the estimated three-dimensional coordinate values ​​of the predetermined physical feature points. An angle calculation means for calculating joint angles associated with the movement of the target body segment by calculating the Euler angles between the Cartesian coordinate system of the target body segment and another Cartesian coordinate system determined by the three points of the hip, knee, and ankle joints, A feature point position estimation device equipped with the following features.

2. The aforementioned predetermined physical characteristic point is the fifth metatarsal bone, The aforementioned condition is that the fifth metatarsal bone, which is the predetermined physical characteristic point, is the vertex on the lateral side of the major axis of the spatial ellipse. The position estimation means estimates the three-dimensional coordinate values ​​of the fifth metatarsal bone, which is a predetermined physical characteristic point. The feature point position estimation device according to claim 1.

3. A method performed by a computer that takes as input the three-dimensional coordinate values ​​of a first and second physical feature point of a target body segment acquired using a distance image sensor, and the three-dimensional coordinate values ​​of a plurality of body surface points of the target, and estimates the three-dimensional coordinate values ​​of a predetermined physical feature point located in the body segment, The aforementioned segment is the foot, The first physical characteristic point mentioned above is the toes, The second physical feature mentioned above is the ankle joint, The aforementioned predetermined physical characteristic point is the widthwise end of the toe side of the foot, A position input step in which the three-dimensional coordinate values ​​of the first and second body feature points acquired by the distance image sensor, and the three-dimensional coordinate values ​​of the plurality of body surface points acquired by the distance image sensor are input, A plane calculation step in which a plane in three-dimensional space passing through the first physical feature point, and perpendicular to the line connecting the first physical feature point and the second physical feature point, is calculated using the three-dimensional coordinate values ​​of the first physical feature point and the second physical feature point; An extraction step of extracting a plurality of plane-based body surface points from the plurality of body surface points whose distance from the plane is within a predetermined range, An approximation step in which the plurality of plane surface points are approximated by a spatial ellipse, which is an ellipse in three-dimensional space, using the three-dimensional coordinate values ​​of the plurality of plane surface points. A condition input step involves inputting a condition indicating the location on the spatial ellipse where the predetermined physical feature point is located, A position estimation step in which the three-dimensional coordinate values ​​of the predetermined physical feature points are estimated based on the spatial ellipse and the conditions, A coordinate system generation step in which a Cartesian coordinate system of the target body segment is generated using the three-dimensional coordinate values ​​of the first and second physical feature points and the estimated three-dimensional coordinate values ​​of the predetermined physical feature points, An angle calculation step to calculate the joint angles associated with the movement of the target body segment by calculating the Euler angles between the Cartesian coordinate system of the target body segment and another Cartesian coordinate system determined by the three points of the hip, knee, and ankle joints, A feature point location estimation method that performs this operation.

4. A program that causes a computer to execute a program that estimates the three-dimensional coordinate values ​​of a predetermined physical feature point located in a body segment, taking as input the three-dimensional coordinate values ​​of a first physical feature point and a second physical feature point of a body segment of a target acquired using a distance image sensor, and the three-dimensional coordinate values ​​of a plurality of body surface points of the target, The aforementioned segment is the foot, The first physical characteristic point mentioned above is the toes, The second physical feature mentioned above is the ankle joint, The aforementioned predetermined physical characteristic point is the widthwise end of the toe side of the foot, The aforementioned computer, A position input means for inputting the three-dimensional coordinate values ​​of the first and second body feature points acquired by the distance image sensor, and the three-dimensional coordinate values ​​of the plurality of body surface points acquired by the distance image sensor, A plane calculation means that calculates a plane in three-dimensional space passing through the first physical feature point, and perpendicular to the line connecting the first physical feature point and the second physical feature point, using the three-dimensional coordinate values ​​of the first and second physical feature points, An extraction means for extracting a plurality of plane surface points from the plurality of body surface points whose distance from the plane is within a predetermined range, An approximation means for approximating the plurality of plane surface points of the body using the three-dimensional coordinate values ​​of the plurality of plane surface points of the body as a spatial ellipse which is an ellipse in three-dimensional space, A condition input means for inputting conditions indicating where the predetermined physical feature point is located on the spatial ellipse, A position estimation means for estimating the three-dimensional coordinate values ​​of the predetermined physical feature points based on the spatial ellipse and the conditions, A coordinate system generation means generates an orthogonal coordinate system of the target body segment using the three-dimensional coordinate values ​​of the first and second physical feature points and the estimated three-dimensional coordinate values ​​of the predetermined physical feature points. An angle calculation means for calculating joint angles associated with the movement of the target body segment by calculating the Euler angles between the Cartesian coordinate system of the target body segment and another Cartesian coordinate system determined by the three points of the hip, knee, and ankle joints, A program designed to function as such.

Images