Pose recognition system and pose recognition method

Abstract

To achieve highly accurate attitude estimation.SOLUTION: An attitude recognition system 150 recognizing the attitude of a subject comprises: a sensor attitude estimation unit 152 as a first attitude estimation unit that estimates the attitude of the subject on the basis of a result of sensing performed by an attitude sensor 111 that is a sensor amounted on the subject; a camera attitude estimation unit 153 as a second attitude estimation unit that estimates the attitude of the subject on the basis of an image obtained by photographing the subject; an attitude reliability estimation unit 154 as a second reliability estimation unit that determines, as second reliability, the reliability as to a second estimation result that is a result of estimation made by the second attitude estimation unit; and a sensor attitude correction unit 155 as a correction unit that corrects a first estimation result that is a result of estimation made by the first attitude estimation unit on the basis of the second estimation result and the second reliability.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a posture recognition system and a posture recognition method.

Background Art

[0002] Conventionally, in order to recognize the posture of a subject, there is a technique described in WO2021 / 075004 (Patent Document 1). In this publication, it is described that "at least information on the moving acceleration and the posture angular velocity of the mounting site to which each sensor is attached is acquired from a plurality of sensors, and based on the acquired information on the moving acceleration and the posture angular velocity, the moving velocity in a predetermined coordinate system of the mounting site to which each sensor is attached is estimated. Then, the position of each part of the subject determined in advance is estimated based on the information on the moving velocity of each mounting site estimated above." Also, in this publication, it is described that "for the nodes of the reference part, for the nodes whose position and posture can be directly acquired from an image or the like obtained by the camera C imaging the sensor device 20, or the nodes that can be estimated from the directly acquired information, their position, posture information, and moving acceleration are set based on the acquired or estimated information."

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In configurations that accumulate the detection results of acceleration and angular velocity, there is a problem that the error increases over time. According to the conventional technology described above, if a sensor attached to a subject can be imaged, the position, posture information, and motion acceleration of the sensor can be reflected. However, the sensor is not always suitable for being captured by a camera. From the viewpoint of posture detection accuracy and the ease of movement of the subject, it is preferable to attach many small, inconspicuous sensors, but such sensors are difficult to identify in the image. Therefore, for high-precision pose estimation, it is necessary to correct for sensor errors even when it is not possible to directly photograph the sensor.

[0005] Therefore, the present invention aims to correct sensor errors and achieve highly accurate posture estimation even when the sensor cannot be directly photographed. [Means for solving the problem]

[0006] To achieve the above objective, one representative posture recognition system of the present invention is a posture recognition system for recognizing the posture of a subject, comprising: a first posture estimation unit that estimates the posture of the subject based on sensing results of sensors attached to the subject; a second posture estimation unit that estimates the posture of the subject based on images taken of the subject; a second reliability estimation unit that determines the reliability of the second estimation result, which is the estimation result by the second posture estimation unit, as a second reliability; and a correction unit that corrects the first estimation result, which is the estimation result by the first posture estimation unit, based on the second estimation result and the second reliability. Furthermore, one representative posture recognition method of the present invention is a posture recognition method for recognizing the posture of a subject, characterized in that it includes: a first posture estimation step in which a computer estimates the posture of the subject based on sensing results of sensors attached to the subject; a second posture estimation step in which a computer estimates the posture of the subject based on an image taken of the subject; a second confidence estimation step in which the confidence level of the second estimation result, which is the estimation result of the second posture estimation step, is determined as the second confidence level; and a correction step in which the first estimation result, which is the estimation result of the first posture estimation step, is corrected based on the second estimation result and the second confidence level. [Effects of the Invention]

[0007] According to the present invention, highly accurate posture estimation can be achieved. Other problems, configurations, and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0008] [Figure 1] Configuration diagram of the posture recognition system in Example 1 [Figure 2] Diagram illustrating the attachment of the posture sensor 111. [Figure 3] Diagram illustrating attitude estimation based on sensor output [Figure 4] Diagram illustrating the first estimation result based on sensor output. [Figure 5] Diagram illustrating the second estimation result based on camera output. [Figure 6] Flowchart for second estimation results and second confidence level estimation [Figure 7] Example of the calculation result for the second confidence level [Figure 8] Specific examples of correction factors according to the second confidence level [Figure 9] Diagram illustrating the structural tree of feature points. [Figure 10] Diagram explaining posture correction [Figure 11] Posture correction flowchart [Figure 12] Example of posture recognition result output [Figure 13] Modification Example of the Second Estimation Result [Figure 14] Explanatory Diagram of Calculation of Coordinates of Virtual Feature Points [Figure 15] Explanatory Diagram of Decrease in Second Reliability Due to Obstacle [Figure 16] Explanatory Diagram of Handling of Undetected Feature Points [Figure 17] Configuration Diagram of the Posture Recognition System of Example 2 [Figure 18] Explanatory Diagram of Sensor Posture Reliability Estimation Unit [Figure 19] Specific Example of Correction Coefficient in Example 2 [Figure 20] Explanatory Diagram of Use of Posture Recognition Result [Figure 21] Explanatory Diagram of Posture Recognition System for Obtaining a Posture for Correction [Figure 22] Explanatory Diagram of Posture Estimation Based on Detection Result of Object

Mode for Carrying Out the Invention

[0009] Hereinafter, examples will be described with reference to the drawings.

Example

[0010] FIG. 1 is a configuration diagram of the posture recognition system of Example 1. The person to be recognized by the posture recognition system 150 of Example 1 is the worker 110 who performs work in a factory or the like. The worker 110 wears posture sensors 111 at a plurality of locations on the body. The plurality of posture sensors 111 measure acceleration, angular velocity, geomagnetism, etc. The measured values of each posture sensor 111 are transmitted as motion data 120 to the posture recognition system 150 via the communication unit 112.

[0011] Also, a camera 130 is installed in the area where the worker 110 performs work. The camera 130 includes an imaging unit 131 and a communication unit 132. The imaging unit 131 captures an image of the worker 110. The communication unit 132 transmits the image data 140 obtained as a result of the imaging by the imaging unit 131 to the posture recognition system 150.

[0012] The posture recognition system 150 includes a communication unit 151, a sensor posture estimation unit 152, a camera posture estimation unit 153, a posture reliability estimation unit 154, a sensor posture correction unit 155, and a control unit 156.

[0013] The communication unit 151 receives the operation data 120 and the image data 140. The sensor posture estimation unit 152 is a first posture estimation unit that estimates the posture of the worker 110 based on the motion data 120. The sensor posture estimation unit 152 manages the relative positional relationship of the posture sensors 111 in their initial state. Furthermore, the sensor posture estimation unit 152 receives the motion data 120 and manages the accumulated measured values ​​for each posture sensor 111. The posture sensors 111 estimate the changes in the posture and position of the worker 110 from the relative positional changes in their initial state and the accumulation of the measured values ​​of each posture sensor 111. The posture estimation result by the sensor posture estimation unit 152 is the first estimation result.

[0014] The camera posture estimation unit 153 is a second posture estimation unit that estimates the posture of the worker 110 based on the image data 140. Any existing technology can be used for estimating posture from the image. The posture estimation result by the camera posture estimation unit 153 is the second estimation result.

[0015] The posture confidence estimation unit 154 is a second confidence estimation unit that determines the confidence level of the second estimation result, which is the estimation result from the camera posture estimation unit 153, as the second confidence level. The sensor attitude correction unit 155 is a correction unit that corrects the first estimation result based on the second estimation result and the second confidence level. In other words, the sensor attitude correction unit 135 corrects the attitude estimated based on the attitude sensor 111 based on the attitude estimated from the camera image and its confidence level.

[0016] The control unit 156 controls the operation of the attitude recognition system 150. When the computer operates as the attitude recognition system 150 by executing the attitude recognition program, the control unit 156 is the CPU (Central Processing Unit), and by loading the program into memory and executing it, it realizes the functions of the communication unit 151, sensor attitude estimation unit 152, camera attitude estimation unit 153, attitude reliability estimation unit 154, and sensor attitude correction unit 155.

[0017] Figure 2 is an explanatory diagram of the attachment of the posture sensor 111. In Figure 2, the worker 110 is wearing a garment-type sensor with posture sensors 111 attached to the hat, left and right shoulders, left and right upper arms, left and right forearms, left and right waist, left and right thighs, and left and right shins. These posture sensors 111 do not need to be visible from the outside. In Figure 2, the garment-type sensor has a communication unit 112 on the belt portion. The communication unit 112 may be placed in any position as long as it can acquire measured values ​​from each posture sensor 111 and transmit them to the posture recognition system 150.

[0018] Figure 3 is an explanatory diagram of posture estimation based on sensor output. The sensor posture estimation unit 152 has estimation units for each body part, such as the upper arm state estimation unit 301 and the waist state estimation unit 302. Each estimation unit for a body part has a corresponding posture sensor 111, and estimates the state of the body part from the measured values ​​of the corresponding posture sensor 111. For example, the upper arm state estimation unit 301 estimates the time change in the rotation angle of the upper arm using the measured values ​​from the posture sensors 111 attached to the shoulder, upper arm, and forearm. Furthermore, the waist state estimation unit 302 estimates the time change in the waist bending angle using the measured values ​​from the posture sensors 111 attached to the waist and thighs.

[0019] Figure 4 is an explanatory diagram of the first estimation result based on sensor output. As shown in Figure 4, the first estimation result is data that shows the subject's skeleton by connecting multiple feature points. The feature points include the positions of the subject's joints. For example, in Figure 4, the head, lumbar vertebrae, spine, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles are used as feature points. Here, the shoulder feature points can be made to correspond one-to-one with the shoulder posture sensor 111. That is, the position of the shoulder feature point is the position of the corresponding single posture sensor 111. There is no posture sensor 111 that corresponds one-to-one to the elbow feature point. However, the position of the elbow feature point can be determined from the positions of the shoulder, upper arm, and forearm posture sensors 111. Thus, it is not necessary for the posture sensors 111 and feature points to be one-to-one; the skeleton of the worker 110 can be estimated from the relative positions of multiple posture sensors 111, and its joints, etc., can be used as feature points. The location of feature points is indicated by three-dimensional spatial coordinates. The connections between feature points are set according to the body part. For example, the connection between the feature point of the right hip and the feature point of the right knee corresponds to the right thigh.

[0020] Figure 5 is an explanatory diagram of the second estimation result based on camera output. The camera posture estimation unit 153 processes the image data 140 captured by the camera 130 to obtain the second estimation result. The second estimation result is data that shows the subject's skeleton by connecting multiple feature points. The feature points include the positions of the subject's joints.

[0021] The position of feature points is indicated by three-dimensional spatial coordinates. For this reason, the camera 130 may be replaced with a depth camera or a stereo camera to measure the distance from the camera 130. Alternatively, distance indicators may be placed within the shooting range of the camera 130, and the distance from the camera 130 may be determined by image processing. Furthermore, Lidar (Light Detection and Ranging) may be used in combination, or Lidar may be used instead of the camera 130.

[0022] The second estimation result based on camera output does not involve cumulative errors. In particular, the position of feature points can be determined with high confidence for parts that are directly visible in the image. On the other hand, for parts that are not directly visible in the image, such as those that are obscured by other parts or objects, the position of feature points is estimated based on their relative relationship to other parts, resulting in relatively low confidence.

[0023] Therefore, the posture confidence estimation unit 154 determines that if a predetermined pair of feature points included in the second estimation result is estimated to be close within a predetermined range in the image and separated by a predetermined distance or more in spatial coordinates, it sets the second confidence of the feature point that is farther from the camera that took the image to a lower value compared to the other feature points. If two feature points are close within a predetermined range in the image and separated by a predetermined distance or more in spatial coordinates, it is highly likely that the two feature points are aligned in the depth direction with respect to the camera 130. And if the two feature points are aligned in the depth direction with respect to the camera 130, it is highly likely that the body part corresponding to the feature point further away is hidden by the body part corresponding to the feature point closer to the camera.

[0024] Figure 6 is a flowchart of the second estimation result and the estimation of the second confidence level. This flowchart starts from step S601. Step S601: The camera pose estimation unit 153 calculates the coordinates of the feature points of each body part. Initially, the second confidence level of each feature point is set to "high". Then, the process proceeds to step S602. Step S602: The posture reliability estimation unit 154 selects feature points for any two body parts. Then, the process proceeds to step S603. Step S603: The posture confidence estimation unit 154 determines whether the distance between the x and y coordinates of two feature points in the image is less than or equal to a certain value. If yes, proceed to step S604. If no, terminate the process. Step S604: The posture confidence estimation unit 154 determines whether the difference in the distance (z coordinate) of two feature points from the camera is greater than or equal to a certain value. If yes, proceed to step S6045. If no, terminate the process. Step S605: The posture confidence estimation unit 154 sets the second confidence level of the feature point that is farther from the camera to "low". Then, the process ends.

[0025] The comparison of two feature points in Figure 6 can be performed by sequentially selecting all possible combinations. Alternatively, feature points that are nearby other feature points in the image can be extracted, and then combinations can be selected.

[0026] Figure 7 shows an example of the calculation results for the second confidence level. In Figure 7, the worker's left arm is further away than their right arm. Therefore, the second confidence level for the feature points of the left shoulder, left elbow, and left wrist is "low". Also, the left hip is further away than the lumbar spine. Therefore, the second confidence level for the feature point of the left hip is "low".

[0027] Figure 8 shows specific examples of correction coefficients according to the second confidence level. In Figure 8, the correction coefficient for a "high" second confidence level is set to "0.9", and the correction coefficient for a "low" second confidence level is set to "0.4".

[0028] Figure 9 is an explanatory diagram of the feature point structure tree. The first estimation result based on the posture sensor 111 and the second estimation result based on the camera 130 show the skeleton of the worker 110 through the feature point structure tree.

[0029] In Figure 9, the lumbar vertebrae (for convenience, the lowest part of the lumbar vertebrae is considered the lumbar vertebral feature point) is used as the reference point. Specifically, the feature point "lumbar vertebrae" is connected to the feature point "spine (for convenience, the lowest part of the thoracic vertebrae and the spine feature point)", the feature point "right buttock", and the feature point "left buttock". Furthermore, the feature point "spine" is connected to the feature point "left shoulder," the feature point "head," and the feature point "right shoulder." The feature point "left elbow" is further connected to the feature point "left shoulder," and the feature point "left wrist" is connected to the feature point "left elbow." Similarly, the feature point "right shoulder" is further connected to the feature point "right elbow," and the feature point "right wrist" is connected to the feature point "right elbow." Furthermore, the feature point "right hip" is further connected to the feature point "right knee," and the feature point "right ankle" is connected to the feature point "right knee." Similarly, the feature point "left hip" is further connected to the feature point "left knee," and the feature point "left ankle" is connected to the feature point "left knee."

[0030] Here, for convenience, the vector from the position coordinates of one of the two connected feature points to the position coordinates of the other feature point is called the frame vector. In Figure 9, the lumbar spine is the reference point, so the vector from the lumbar spine to the spine is the frame vector. Feature points whose vectors are defined from a reference position can be used as the next reference point. Therefore, for the spine and the left shoulder, the spine is the reference point, and the vector from the spine to the left shoulder is the frame vector. Similarly, the frame vector from the left shoulder to the left elbow and the frame vector from the left elbow to the left wrist can be identified.

[0031] Figure 10 is an explanatory diagram of posture correction. The sensor posture correction unit 155 corrects the first estimation result by correlating the feature points included in the first estimation result with the feature points included in the second estimation result. In Figure 10, the feature points of the lumbar spine and vertebral spine are compared between the first estimation result (sensor-based) and the second estimation result (camera-based).

[0032] First, the position coordinates of the reference lumbar vertebral feature points are aligned between the first and second estimation results. As a result, a discrepancy occurs between the position of the vertebra in the first estimation result and the position of the vertebra in the second estimation result. The sensor posture correction unit 155 uses the vector from the position coordinates of the vertebra in the first estimation result to the position coordinates of the vertebra in the second estimation result as the correction vector for the feature point "vertebra". By performing a similar process, the sensor attitude correction unit 155 obtains a correction vector for each feature point included in the first estimation result.

[0033] For example, using the characteristic points of the spine as a reference, the position coordinates of the characteristic points of the spine are aligned in the first estimation result and the second estimation result. As a result, a discrepancy occurs between the position coordinates of the first estimation result and the position coordinates of the second estimation result for the left shoulder, head, and right shoulder. The sensor posture correction unit 155 calculates the vector from the position coordinates of the first estimation result to the position coordinates of the second estimation result for the left shoulder, head, and right shoulder, respectively, and uses these as correction vectors for the left shoulder, head, and right shoulder.

[0034] Figure 11 is a flowchart of the posture correction process. This flowchart begins with steps S1101, S1102, and S1103. Step S1101: The sensor posture correction unit 155 obtains the coordinates of the feature points of each body part output by the sensor posture estimation unit 152 as the first estimation result. The sensor posture estimation unit 152 outputs the coordinates of the feature points in a three-dimensional spatial coordinate system, for example, with the floor plane as the XY plane and the height direction as the Z direction. Then, the process proceeds to step S1104. Step S1102: The sensor pose correction unit 155 obtains the coordinates of the feature points of each body part output by the camera pose estimation unit 153 as the second estimation result. The camera pose estimation unit 153 calculates the coordinates of the feature points in three-dimensional spatial coordinates where the camera image is in the XY plane and the distance from the camera is the Z axis, but these spatial coordinates are converted to the spatial coordinates of the sensor pose estimation unit 152 and output. Then, the process proceeds to step S1105. Step S1103: The sensor posture correction unit 155 obtains correction coefficients for the feature points of each body part output by the posture reliability estimation unit 154. Then, the process proceeds to step S1107.

[0035] Step S1104: The sensor attitude correction unit 155 calculates frame vectors between adjacent feature points along the structure tree of the feature points of the first estimation result. Then, the process proceeds to step S1106. Step S1105: The sensor attitude correction unit 155 calculates frame vectors between adjacent feature points along the structure tree of the feature points of the second estimation result. Then, the process proceeds to step S1106.

[0036] Step S1106: The sensor attitude correction unit 155 calculates the difference between the corresponding frame vectors in the first estimation result and the second estimation result, and calculates a correction vector for each feature point. Then, the process proceeds to step S1107. Step S1107: The sensor attitude correction unit 155 multiplies the correction vector by the correction coefficient of the corresponding trust part. Then proceed to step S1108. Step S1108: Add the correction vector, multiplied by the correction coefficient, to the coordinates of the feature points of the first estimation result to obtain the coordinates of the corrected feature points. Then, terminate the process.

[0037] The coordinate correction of each feature point can be performed independently, or it can be performed sequentially starting from a reference feature point. The reference feature point can be a specific feature point (e.g., the lumbar vertebrae), or it can be the feature point in the image from which the coordinates were estimated with the highest confidence level. For example, if the second confidence level of the feature point of the right elbow is the highest, the structure tree can be referenced using the right elbow as the reference point, and the frame vectors for the right wrist and right shoulder can be obtained and corrected accordingly. Then, the next frame vectors can be obtained and corrected using the corrected coordinates of the right wrist and right shoulder as the reference point. By repeating this process, the position of the entire body can be corrected with high accuracy.

[0038] Figure 12 shows an example of the output of the posture recognition result. Figure 12 shows a graphic user interface that allows users to check the posture recognition result by superimposing the first estimation result (posture estimated based on the clothing-type sensor), the second estimation result (posture estimated based on the camera image), and the corrected posture onto the camera image. The first estimation result, the second estimation result, and the corrected posture may be displayed unconditionally or they may be selectable. Alternatively, the image may be rendered using computer graphics instead of the camera image.

[0039] Up to this point, the explanation has used the example where the feature points of the first estimation result and the feature points of the second estimation result correspond one-to-one. However, the feature points of the first estimation result and the second estimation result do not necessarily have to correspond.

[0040] Figure 13 shows a modified version of the second estimation result. In Figure 13, the second estimation result by the camera posture estimation unit 153 does not include feature points corresponding to the spine, but it does include feature points corresponding to the cervical vertebrae (for convenience, the lowest part of the cervical vertebrae is considered the cervical vertebra feature point).

[0041] Thus, if the feature points constituting the first estimation result include feature points that do not correspond to the feature points constituting the second estimation result, the camera pose estimation unit 153 determines virtual feature points corresponding to the feature points constituting the first estimation result from the positional relationship of the multiple feature points constituting the second estimation result. It is preferable to pre-determine the correspondence between the feature points of the first estimation result and the feature points of the second estimation result, the necessary virtual feature points, and the method for calculating the coordinates of the virtual feature points.

[0042] In Figure 14, the coordinates of the spinal feature points are defined as the points that divide the cervical and lumbar vertebrae in an N:M ratio. In this case, the second confidence level can also be calculated by, for example, weighting the second confidence levels of the cervical and lumbar vertebrae according to the ratio of internal division and adding them together.

[0043] Next, we will further explain the second confidence level of the second estimation result. Figure 15 is an explanatory diagram illustrating the decrease in second confidence due to obstacles. If an obstacle is present in the shooting range of camera 130 and part of the worker's body 110 is hidden, the camera posture estimation unit 153 can only estimate the posture for the part that was captured in the image. In other words, feature points whose coordinates cannot be determined will be included in the second estimation result. Feature points whose coordinates cannot be determined are treated as undetected feature points and are not subject to correction. As a result, the feature points of the first estimation result are used without correction.

[0044] Figure 16 is an explanatory diagram for handling undetected feature points. In Figure 16, the second confidence level is divided into three levels: "high," "low," and "undetected." The correction coefficient for feature points with a second confidence level of "undetected" is set to "0." If the correction coefficient is "0," the first estimation result is not corrected for that feature point. Note that although the correction coefficient for undetected feature points is set to "0" here, it is also possible to configure the system to exclude feature points that were undetected in the second estimation result from the first estimation result before correction.

[0045] As described above, the posture recognition system 150 of Embodiment 1 performs a first posture estimation step, which estimates the posture of the worker 110 based on the sensing results of the posture sensor 111 attached to the worker 110, and a second posture estimation step, which estimates the posture of the worker 110 based on an image taken of the worker 110. Then, it performs a second confidence estimation step, which calculates the confidence level of the second estimation result, which is the estimation result from the second posture estimation step, as the second confidence level, and a correction step, which corrects the first estimation result, which is the estimation result from the first posture estimation step, based on the second estimation result and the second confidence level. Therefore, even if the posture sensor 111 cannot be directly photographed, the error of the posture sensor 111 can be corrected, and highly accurate posture estimation can be achieved. [Examples]

[0046] This second embodiment describes a posture recognition system that further corrects the first estimation result using the confidence level of the feature points of the first estimation result. Figure 17 is a diagram showing the configuration of the posture recognition system of Embodiment 2. The posture recognition system 150 shown in Figure 17 further comprises a sensor posture reliability estimation unit 1507.

[0047] The sensor attitude confidence estimation unit 1507 is a first confidence estimation unit that determines the confidence level of each feature point included in the first estimation result, which is the estimation result by the sensor attitude estimation unit 152, as the first confidence level. The sensor attitude correction unit 155 is a correction unit that corrects the first estimation result by further using the first confidence level in addition to the second estimation result and the second confidence level. Other configurations and operations are the same as in Example 1, so their explanation will be omitted.

[0048] Figure 18 is an explanatory diagram of the sensor posture reliability estimation unit 1507. The sensor posture estimation unit 152 estimates the position of the posture sensor 111 by accumulating the measured values ​​(such as acceleration) of the posture sensor 111. At this time, if the measured values ​​are small, the signal-to-noise ratio worsens and errors tend to accumulate. Therefore, the sensor posture reliability estimation unit 1507 gradually reduces the first reliability of the corresponding feature point when the body is not moving much, i.e., when the measured values ​​remain small for a certain period of time.

[0049] Figure 19 shows a specific example of the correction coefficient in Example 2. In Figure 19, the first confidence level α is used, and the correction coefficient for a "high" second confidence level is set to "0.9*(1-α)", while the correction coefficient for a "low" second confidence level is set to "0.4*(1-α)". Note that the correction coefficients in Figure 19 are merely examples. Any method can be used as long as the correction coefficients can be manipulated to be inversely proportional to the first confidence level.

[0050] When correcting the first estimation result, the first confidence level may be reset or increased. For example, for feature points with a "high" second confidence level, the first confidence level may be reset to its initial value (a value with no accumulated error), and for feature points with a "low" second confidence level, the first confidence level may be increased by a predetermined amount.

[0051] (modified version) Next, we will explain how to use the posture recognition results. Figure 20 is an explanatory diagram of how the posture recognition results are used. In Figure 20, the posture recognition system 150 corrects the first estimation result and then outputs it to the work recognition system 160 as posture data 1800.

[0052] The work recognition system 160 includes a communication unit 161 and an action recognition unit 162. The communication unit 161 receives posture data 1800 from the posture recognition system 150. The action recognition unit 162 recognizes the actions of the worker 110 based on the posture data 1800. This action recognition allows for real-time recognition of the worker's work status and visualization of the workload.

[0053] Furthermore, the posture recognition system 150 can identify the timing for correction based on the actions that the worker 110 is expected to perform. The work performed by the worker 110 may have a defined sequence of actions. If, for example, the work recognition system 160 manages the actions that the worker 110 should perform, the sensor posture correction unit 155 can obtain information about the actions that the worker 110 should perform from the work recognition system 160 in advance, identify the timing for a posture suitable for correcting the first estimated result, and perform the correction.

[0054] In addition to utilizing the actions performed by worker 110, instructions may also be issued to worker 110 at the beginning or in the middle of the work to assume a corrective posture. Figure 21 is an explanatory diagram of a posture recognition system that prompts the user to assume a corrective posture. In Figure 21, the posture recognition system 150 is connected to the speaker system 1910. The speaker system 1910 outputs the audio signal received from the posture recognition system 150 by the communication unit 1902 through the speaker 1901.

[0055] In the configuration shown in Figure 21, the control unit 156 of the posture recognition system 150 functions as an instruction output unit that requests the worker 110 to assume a predetermined posture by transmitting and outputting an audio signal to the speaker system 1910. The sensor posture correction unit 155 performs correction after requesting the worker to assume a predetermined posture by outputting audio from the speaker 1901. As a result, motion data 120 and image data 140 are acquired in a posture suitable for correction, enabling highly accurate correction. Furthermore, other systems, such as the work recognition system 160, may also be equipped with a function as an instruction output unit that requests the worker 110 to assume a predetermined posture. Furthermore, the posture recognition system 150 may determine whether the worker 110 has assumed a predetermined posture, and may report an error if the worker has not assumed the predetermined posture.

[0056] Figure 22 is an explanatory diagram of posture estimation based on object detection results. Even if an object is present around worker 110 and part of worker 110's body is obscured and cannot be photographed, the object itself may contribute to posture estimation. For example, if a manual is provided on how to lift a specific object while working with it, the position coordinates of parts not visible in the image can be estimated.

[0057] In Figure 22, the posture recognition system 150 includes an object detection unit 157 in addition to a camera posture estimation unit 153. The object detection unit 157 performs image detection processing to detect a specific object from the image data captured by the camera 130. If a specific object is detected, the camera posture estimation unit 153 further uses the object's position and type to estimate the posture of the worker 110.

[0058] As described above, the disclosed system is a posture recognition system 150 that recognizes the posture of a subject, and comprises: a sensor posture estimation unit 152 as a first posture estimation unit that estimates the posture of the subject based on the sensing results of a posture sensor 111 which is a sensor attached to the subject; a camera posture estimation unit 153 as a second posture estimation unit that estimates the posture of the subject based on an image taken of the subject; a posture reliability estimation unit 154 as a second reliability estimation unit that determines the reliability of the second estimation result, which is the estimation result by the second posture estimation unit, as a second reliability; and a sensor posture correction unit 155 as a correction unit that corrects the first estimation result, which is the estimation result by the first posture estimation unit, based on the second estimation result and the second reliability. Therefore, even when it is not possible to directly photograph the sensor, the sensor's errors can be corrected, enabling highly accurate pose estimation.

[0059] Furthermore, the sensors are sensors that measure acceleration and / or angular velocity, and the sensors are attached to multiple locations on the subject's body. The first posture estimation unit accumulates the measured values ​​from the multiple sensors attached to multiple locations on the subject's body, determines the relative positional relationship of the multiple sensors, and estimates the subject's skeleton from the relative positional relationship of the multiple sensors. Therefore, by correcting the posture estimation results from wearable sensors with camera images, highly accurate posture estimation can be achieved.

[0060] Furthermore, the first posture estimation unit and the second posture estimation unit estimate a plurality of feature points including the positions of the subject's joints, connect the plurality of feature points to estimate the subject's skeleton, and the correction unit corrects the first estimation result by matching the feature points included in the first estimation result with the feature points included in the second estimation result. In this way, instead of directly correcting the sensor position from the camera image, the posture estimation result can be corrected regardless of whether the sensor was able to capture an image or not by comparing the skeletal structure estimated from both the sensor output and the camera output.

[0061] Furthermore, if the second attitude estimation unit finds a feature point among the feature points constituting the first estimation result that does not correspond to the feature points constituting the second estimation result, it will determine a virtual feature point corresponding to the feature point constituting the first estimation result from the positional relationship of the multiple feature points constituting the second estimation result. Therefore, any method for estimating the skeleton from camera images can be used without being limited by the model used to represent the posture. Furthermore, if the second confidence estimation unit estimates that a predetermined set of feature points included in the second estimation result are close together within a predetermined range in the image and separated by a predetermined distance or more in spatial coordinates, it sets the second confidence of the feature point that is farther from the camera that captured the image among the feature points forming the predetermined set to a lower value compared to the other feature points. Therefore, by identifying feature points not visible in the image, the second confidence level can be reduced, and the first estimation result can be prioritized for feature points with a low second confidence level.

[0062] Furthermore, the correction unit obtains a frame vector from the spatial coordinates of one of the connected feature points to the spatial coordinates of the other feature point, obtains the difference between the corresponding frame vectors of the first estimation result and the second estimation result as a correction vector, multiplies the correction vector by a correction coefficient corresponding to the second confidence level of the other feature point in the second estimation result, and adds the correction vector multiplied by the correction coefficient to the spatial coordinates of the other feature point in the first estimation result to correct the position of the other feature point in the first estimation result. Therefore, the position of feature points can be accurately corrected along the frame structure of the skeleton.

[0063] Furthermore, the correction unit calculates and corrects the frame vector based on the second high-confidence feature point, and repeats the process of calculating and correcting the next frame vector based on the corrected feature point. Therefore, by utilizing the positional coordinates of feature points identified with high precision from the image, the position and orientation of the object can be estimated with high accuracy.

[0064] Furthermore, the disclosed system further comprises a first confidence estimation unit that determines the confidence level for each of the feature points included in the first estimation result as a first confidence level, and the correction unit further uses the first confidence level to correct the first estimation result, which is the estimation result by the first posture estimation unit. Specifically, the sensor is a sensor that measures acceleration and / or angular velocity, and the first reliability estimation unit lowers the first reliability of a feature point identified based on the sensor's measurement value compared to other feature points if the measurement value of the sensor remains below a predetermined value for a predetermined period of time. Therefore, by taking into account the accumulation of errors in the first estimation result, highly accurate pose estimation can be achieved.

[0065] Furthermore, the correction unit is characterized in that, if a part of the subject corresponding to a characteristic point is not visible in the image, the characteristic point is excluded from the correction. Alternatively, if the part of the subject corresponding to the feature point is not visible in the image, the second confidence estimation unit lowers the second confidence level of that feature point compared to other feature points. In this way, for feature points that are not visible in the image, the first estimation result can be given priority.

[0066] Furthermore, the correction unit identifies the timing for performing the correction based on the actions that the subject is expected to perform. Alternatively, the system further includes an instruction output unit that requests the subject to assume a predetermined posture, and the correction unit performs corrections after the instruction output unit has requested the subject to assume a predetermined posture. In this way, by utilizing conditions suitable for correction, the effectiveness of the correction can be enhanced, thereby achieving highly accurate pose estimation.

[0067] Furthermore, the disclosed system further comprises an object detection unit for detecting objects in the image, and the second confidence estimation unit further uses the object detection results from the object detection unit to determine a second confidence level for each feature point. Therefore, the presence of an object can be used to estimate the position of the part that is obscured by the object, and this can be used to correct the posture estimation result.

[0068] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace or add configurations, not just delete them. For example, the above embodiment illustrates a configuration in which the second confidence level is assigned as "high" or "low" depending on the positional relationship of feature points identified from the image. As a variation, the second confidence level may be a numerical value. Alternatively, the second confidence level may be determined based on the distance from the camera 130 to the worker 110 or the image shooting conditions (such as ambient brightness). [Explanation of Symbols]

[0069] 20: Sensor device, 110: Worker, 111: Posture sensor, 112: Communication unit, 120: Motion data, 130: Camera, 131: Imaging unit, 132: Communication unit, 135: Sensor posture correction unit, 140: Image data, 150: Posture recognition system, 151: Communication unit, 152: Sensor posture estimation unit, 153: Camera posture estimation unit, 154: Posture reliability estimation unit, 155: Sensor posture correction unit, 156: Control unit, 157: Object detection unit, 160: Work recognition system, 161: Communication unit, 162: Action recognition unit, 301: Upper arm state estimation unit, 302: Waist state estimation unit, 1507: Sensor posture reliability estimation unit, 1800: Posture data, 1901: Speaker, 1902: Communication unit, 1910: Speaker system

Claims

1. A posture recognition system that recognizes the posture of a subject, A first posture estimation unit estimates the posture of the subject based on the sensing results of sensors attached to the subject, A second posture estimation unit estimates the posture of the subject based on an image of the subject, A second confidence estimation unit determines the confidence level of the second estimation result, which is the estimation result of the second posture estimation unit, as the second confidence level, A correction unit corrects the first estimation result, which is the estimation result by the first attitude estimation unit, based on the second estimation result and the second confidence level. Equipped with, The first posture estimation unit and the second posture estimation unit estimate a plurality of feature points including the joint positions of the subject, and estimate the subject's skeleton by connecting the plurality of feature points. The correction unit corrects the first estimation result by matching the feature points included in the first estimation result with the feature points included in the second estimation result. The second confidence estimation unit, when it estimates that a predetermined set of feature points included in the second estimation result are close together within a predetermined range in the image and separated by a predetermined distance or more in spatial coordinates, sets the second confidence of the feature point that is farther from the camera that captured the image among the feature points in the predetermined set to a lower value compared to the other feature points. A posture recognition system characterized by the following features.

2. A posture recognition system for recognizing the posture of a subject, A first posture estimation unit estimates the posture of the subject based on the sensing results of sensors attached to the subject, A second posture estimation unit estimates the posture of the subject based on an image of the subject, A second confidence estimation unit determines the confidence level of the second estimation result, which is the estimation result of the second posture estimation unit, as the second confidence level, A correction unit corrects the first estimation result, which is the estimation result by the first attitude estimation unit, based on the second estimation result and the second confidence level. Equipped with, The first posture estimation unit and the second posture estimation unit estimate a plurality of feature points including the joint positions of the subject, and estimate the subject's skeleton by connecting the plurality of feature points. The correction unit corrects the first estimation result by matching the feature points included in the first estimation result with the feature points included in the second estimation result. The correction unit obtains a frame vector from the spatial coordinates of one of the connected feature points to the spatial coordinates of the other feature point, obtains a correction vector from the difference between the corresponding frame vectors of the first estimation result and the second estimation result, multiplies the correction vector by a correction coefficient corresponding to the second confidence level of the other feature point in the second estimation result, and adds the correction vector multiplied by the correction coefficient to the spatial coordinates of the other feature point in the first estimation result to correct the position of the other feature point in the first estimation result. A posture recognition system characterized by the following features.

3. A posture recognition system according to Claim 2, The posture recognition system is characterized in that the correction unit calculates and corrects the frame vector based on the second highly reliable feature point, and repeats the process of calculating and correcting the next frame vector based on the corrected feature point.

4. A posture recognition system according to claim 1 or 2, The aforementioned sensor is a sensor that measures acceleration and / or angular velocity, The sensors are attached to multiple locations on the subject's body, The posture recognition system is characterized in that the first posture estimation unit accumulates measurement values ​​from a plurality of sensors attached to multiple locations on the subject's body, determines the relative positional relationship of the plurality of sensors, and estimates the subject's skeleton from the relative positional relationship of the plurality of sensors.

5. A posture recognition system according to claim 1 or 2, The posture recognition system is characterized in that, if the second posture estimation unit includes feature points that do not correspond to the feature points that constitute the second estimation result, it determines a virtual feature point corresponding to the feature point that constitutes the first estimation result from the positional relationship of the plurality of feature points that constitute the second estimation result.

6. A posture recognition system according to claim 1 or 2, The system further comprises a first confidence estimation unit that determines the confidence level for each of the feature points included in the first estimation result as the first confidence level, The posture recognition system is characterized in that the correction unit further uses the first confidence level to correct the first estimation result, which is the estimation result by the first posture estimation unit.

7. A posture recognition system according to claim 6, The aforementioned sensor is a sensor that measures acceleration and / or angular velocity, The posture recognition system is characterized in that, when the measurement value of the sensor remains below a predetermined value for a predetermined period of time, the first confidence level of a feature point identified based on the measurement value of the sensor is lowered compared to other feature points.

8. A posture recognition system according to claim 1 or 2, The posture recognition system is characterized in that the correction unit excludes a feature point from correction if the part corresponding to the feature point of the subject is not visible in the image.

9. A posture recognition system according to claim 1 or 2, The posture recognition system is characterized in that, if the part of the subject corresponding to the feature point is not visible in the image, the second confidence level of the feature point is lowered compared to other feature points.

10. A posture recognition system according to claim 1 or 2, The posture recognition system is characterized in that the correction unit identifies the timing for performing corrections based on actions that are expected to be performed by the subject.

11. A posture recognition system according to claim 1 or 2, The system further includes an instruction output unit that requests the subject to assume a predetermined posture. The posture recognition system is characterized in that the correction unit performs correction after being requested by the instruction output unit to assume a predetermined posture.

12. A posture recognition system according to claim 1 or 2, The system further includes an object detection unit that detects objects in the aforementioned image, The posture recognition system is characterized in that the second confidence estimation unit further uses the object detection results from the object detection unit to determine the second confidence level of each feature point.

13. A posture estimation method for recognizing the posture of a subject, Computers A first posture estimation step in which the posture of the subject is estimated based on the sensing results of sensors attached to the subject, A second posture estimation step involves estimating the posture of the subject based on an image taken of the subject, A second confidence estimation step is performed to determine the confidence level of the second estimation result, which is the estimation result obtained by the second posture estimation step, as the second confidence level. A correction step is performed to correct the first estimation result, which is the estimation result obtained by the first posture estimation step, based on the second estimation result and the second confidence level. Includes, The first posture estimation step and the second posture estimation step estimate a plurality of feature points including the positions of the subject's joints, and connect the plurality of feature points to estimate the subject's skeleton. The correction step involves correcting the first estimation result by matching the feature points included in the first estimation result with the feature points included in the second estimation result. The second confidence estimation step, when it is estimated that a predetermined set of feature points included in the second estimation result are close together within a predetermined range in the image and separated by a predetermined distance or more in spatial coordinates, sets the second confidence of the feature point that is farther from the camera that captured the image among the feature points in the predetermined set to a lower value compared to the other feature points. A method for recognizing posture characterized by the following features.

14. A posture estimation method for recognizing the posture of a subject, Computers A first posture estimation step in which the posture of the subject is estimated based on the sensing results of sensors attached to the subject, A second posture estimation step involves estimating the posture of the subject based on an image taken of the subject, A second confidence estimation step is performed to determine the confidence level of the second estimation result, which is the estimation result obtained by the second posture estimation step, as the second confidence level. A correction step is performed to correct the first estimation result, which is the estimation result obtained by the first posture estimation step, based on the second estimation result and the second confidence level. Includes, The first posture estimation step and the second posture estimation step estimate a plurality of feature points including the positions of the subject's joints, and connect the plurality of feature points to estimate the subject's skeleton. The correction step involves correcting the first estimation result by matching the feature points included in the first estimation result with the feature points included in the second estimation result. The correction step involves determining a frame vector from the spatial coordinates of one of the connected feature points to the spatial coordinates of the other feature point, determining the difference between the corresponding frame vectors of the first estimation result and the second estimation result as a correction vector, multiplying the correction vector by a correction coefficient corresponding to the second confidence level of the other feature point in the second estimation result, and adding the correction vector multiplied by the correction coefficient to the spatial coordinates of the other feature point in the first estimation result to correct the position of the other feature point in the first estimation result. A method for recognizing posture characterized by the following features.

Images