Posture recognition system and posture recognition method

Abstract

To achieve highly accurate posture estimation.SOLUTION: A posture recognition system 150 for recognizing the posture of a target person includes: a sensor posture estimation unit 152 that is a posture estimation unit that estimates three dimensional posture data indicating a posture of the target person on the basis of a sensing result of a posture sensor 111 that is a sensor attached to the target person; a camera feature extraction unit 153 that is a feature extraction unit that extracts two dimensional feature points of the target person on the basis of a captured image of the target person; a projection information estimation unit 154 that estimates projection information on the three dimensional posture data that reproduces arrangement of the two dimensional feature points extracted by the feature extraction unit; and a sensor posture correction unit 155 that is a posture correction unit that corrects the three dimensional posture data on the basis of the projection information estimated by the projection information estimation unit 154.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a posture recognition system and a posture recognition method. [Background technology]

[0002] Conventionally, there is a technology for recognizing the posture of a subject described in WO2021 / 075004 (Patent Document 1). This publication states, "At least information on the movement acceleration and attitude angular velocity of the attachment site where each sensor is attached is acquired from a plurality of sensors, and the movement speed of the attachment site where each sensor is attached within a predetermined coordinate system is estimated based on the acquired movement acceleration and attitude angular velocity information. The position of each predetermined part of the subject is then estimated based on the estimated movement speed information of each attachment site." This publication also states, "For nodes of the reference site whose position and orientation can be directly acquired from an image of the sensor device 20 captured by a camera C or estimated from the directly acquired information, their position, orientation information, and movement acceleration are set based on the acquired or estimated information." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2021 / 075004 Summary of the Invention [Problem to be solved by the invention]

[0004] A configuration that accumulates the detection results of acceleration and angular velocity has the problem that errors increase over time. According to the above-mentioned conventional technology, if an image of a sensor attached to a subject can be captured, the sensor's position, posture information, and movement acceleration can be reflected. However, sensors are not always suitable for camera capture. From the perspective of posture detection accuracy and the subject's ease of movement, it is preferable to attach multiple small, inconspicuous sensors, but such sensors are difficult to identify in the image. Therefore, for highly accurate pose estimation, it is necessary to correct sensor errors even when the sensor cannot be photographed directly.

[0005] Therefore, an object of the present invention is to correct sensor errors and achieve highly accurate attitude estimation even when the sensor cannot be photographed directly. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, one representative posture recognition system of the present invention is characterized by comprising a posture estimation unit that estimates three-dimensional posture data indicating the posture of the subject based on the sensing results of a sensor attached to the subject; a feature extraction unit that extracts two-dimensional feature points of the subject based on an image of the subject; a projection information estimation unit that estimates projection information of the three-dimensional posture data to reproduce the arrangement of the two-dimensional feature points extracted by the feature extraction unit; and a posture correction unit that corrects the three-dimensional posture data based on the projection information estimated by the projection information estimation unit. Furthermore, one representative posture recognition method of the present invention is a posture recognition method for recognizing the posture of a subject, characterized in that it includes a posture estimation step in which a computer estimates three-dimensional posture data indicating the posture of the subject based on the sensing results of a sensor attached to the subject, a feature extraction step in which two-dimensional feature points of the subject are extracted based on an image of the subject, a projection information estimation step in which projection information of the three-dimensional posture data is estimated to reproduce the arrangement of the two-dimensional feature points extracted by the feature extraction step, and a posture correction step in which the three-dimensional posture data is corrected based on the projection information estimated by the projection information estimation step. [Effects of the Invention]

[0007] According to the present invention, it is possible to realize highly accurate posture estimation. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0008] [Figure 1] Configuration diagram of a posture recognition system according to the first embodiment [Figure 2] An explanatory diagram of how to wear the posture sensor 111 [Figure 3] Illustration of posture estimation based on sensor output [Figure 4] Illustrative diagram of 3D posture data based on sensor output [Figure 5] Illustration of feature point extraction results based on camera output [Figure 6] Flowchart of generating posture correction parameters in the first embodiment [Figure 7] Illustrative diagram of the minutiae structure tree [Figure 8] Illustration of posture correction [Figure 9] Example of posture recognition result output [Figure 10] Modification of feature points extracted by the camera feature extraction unit [Figure 11] An explanatory diagram of calculation of coordinates of virtual feature points [Figure 12]Configuration diagram of a posture recognition system according to a second embodiment [Figure 13] Flowchart of generating posture correction parameters in the second embodiment [Figure 14] Configuration diagram of a posture recognition system according to a third embodiment [Figure 15] Flowchart of generating posture correction parameters in the third embodiment [Figure 16] Range of motion data illustration [Figure 17] Configuration diagram of a posture recognition system according to a fourth embodiment [Figure 18] Flowchart of generating posture correction parameters in the fourth embodiment DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment will be described with reference to the drawings. [Example]

[0010] FIG. 1 is a configuration diagram of a posture recognition system according to a first embodiment. A subject whose posture is recognized by a posture recognition system 150 according to the first embodiment is a worker 110 who works in a factory or the like. The worker 110 wears posture sensors 111 at multiple locations on his or her body. The multiple posture sensors 111 measure acceleration, angular velocity, geomagnetism, and the like. Measurement values ​​of each posture sensor 111 are transmitted as motion data 120 to the posture recognition system 150 via a communication unit 112.

[0011] Furthermore, a camera 130 is installed in the area where the worker 110 performs work. The camera 130 includes an imaging unit 131 and a communication unit 132. The imaging unit 131 captures an image of the worker 110. The communication unit 132 transmits image data 140 obtained as a result of the imaging by the imaging unit 131 to the posture recognition system 150.

[0012] The posture recognition system 150 includes a communication unit 151 , a sensor posture estimation unit 152 , a camera feature extraction unit 153 , a projection information estimation unit 154 , a sensor posture correction unit 155 , and a control unit 156 .

[0013] The communication unit 151 receives the action data 120 and the image data 140 . The sensor posture estimation unit 152 is a posture estimation unit that estimates the posture of the worker 110 based on the action data 120. The sensor posture estimation unit 152 manages the relative positional relationship of the posture sensors 111 in the initial state. Furthermore, the sensor posture estimation unit 152 receives the action data 120 and accumulates and manages the measurement values ​​for each posture sensor 111. The posture sensors 111 estimate three-dimensional posture data that indicates the posture of the worker 110 from the relative positional change in the initial state and the accumulation of the measurement values ​​of each posture sensor 111.

[0014] The camera feature extraction unit 153 is a feature extraction unit that extracts two-dimensional feature points of the worker 110 based on the image data 140. Any existing technology may be used to estimate the feature points from the image.

[0015] The projection information estimation unit 154 estimates projection information of the three-dimensional posture data that reproduces the arrangement of the two-dimensional feature points extracted by the camera feature extraction unit 153. The sensor attitude correction unit 155 corrects the three-dimensional attitude data based on the projection information estimated by the projection information estimation unit 154.

[0016] Control unit 156 controls the operation of posture recognition system 150. When a computer operates as posture recognition system 150 by executing a posture recognition program, control unit 156 is a CPU (Central Processing Unit), and realizes the functions of communication unit 151, sensor posture estimation unit 152, camera feature extraction unit 153, projection information estimation unit 154, and sensor posture correction unit 155 by expanding the program into memory and executing it.

[0017] FIG. 2 is an explanatory diagram of how the posture sensor 111 is worn. In FIG. 2, the worker 110 is wearing a clothing-type sensor with posture sensors 111 attached to a hat, left and right shoulders, left and right upper arms, left and right forearms, left and right waists, left and right thighs, and left and right shins. These posture sensors 111 do not need to be visible from the outside. In FIG. 2, the clothing-type sensor is provided with a communication unit 112 in the belt portion. The communication unit 112 may be provided in any position as long as it can acquire measurement values ​​from each posture sensor 111 and transmit them to the posture recognition system 150.

[0018] 3 is an explanatory diagram of posture estimation based on sensor output. The sensor posture estimation unit 152 has an estimation unit for each body part, such as an upper arm state estimation unit 301 and a waist state estimation unit 302. The estimation unit for each body part has a corresponding posture sensor 111, and estimates the state of the body part from the measurement value of the corresponding posture sensor 111. For example, the upper arm state estimation unit 301 estimates the change in the rotation angle of the upper arm over time using the measurement values ​​of the posture sensors 111 attached to the shoulder, upper arm, and forearm. Furthermore, the waist state estimation unit 302 estimates the time change in the waist bending angle using the measurement values ​​of the posture sensors 111 attached to the waist and thighs.

[0019] FIG. 4 is an explanatory diagram of 3D posture data based on sensor output. As shown in FIG. 4, the 3D posture data is data that indicates the subject's skeleton by connecting multiple feature points. The feature points include the positions of the subject's joints. For example, in FIG. 4, the feature points are the head, lumbar vertebrae, spine, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Here, the shoulder feature points can be in one-to-one correspondence with the shoulder posture sensors 111. That is, the position of the shoulder feature points is the position of a corresponding posture sensor 111. There is no posture sensor 111 that corresponds one-to-one with the elbow feature points. However, the position of the elbow feature points can be determined from the positions of the posture sensors 111 for the shoulders, upper arms, and forearms. In this way, the posture sensors 111 and feature points do not need to be in one-to-one correspondence. The skeleton of the worker 110 can be estimated from the relative positions of multiple posture sensors 111, and the joints, etc., can be used as feature points. The positions of feature points are expressed in three-dimensional spatial coordinates. The connections between feature points are set according to the body parts. For example, the connection between the feature point on the right buttock and the feature point on the right knee corresponds to the right thigh.

[0020] 5 is an explanatory diagram of the result of feature point extraction based on camera output. Camera feature extraction unit 153 processes image data 140 captured by camera 130 to extract feature points. Feature points include the positions of the subject's joints. Multiple feature points can be connected to indicate the subject's skeleton. The positions of the feature points are indicated by coordinates on the image, i.e., coordinates on a two-dimensional plane. Feature points based on camera output are free of cumulative errors, and feature points can be located with high reliability, especially for areas directly visible in the image.

[0021] Therefore, the projection information estimation unit 154 estimates projection information that projects the three-dimensional coordinates of feature points included in the three-dimensional posture data onto the coordinates of corresponding feature points on a two-dimensional plane, applies the projection information to the three-dimensional posture data to correct the three-dimensional posture data, and sets the difference in coordinates before and after the correction as a posture correction parameter.

[0022] 6 is a flowchart showing the generation of posture correction parameters, which starts with steps S601 and S602. Step S601: The sensor posture estimation unit 152 calculates the coordinates of three-dimensional feature points of each body part. Specifically, first, the sensor posture estimation unit 152 uses the motion data 120 to accumulate measurement values ​​for the multiple posture sensors 111 and determine the relative positional relationship of the multiple posture sensors 111. The sensor posture estimation unit 152 estimates the skeleton of the subject from the relative positional relationship of the multiple posture sensors 111. The sensor posture estimation unit 152 identifies feature points including joints from the estimated skeleton and calculates the coordinates of the feature points in three-dimensional space. The three-dimensional space is, for example, a space set in common for the multiple posture sensors 111. In other words, the coordinates of the multiple posture sensors 111 are expressed as coordinates in a given three-dimensional space, and the coordinates of the feature points are also calculated as coordinates in the same space.

[0023] Step S602: The camera feature extraction unit 153 calculates the coordinates of two-dimensional feature points of each body part. Specifically, the camera feature extraction unit 153 recognizes the image of the worker 110 from the image data 140, extracts a plurality of feature points including the positions of the worker's 110 joints, and calculates the coordinates of the feature points on a two-dimensional plane. This two-dimensional plane is, for example, a plane in which the horizontal direction of the image data 140 is the X-axis and the vertical direction is the Y-axis.

[0024] Steps S601 and S602 are executed independently and in parallel, after which the process proceeds to step S603. Step S603: The projection information estimation unit 154 repeats the following processes of S604 and S605 multiple times to calculate the projection error.

[0025] Step S604: The projection information estimation unit 154 samples a partial three-dimensional coordinate set from the three-dimensional coordinate set obtained by the sensor attitude estimation unit 152, and also samples a corresponding two-dimensional coordinate set. This process will be described. First, the set of all coordinates of multiple feature points in 3D space is defined as an original 3D coordinate set. Then, some feature point coordinates are excluded from the original 3D coordinate set, and this partial set of 3D coordinates is defined as a partial 3D coordinate set. Next, an original two-dimensional coordinate set is a set of all coordinates of a plurality of feature points on a two-dimensional plane based on image data 140. Then, a partial two-dimensional coordinate set is generated by excluding from the original two-dimensional coordinate set the coordinates of feature points corresponding to the feature points excluded from the partial three-dimensional coordinate set.

[0026] Step S605: The projection information estimation unit 154 calculates projection information that most appropriately projects the sampled 3D coordinate set, and calculates the calculation error. That is, the projection information estimation unit 154 generates projection information that most closely approximates the projection result of the partial 3D coordinate set to the partial 2D coordinate set, and calculates the projection error. Specifically, when the partial three-dimensional coordinate set is A and the partial two-dimensional coordinate set is B, a matrix P that satisfies B=PA is obtained as projection information.

[0027] The projection information estimation unit 154 repeatedly executes steps S604 to S605 by changing the feature points to be excluded, and then proceeds to step S606. In step S606, the projection information estimation unit 154 adopts the projection information with the smallest projection error from among the projection information obtained repeatedly as the optimum projection information, and the process proceeds to step S607. In step S607, the projection information estimation unit 154 converts all the sets of two-dimensional coordinates into corrected three-dimensional coordinate sets based on the adopted projection information, and then proceeds to step S608. Specifically, the projection information estimation unit 154 generates corrected three-dimensional coordinate sets by projecting the original two-dimensional coordinate sets into three-dimensional space using the projection information.

[0028] In step S608, the projection information estimation unit 154 calculates the positional deviation between the corrected 3D coordinate set and the original 3D coordinate set obtained by the sensor posture estimation unit 152, and stores the deviation as a posture correction parameter. Thereafter, the sensor posture correction unit 155 corrects the feature points estimated from the motion data 120 with the posture correction parameter, thereby enabling accurate recognition of the posture of the worker 110.

[0029] In this way, the projection information estimation unit 154 repeats steps S604 to S605 by changing the feature points to be excluded. If a large error occurs in any of the feature points in the original 3D coordinate set, the projection error will be minimized when that feature point is excluded. Therefore, by using the projection information that minimizes the projection error, it is possible to obtain posture correction parameters that correct the coordinates of the feature points in 3D space with high accuracy. It is possible to appropriately set which feature points to exclude. All feature points may be excluded sequentially, or feature points may be excluded randomly. When excluding feature points, it is desirable to take into account the hierarchical structure of the skeleton. For example, if there is a large error in the feature point of the elbow, all feature points on the distal side of the elbow (wrist, fingers, etc.) will be affected by the error of the elbow. Therefore, when excluding a feature point corresponding to a joint, it is sufficient to also exclude feature points corresponding to joints located distal to the joint in question.

[0030] 7 is an explanatory diagram of a skeleton structure tree. The feature points based on the orientation sensor 111 and the feature points based on the camera 130 indicate the skeleton of the worker 110 by a skeleton structure tree.

[0031] In Figure 7, the lumbar vertebrae (for convenience, the lowest point of the lumbar vertebrae is taken as the feature point of the lumbar vertebrae) which is one of the feature points is used as the reference. Specifically, the feature point "lumbar vertebrae" is connected to the feature point "spine (for convenience, the lowest point of the thoracic vertebrae is taken as the feature point of the spine)", the feature point "right buttocks", and the feature point "left buttocks". Furthermore, the feature point "spine" is connected to the feature point "left shoulder," feature point "head," and feature point "right shoulder." The feature point "left shoulder" is further connected to the feature point "left elbow," and the feature point "left elbow" is connected to the feature point "left wrist." Similarly, the feature point "right shoulder" is further connected to the feature point "right elbow," and the feature point "right elbow" is connected to the feature point "right wrist." Furthermore, the feature point "right buttock" is further connected to the feature point "right knee," and the feature point "right knee" is connected to the feature point "right ankle." Similarly, the feature point "left buttock" is further connected to the feature point "left knee," and the feature point "left knee" is connected to the feature point "left ankle."

[0032] 8 is an explanatory diagram of the correction of the attitude. The sensor attitude correction unit 155 associates the feature points included in the original three-dimensional coordinate set with the feature points included in the corrected three-dimensional coordinate set to find the correction parameters. The original 3D coordinate set is a set of coordinates of feature points estimated by a clothing-type sensor equipped with an orientation sensor 111. The corrected 3D coordinate set is a set of coordinates obtained by converting the positions of feature points estimated by the camera 130 into 3D positions.

[0033] In Figure 8, the lumbar and spinal feature points are compared between the original and corrected 3D coordinate sets. Furthermore, a correction vector that corrects the position of the spine in the original three-dimensional coordinate set to the position of the spine in the corrected three-dimensional coordinate set is the correction parameter for the spine.

[0034] FIG. 9 shows an example of the output of posture recognition results. FIG. 9 shows a graphic user interface that allows the user to check the posture recognition results by superimposing the projection results of posture recognition by the clothing-mounted sensor, the posture recognition results by the camera, and the corrected posture on the camera image. The projection results of posture recognition by the clothing-mounted sensor, the posture recognition results by the camera, and the corrected posture may be superimposed unconditionally or may be selectable. Furthermore, they may be drawn using computer graphics instead of camera images.

[0035] In the explanation so far, an example has been given in which the feature points estimated by the sensor attitude estimation unit 152 and the feature points extracted by the camera feature extraction unit 153 correspond one-to-one to each other, but the feature points estimated by the sensor attitude estimation unit 152 and the feature points extracted by the camera feature extraction unit 153 do not necessarily have to correspond to each other.

[0036] Fig. 10 shows a modified example of feature points extracted by the camera feature extraction unit 153. In Fig. 10, the feature points extracted by the camera feature extraction unit 153 do not include feature points corresponding to the spine, but include feature points corresponding to the cervical vertebrae (for convenience, the lowest part of the cervical vertebrae is considered to be the feature points of the cervical vertebrae).

[0037] In this way, when feature points that do not correspond to feature points estimated by sensor orientation estimation unit 152 are included, camera feature extraction unit 153 finds virtual feature points that correspond to the feature points estimated by sensor orientation estimation unit 152 from the positional relationships of the extracted feature points. It is preferable to determine in advance the correspondence relationships between the feature points estimated by sensor orientation estimation unit 152 and the feature points extracted by camera feature extraction unit 153, the necessary virtual feature points, and a method for calculating the coordinates of the virtual feature points. In Figure 11, the coordinates of the feature points of the spine are the points that divide the cervical vertebrae and lumbar vertebrae internally in an N:M ratio. When virtual feature points are obtained in this way, the coordinate set including the virtual feature points is set as the original two-dimensional coordinate set. That is, the feature points of the original two-dimensional coordinate set and the feature points of the original three-dimensional coordinate set correspond one-to-one.

[0038] As described above, the posture recognition system 150 of the first embodiment executes a posture estimation step of estimating three-dimensional posture data indicating the posture of the worker 110 based on the sensing result of the posture sensor 111 worn by the worker 110, a feature extraction step of extracting two-dimensional feature points of the worker 110 based on a captured image of the worker 110, a projection information estimation step of estimating projection information of the three-dimensional posture data that reproduces the arrangement of the two-dimensional feature points extracted in the feature extraction step, and a posture correction step of correcting the three-dimensional posture data based on the projection information estimated in the projection information estimation step. As a result, errors in the posture sensor 111 can be corrected, and highly accurate posture estimation can be achieved. The correction parameters for correcting the three-dimensional posture data are generated using the sensing results and the image when the image contains the image of the worker 110. Once generated, the correction parameters can also be used to correct the three-dimensional posture data when the image does not contain the image of the worker 110. [Example]

[0039] In the second embodiment, a posture recognition system will be described that stores generated projection information and uses past projection information when generating new projection information. 12 is a configuration diagram of a posture recognition system according to Example 2. The posture recognition system 150 shown in FIG.

[0040] The projection information storage unit 1201 stores the projection information generated by the projection information estimation unit 154. Since the projection information is generated from the motion data 120 and the image data 140 at a certain point in time, it is preferable to store the generated projection information with corresponding time information. The projection information includes data specifying the matrix P and data indicating which feature points have been excluded. Other configurations and operations are the same as those in the first embodiment, so a description thereof will be omitted.

[0041] Fig. 13 is a flowchart of the generation of the posture correction parameters according to the embodiment 2. The difference from the flowchart of the generation of the posture correction parameters according to the embodiment 1 shown in Fig. 6 is that step S604 is replaced with step S1301.

[0042] In step S1301, the projection information estimation unit 154 samples a partial 3D coordinate set from the original 3D coordinate set obtained by the sensor attitude estimation unit 152. The sampling refers to past projection information stored in the projection information storage unit 1201, and prioritizes partial 3D coordinate sets for which projection was successful. The corresponding partial 2D coordinate sets are also extracted. In other words, when sampling is redone for newly acquired data, past sampling results (which feature points resulted in the least error when projected) are referenced, and the results are prioritized in order of success. The other steps are the same as those in the first embodiment, and therefore the explanation will be omitted.

[0043] In this way, in the posture recognition system 150 of the second embodiment, when there is a partial 3D coordinate set that has been used to generate posture correction parameters in the past, the projection information estimation unit 154 obtains new posture correction parameters by prioritizing a partial 3D coordinate set that is configured with the same combination of feature points as the partial 3D coordinate set. This makes it possible to obtain new posture correction parameters efficiently. [Example]

[0044] In the third embodiment, a posture recognition system that generates projection information taking into consideration whether the posture is appropriate will be described. 14 is a configuration diagram of a posture recognition system according to Example 3. The posture recognition system 150 shown in FIG.

[0045] The abnormal posture determination unit 1401 determines whether or not there is an abnormality in the posture estimated from the motion data 12. The abnormal posture determination unit 1401 holds range of motion data indicating the range of motion of body parts, and determines whether or not each body part in the estimated posture is within the range of motion. Then, a body part that is outside the range of motion is determined to be a body part with an abnormal posture. For example, if an elbow is bent in the opposite direction to its normal position, the elbow is determined to be a body part with an abnormal posture. Other configurations and operations are the same as those in the first embodiment, so a description thereof will be omitted.

[0046] Fig. 15 is a flowchart of generating a posture correction parameter according to the third embodiment. The difference from the flowchart of generating a posture correction parameter according to the first embodiment shown in Fig. 6 is that step S1501 is added before step S603 and step S604 is replaced with step S1502.

[0047] In step S1501, the abnormal posture determining unit 1401 determines whether or not the three-dimensional feature points acquired in step S601 include a body part in an abnormal posture, and the process proceeds to step S603.

[0048] In step S1502, the projection information estimation unit 154 samples a partial 3D coordinate set from the original 3D coordinate set obtained by the sensor attitude estimation unit 152. The sampling refers to the determination result by the abnormal attitude determination unit 1401, and prioritizes feature points with a low degree of abnormality. Then, the corresponding partial 2D coordinates are also extracted. That is, if there is a clearly abnormal body part among the three-dimensional feature points obtained from the output of the posture sensor 111, the priority at the time of sampling is lowered after a pre-determined judgment. The other steps are the same as those in the first embodiment, and therefore the explanation will be omitted.

[0049] FIG. 16 is an explanatory diagram of the range of motion data. The range of motion data includes items such as body part, rotation direction, minimum value, and maximum value. A body part is a movable part of the body, such as the neck or waist. One or more rotation directions are associated with each body part, and a minimum and maximum value are set for each rotation direction. The minimum and maximum values ​​are angles relative to the reference direction, which is 0 degrees.

[0050] Specifically, it has been shown that the neck can rotate forward and backward within a range of -90 degrees to +90 degrees. It has also been shown that the neck can rotate left and right within a range of -90 degrees to +90 degrees. It has also been shown that the neck can twist within a range of -90 degrees to +90 degrees. Similarly, the hips are shown to be able to rotate forward and backward between -45 and +90 degrees, left and right between -60 and +60 degrees, and twist between -60 and +60 degrees.

[0051] In this way, in the posture recognition system 150 of the third embodiment, when there is a joint located outside the range of motion of the skeleton, projection information is generated by using the feature point corresponding to the joint as a candidate for exclusion, thereby enabling new posture correction parameters to be obtained efficiently. [Example]

[0052] In the fourth embodiment, a posture recognition system that generates correction parameters when a worker assumes a predetermined posture will be described. 17 is a configuration diagram of a posture recognition system according to Example 4. The posture recognition system 150 shown in FIG.

[0053] The posture storage unit 1701 stores a predetermined posture suitable for generating correction parameters. A posture suitable for generating correction parameters is a posture from which as many feature points as possible can be extracted with high accuracy from the image data 140. For example, when the worker 110 faces the camera 130, stands with his arms outstretched, and his entire body is captured in a sufficiently large image, as many feature points as possible can be extracted with high accuracy. In this case, it is preferable that as many feature points as possible can also be estimated with high accuracy from the posture sensor 111.

[0054] The posture correction determination unit 1702 compares the posture estimated by the sensor posture estimation unit 152 with the posture stored in the posture storage unit 1701 to determine the similarity. If the similarity is equal to or greater than a threshold, the unit generates correction parameters. The similarity may be calculated, for example, by referring to the angles of each joint and determining whether they fall within a certain range. Alternatively, the posture similarity may be determined using machine learning. Other configurations and operations are the same as those in the first embodiment, so a description thereof will be omitted.

[0055] Fig. 18 is a flowchart of the generation of posture correction parameters according to the fourth embodiment. This flowchart differs from the flowchart of the generation of posture correction parameters according to the first embodiment shown in Fig. 6 in that step S1801 is added before step S603.

[0056] In step S1801, the attitude correction determination unit 1702 determines whether the attitude estimated by the sensor attitude estimation unit 152 is suitable for correction. Specifically, if the similarity between the attitude estimated by the sensor attitude estimation unit 152 and the attitude stored in the attitude storage unit 1701 is equal to or greater than a threshold, it is determined that the attitude is suitable for correction. If it is determined that the attitude is suitable for correction, the process proceeds to step S603. If it is determined that the attitude is not suitable for correction, the process ends.

[0057] In this way, the posture recognition system 150 of the fourth embodiment generates correction parameters for correcting three-dimensional posture data using the sensing results and images obtained when the worker 110 is in a predetermined posture. These correction parameters can be used to correct three-dimensional posture data when the worker 110 is not in the predetermined posture. Therefore, highly accurate correction parameters can be generated efficiently.

[0058] As described above, posture recognition system 150 that recognizes the posture of a subject is characterized by including sensor posture estimation unit 152, which is a posture estimation unit that estimates three-dimensional posture data indicating the posture of the subject based on the sensing result of posture sensor 111, which is a sensor worn by the subject; camera feature extraction unit 153, which is a feature extraction unit that extracts two-dimensional feature points of the subject based on an image of the subject; projection information estimation unit 154 that estimates projection information of the three-dimensional posture data to reproduce the arrangement of the two-dimensional feature points extracted by the feature extraction unit; and sensor posture correction unit 155, which is a posture correction unit that corrects the three-dimensional posture data based on the projection information estimated by projection information estimation unit 154. With this configuration and operation, the posture recognition system 150 can achieve highly accurate posture estimation.

[0059] Further, the sensors are sensors that measure acceleration and / or angular velocity, and the sensors are attached to multiple locations on the subject's body, and the posture estimation unit accumulates measurement values ​​from the multiple sensors attached to multiple locations on the subject's body, determines the relative positional relationship of the multiple sensors, and estimates the subject's skeleton from the relative positional relationship of the multiple sensors. Therefore, highly accurate posture estimation can be achieved by correcting the posture estimation results from the wearable sensor using camera images.

[0060] In addition, the posture estimation unit estimates a skeleton of the subject that connects a plurality of feature points including positions of the subject's joints, and outputs the coordinates of the plurality of feature points in three-dimensional space as the three-dimensional posture data, the feature extraction unit extracts coordinates of the plurality of feature points including the positions of the subject's joints in a two-dimensional plane, and the projection information estimation unit estimates projection information that projects the coordinates of the plurality of feature points in the three-dimensional space onto the coordinates of corresponding feature points in the two-dimensional plane. Therefore, the posture recognition system 150 can use feature points on a two-dimensional plane where errors do not accumulate, and correct feature points in three-dimensional space where errors accumulate.

[0061] The projection information estimation unit also generates a partial three-dimensional coordinate set by excluding coordinates of some feature points from an original three-dimensional coordinate set, which is a set of coordinates of a plurality of feature points in the three-dimensional space; generates a partial two-dimensional coordinate set corresponding to the partial three-dimensional coordinate set from an original two-dimensional coordinate set, which is a set of coordinates of a plurality of feature points on the two-dimensional plane; generates projection information such that the projection result of the partial three-dimensional coordinate set most closely approximates the partial two-dimensional coordinate set; generates a corrected three-dimensional coordinate set by projecting the original two-dimensional coordinate set into three-dimensional space using the projection information; and defines the difference between the original three-dimensional coordinate set and the corrected three-dimensional coordinate set as an attitude correction parameter. This configuration and operation makes it possible to eliminate the influence of local errors and obtain highly accurate correction parameters.

[0062] In addition, the projection information estimation unit generates multiple partial 3D sets with different excluded feature points, generates projection information for each of the multiple partial 3D sets, calculates a projection error for each piece of projection information, and generates the posture correction parameter from the projection information with the smallest projection error. According to this configuration and operation, local errors can be efficiently eliminated and correction parameters can be obtained.

[0063] Furthermore, the projection information estimation unit excludes feature points corresponding to any of the joints and feature points corresponding to joints located distal to the joint in question, based on the connection relationships of the joints in the skeleton. According to this configuration and operation, highly accurate correction parameters can be obtained efficiently by taking into consideration the connection relationships of the joints in the skeleton.

[0064] In addition, when a partial 3D coordinate set that was previously used to generate the posture correction parameters exists, the projection information estimation unit prioritizes a partial 3D coordinate set that is composed of the same combination of feature points as the partial 3D coordinate set, and calculates new posture correction parameters. According to this configuration and operation, highly accurate correction parameters can be obtained efficiently by utilizing past performance.

[0065] Furthermore, if there is a joint located outside the range of motion of the skeleton, the posture recognition system 150 excludes the feature points corresponding to that joint. According to this configuration and operation, highly accurate correction parameters can be obtained efficiently by taking the range of motion into consideration.

[0066] The posture correction unit generates correction parameters for correcting the three-dimensional posture data using the sensing results and images when the subject is in a predetermined posture. According to this configuration and operation, highly accurate correction parameters can be easily obtained.

[0067] In addition, the posture correction unit generates correction parameters for correcting the three-dimensional posture data using the sensing results and images when the image contains the image of the subject, and uses these to correct the three-dimensional posture data when the image does not contain the image of the subject. According to this configuration and operation, the generated correction parameters can be effectively utilized.

[0068] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, not only can the configurations be deleted, but also replacements and additions of configurations are possible. [Explanation of symbols]

[0069] 110: worker, 111: posture sensor, 112: communication unit, 120: motion data, 130: camera, 131: imaging unit, 132: communication unit, 135: sensor posture correction unit, 140: image data, 150: posture recognition system, 151: communication unit, 152: sensor posture estimation unit, 153: camera feature extraction unit, 154: projection information estimation unit, 155: sensor posture correction unit, 156: control unit, 1201: projection information storage unit, 1401: abnormal posture determination unit, 1701: posture storage unit, 1702: posture correction determination unit

Claims

1. A posture recognition system for recognizing a posture of a subject, a posture estimation unit that estimates three-dimensional posture data indicating a posture of the subject based on a sensing result of a sensor attached to the subject; a feature extraction unit that extracts two-dimensional feature points of the subject based on an image of the subject; a projection information estimation unit that reproduces the arrangement of the two-dimensional feature points extracted by the feature extraction unit and estimates projection information of the three-dimensional posture data; an attitude correction unit that corrects the three-dimensional attitude data based on the projection information estimated by the projection information estimation unit; A posture recognition system comprising:

2. The posture recognition system according to claim 1 , the sensor is a sensor that measures acceleration and / or angular velocity, The sensors are attached to a plurality of locations on the subject's body, The posture estimation unit accumulates measurement values ​​from multiple sensors attached to multiple locations on the subject's body, determines the relative positional relationship of the multiple sensors, and estimates the subject's skeleton from the relative positional relationship of the multiple sensors.

3. The posture recognition system according to claim 1 , the posture estimation unit estimates a skeleton of the subject by connecting a plurality of feature points including positions of the subject's joints, and outputs coordinates of the plurality of feature points in a three-dimensional space as the three-dimensional posture data; the feature extraction unit extracts coordinates of a plurality of feature points including positions of the joints of the subject in a two-dimensional plane; The projection information estimation unit estimates projection information for projecting coordinates of a plurality of feature points in the three-dimensional space onto coordinates of corresponding feature points on the two-dimensional plane. A posture recognition system characterized by:

4. The posture recognition system according to claim 3, The projection information estimation unit generating a partial three-dimensional coordinate set by excluding coordinates of some of the feature points from an original three-dimensional coordinate set, which is a set of coordinates of a plurality of feature points in the three-dimensional space; generating a partial two-dimensional coordinate set corresponding to the partial three-dimensional coordinate set from an original two-dimensional coordinate set which is a set of coordinates of a plurality of feature points on the two-dimensional plane; generating projection information that most closely approximates the partial three-dimensional coordinate set to the partial two-dimensional coordinate set; generating a modified three-dimensional coordinate set by projecting the original two-dimensional coordinate set into three-dimensional space using the projection information; The difference between the original three-dimensional coordinate set and the corrected three-dimensional coordinate set is defined as a posture correction parameter. A posture recognition system characterized by:

5. 5. The posture recognition system according to claim 4, The projection information estimation unit generates a plurality of partial 3D sets with different excluded feature points, generates projection information for each of the plurality of partial 3D sets, calculates a projection error for each projection information, and generates the posture correction parameter from the projection information with the smallest projection error.

6. 5. The posture recognition system according to claim 4, a projection information estimation unit that excludes feature points corresponding to any joint and feature points corresponding to joints located distal to the joint in question based on a connection relationship between the joints in the skeleton.

7. 5. The posture recognition system according to claim 4, A posture recognition system characterized in that, when a partial 3D coordinate set that was previously used to generate the posture correction parameters exists, the projection information estimation unit prioritizes a partial 3D coordinate set that is composed of the same combination of feature points as the partial 3D coordinate set in order to determine new posture correction parameters.

8. 5. The posture recognition system according to claim 4, A posture recognition system characterized in that, if there is a joint located outside the range of motion of the skeleton, feature points corresponding to that joint are excluded.

9. The posture recognition system according to claim 1 , The posture correction unit generates correction parameters for correcting the three-dimensional posture data using sensing results and images obtained when the subject is in a predetermined posture.

10. The posture recognition system according to claim 1 , The posture correction unit generates correction parameters for correcting the three-dimensional posture data using sensing results and images when the image contains an image of the subject, and uses the correction parameters to correct the three-dimensional posture data when the image does not contain an image of the subject.

11. A posture recognition method for recognizing a posture of a subject, comprising: The computer a posture estimation step of estimating three-dimensional posture data indicating a posture of the subject based on a sensing result of a sensor attached to the subject; a feature extraction step of extracting two-dimensional feature points of the subject based on an image of the subject; a projection information estimation step of estimating projection information of the three-dimensional posture data by reproducing an arrangement of the two-dimensional feature points extracted by the feature extraction step; an attitude correction step of correcting the three-dimensional attitude data based on the projection information estimated by the projection information estimation step; A posture recognition method comprising:

Images