Automatic posture evaluation device, automatic posture evaluation method, and automatic posture evaluation program

Abstract

To provide a posture automatic evaluation device, a posture automatic evaluation method, and a posture automatic evaluation program, which quantitatively measure a load of a posture of a moving body at every fixed time in a non-contant manner to evaluate the load.SOLUTION: A posture automatic evaluation device 1 for a moving body MO includes a measurement unit 10, an evaluation unit 20, and an evaluation result display unit 30. The measurement unit 10 includes: an image acquisition unit 11 that images the moving body; and a distance acquisition unit 12 that is temporally synchronized with photographing by the image acquisition unit 11, has a field of view overlapping with the photographed image, and measures a distance to the moving body. The evaluation unit 20 includes: a feature point (articulation) position determination unit 21 that extracts any feature point from the photographed image; a feature point (articulation) distance calculation unit 22 that calculates a space distance to the extracted feature point; a three-dimensional skeleton reconstruction unit 23 that reconstructs a skeleton of the moving body; and a posture evaluation unit 24 that evaluates a posture of the moving body. The evaluation result display unit 30 displays an evaluation result with at least one of a numerical value, a character, a sound and a three-dimensional skeleton diagram.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an automatic posture evaluation device, an automatic posture evaluation method, and an automatic posture evaluation program for a moving object. [Background technology]

[0002] Conventionally, various efforts have been made to manage worker safety and reduce workloads at manufacturing sites in the manufacturing industry and at manual construction sites. To obtain basic data for work improvement, workers' postures are periodically visually checked by a person in charge of measurement. However, there are problems with measuring over long periods of time, arbitrariness due to human judgment, and workers hiding behind obstacles. To address these situations, for example, safety management devices (see Patent Document 1) and construction machines (see Patent Document 2) have been proposed that monitor the postures and positions of workers at manufacturing and construction sites to manage safety. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-005229 [Patent Document 2] Japanese Patent Publication No. 2020-183623 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the measurer visually checks the worker, it is difficult to eliminate the possibility of inaccurate measurements or subjective judgments over long periods of time, and the proposals in the prior art have the problem that they are not necessarily sufficient to grasp and evaluate the posture and position of moving objects in general, including moving objects other than the human body of the worker, such as animals and robots, or in response to the various behavioral patterns and behavioral environments of each moving object.

[0005] The present invention has been made with an eye on such problems, and aims to provide an automatic posture evaluation device, an automatic posture evaluation method, and an automatic posture evaluation program that quantitatively measure the posture load of a moving object at regular intervals without contact and evaluate that load. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, a first aspect of the present invention is an automatic posture evaluation device for a moving object, comprising a measurement unit that measures the moving object, an evaluation unit that evaluates the measurement results of the measurement unit, and an evaluation result display unit that displays the evaluation results of the evaluation unit, wherein the measurement unit comprises an image acquisition unit that photographs the moving object, and a distance acquisition unit that measures the distance to the moving object, which is synchronized in time with the photographing by the image acquisition unit and has a field of view that overlaps with the photographed image, and the evaluation unit comprises a feature point position determination unit that extracts any feature point from the photographed image, a feature point distance calculation unit that calculates the spatial distance to the extracted feature point, a 3D skeletal reconstruction unit that reconstructs the skeleton of the moving object, and a posture evaluation unit that evaluates the posture of the moving object, and the evaluation result display unit displays the evaluation result using at least one of numerical information, text information, sound information, and a 3D skeletal diagram.

[0007] In order to solve the above-mentioned problems, a second aspect of the present invention is a method for automatically evaluating the posture of a moving object, comprising: a measurement step of measuring the moving object; an evaluation step of evaluating the measurement results of the measurement step; and an evaluation result display step of displaying the evaluation results of the evaluation step, wherein the measurement step comprises: an image acquisition step of photographing the moving object; and a distance acquisition step of measuring the distance to the moving object, which is synchronized in time with the photographing by the image acquisition step and has a field of view that overlaps with the photographed image, and the evaluation step comprises: a feature point position determination step of extracting any feature point from the photographed image; a feature point distance calculation step of calculating the spatial distance to the extracted feature point; a 3D skeletal reconstruction step of reconstructing the skeleton of the moving object; and a posture evaluation step of evaluating the posture of the moving object, wherein the evaluation result display step displays the evaluation result using at least one of numerical information, text information, sound information, and a 3D skeletal diagram.

[0008] In order to solve the above-mentioned problems, a third aspect of the present invention is an automatic posture evaluation program for a moving object, which causes a computer to execute a measurement process for measuring the moving object, an evaluation process for evaluating the measurement results of the measurement process, and an evaluation result display process for displaying the evaluation results of the evaluation process, wherein the measurement process comprises an image acquisition process for photographing the moving object, and a distance acquisition process for measuring the distance to the moving object, which is synchronized in time with the photographing by the image acquisition process and has a field of view that overlaps with the photographed image, and the evaluation process comprises a feature point position determination process for extracting any feature point from the photographed image, a feature point distance calculation process for calculating the spatial distance to the extracted feature point, a three-dimensional skeleton reconstruction process for reconstructing the skeleton of the moving object, and a posture evaluation process for evaluating the posture of the moving object, and the evaluation result display process displays the evaluation result using at least one of numerical information, text information, sound information, and a three-dimensional skeleton diagram. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an automatic posture evaluation device, an automatic posture evaluation method, and an automatic posture evaluation program that quantitatively measure the posture load of a moving object at regular intervals without contact and evaluate the load. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating the overall configuration (measurement unit, evaluation unit, evaluation result display unit) of an automatic posture evaluation device according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram (part 1) illustrating the measurement unit. [Figure 3] FIG. 2 is a diagram (part 2) illustrating the measurement unit. [Figure 4] FIG. 3 is a diagram (part 3) for explaining the measurement unit. [Figure 5] FIG. 2 is a diagram illustrating a feature point (joint) position determination unit in the evaluation unit. [Figure 6] FIG. 10 is a diagram illustrating an example of a location of a feature point. [Figure 7] This is a diagram (part 1) explaining the problems with the conventional template matching method using a stereo camera. [Figure 8] This is a diagram (part 2) explaining the problems with the conventional template matching method using a stereo camera. [Figure 9] This is the problem with the conventional template matching method using a stereo camera (Figure 3 explaining it). [Figure 10] This is a diagram (part 4) explaining the problems with the conventional template matching method using a stereo camera. [Figure 11] This is a diagram (part 5) explaining the problems with the conventional template matching method using a stereo camera. [Figure 12] FIG. 10 is a diagram illustrating an algorithm of a feature point (joint) position distance calculation unit. [Figure 13] FIG. 2 is a diagram illustrating a feature point (joint) distance calculation unit in the evaluation unit. [Figure 14] FIG. 10 is a diagram illustrating a three-dimensional skeleton reconstruction unit in the evaluation unit. [Figure 15] FIG. 2 is a diagram illustrating a posture evaluation unit in the evaluation unit. [Figure 16] FIG. 1 is a diagram (part 1) illustrating the OWAS method as an example of posture evaluation. [Figure 17] FIG. 2 is a diagram (part 2) illustrating the OWAS method as an example of posture evaluation. [Figure 18] FIG. 10 is a diagram (part 1) for explaining the algorithm of the posture evaluation unit. [Figure 19] FIG. 10 is a diagram (part 2) for explaining the algorithm of the posture evaluation unit. [Figure 20] FIG. 10 is a diagram (part 3) for explaining the algorithm of the posture evaluation unit. [Figure 21] FIG. 10 is a diagram (part 4) for explaining the algorithm of the posture evaluation unit. [Figure 22] FIG. 5 is a diagram (part 5) for explaining the algorithm of the posture evaluation unit. [Figure 23] FIG. 6 is a diagram (part 6) for explaining the algorithm of the posture evaluation unit. [Figure 24] FIG. 7 is a diagram (part 7) for explaining the algorithm of the posture evaluation unit. [Figure 25] FIG. 10 is a diagram (part 1) for explaining the display by the evaluation result display unit. [Figure 26] FIG. 10 is a diagram (part 2) for explaining the display by the evaluation result display unit. [Figure 27] FIG. 10 is a diagram (part 3) for explaining the display by the evaluation result display unit. [Figure 28] FIG. 10 is a diagram (part 4) for explaining the display by the evaluation result display unit. [Figure 29] FIG. 5 is a diagram (part 5) for explaining the display by the evaluation result display unit. [Figure 30] FIG. 6 is a diagram (part 6) for explaining the display by the evaluation result display unit. [Figure 31] FIG. 7 is a diagram (part 7) for explaining the display by the evaluation result display unit. [Figure 32] FIG. 8 is a diagram (part 8) for explaining the display by the evaluation result display unit. DETAILED DESCRIPTION OF THE INVENTION

[0011] [Embodiment] Hereinafter, embodiments of the present invention will be described in detail with reference to Figures 1 to 32. Note that the present invention is not limited to the following embodiments, and various modifications are possible without departing from the gist of the present invention.

[0012] [Overall configuration of the automatic posture evaluation device] First, the overall configuration of an automatic posture evaluation device 1 according to this embodiment will be described with reference to Fig. 1. In this specification, each component of the automatic posture evaluation device 1 is given a name that represents the operation that it performs.

[0013] The posture automatic evaluation device 1 evaluates the posture of at least one moving object MO, which may be a human body, an animal, or a robot. In Figure 1 and other figures, a human body is illustrated as an example of a moving object MO, but it should be noted that the moving object MO is not limited to a human body. As shown in Figure 1, the posture automatic evaluation device 1 for a moving object MO includes a measurement unit 10 that measures the moving object MO, an evaluation unit 20 that evaluates the measurement results by the measurement unit 10, and an evaluation result display unit 30 that displays the evaluation results by the evaluation unit 20.

[0014] The measurement unit 10 includes an image acquisition unit 11 that captures images of a moving object MO, and a distance acquisition unit 12 that measures the distance to the moving object MO, synchronized with the image capture by the image acquisition unit 11 and having a field of view that overlaps with the captured image. The evaluation unit 20 includes a feature point position determination unit (joint position determination unit) 21 that extracts any feature point (e.g., joint position) from the captured image, a feature point distance calculation unit (joint distance calculation unit) 22 that calculates the spatial distance to the extracted feature point (e.g., joint position), a 3D skeleton reconstruction unit 23 that reconstructs the skeleton of the moving object MO, and a posture evaluation unit 24 that evaluates the posture of the moving object MO. The evaluation result display unit 30 displays the evaluation result using at least one of numerical information, text information, sound information, and a 3D skeleton diagram. Each unit will be described in detail below.

[0015] [Measurement section] Next, examples of the measurement unit 10 will be described with reference to Fig. 2 to Fig. 4. Fig. 2 shows Example 1 of the measurement unit 10, Fig. 3 shows Example 2 of the measurement unit 10, and Fig. 4 shows Example 3 of the measurement unit 10.

[0016] As shown in Figure 2, Example 1 is an example in which multiple optical cameras 11a1, 11a2, and 11a3 whose spatial positions are known in advance are set up in synchronization as the image acquisition unit 11, and multiple images P1, P2, and P3 of a moving object MO are simultaneously captured by these cameras, and the distance acquisition unit 12 calculates the distance from the parallax on the image of the same moving object MO to acquire distance information D.

[0017] As shown in FIG. 3, Example 2 uses an optical camera 11a whose spatial position is known in advance as the image acquisition unit 11, and a distance camera 12a (TOF) whose spatial position is known in advance as the distance acquisition unit 12. <time-of-flight>Camera, pattern projection camera, laser rangefinder, laser scanner, lidar<LiDAR:Light Detection and Ranging> In this example, two distance cameras (devices capable of directly measuring distance information such as distance from a moving object MO, hereinafter referred to as "distance cameras") are installed in synchronization with each other and simultaneously capture images of a moving object MO, thereby separately acquiring an image P and distance information D. In this case, the optical camera 11a and the distance camera 12a do not necessarily have to have the same field of view, but the moving object MO, which is the object to be measured, must be included in each field of view.

[0018] As shown in Fig. 4, Example 3 is an example that combines Example 1 and Example 2 of the measurement unit 10. In Fig. 4, multiple optical cameras 11a1, 11a2, and 11a3 are used as image acquisition unit 11 to acquire multiple wide-area images P1, P2, and P3, and multiple distance cameras 12a1 and 12a2 are used as distance acquisition unit 12 to acquire distance information D of a specific field of view range.

[0019] Example 3 shown in Figure 4 estimates distance information D to a moving object MO over a wide area based on the parallax of the moving object MO in multiple images P1, P2, and P3 obtained by multiple optical cameras 11a1, 11a2, and 11a3, and can be easily expanded to a mode in which the estimated distance information D for the field of view captured by distance cameras 12a1 and 12a2 is supplemented with distance information D obtained by distance cameras 12a1 and 12a2.

[0020] The image acquisition unit 11 outputs the image P1 or P acquired by each of the above examples to the feature point position determination unit (joint position determination unit) 21 of the evaluation unit 20, and the distance acquisition unit 12 outputs the similarly acquired distance information D to the feature point position calculation unit (joint position calculation unit) 22 of the evaluation unit 20.

[0021] [Evaluation Department] [[Joint position determination unit]] Next, an example of the feature point position determination unit (joint position determination unit) 21 of the evaluation unit 20 will be described with reference to FIGS. 5 and 6. While feature points are not limited to joints, the following description will be given assuming that the feature points are joints, and will refer to the joint position determination unit 21. As shown in FIG. 5, the joint position determination unit 21 is provided with weighting data 21a for joint position determination, which has been previously learned by machine learning for video images of a moving object MO (video images of a human body in the case of a worker). Using this weighting data, the unit 21 estimates the presence or absence of images corresponding to joints in the images P1 or P input from the image acquisition unit 11 of the measurement unit 10, and performs joint position determination 21b. Various machine learning methods have been proposed, but the method need not be limited to a specific one. It is sufficient to input one frame of the video and extract, for example, two-dimensional coordinates TwD of the joints (here, the right shoulder SR, left shoulder SL, and right elbow ER) on the optical camera 11a1 or 11a as feature points of the human body shown in FIG. 6, as illustrated in the lower part of FIG. 5.

[0022] 6 illustrates the following examples of parts including joints and line segments as feature points when the moving object MO is a human body: HD: head, S: shoulder width, shoulders: SL / SR, midpoint of shoulders: S0, B: back, W: waist width, hips: WL / WR, midpoint of hips: W0, upper arms: ArmL1 / ArmR1, elbows: EL / ER, forearms: ArmL2 / ArmR2, wrists: HL / HR, thighs: ThiL / ThiR, knees: KL / KR, lower legs: LlimL / LlimR, ankles: AnL / AnR.

[0023] As mentioned above, the feature points are not limited to the various parts including the joint parts shown in FIG. 6 . They may be set arbitrarily depending on the type of moving object MO, or even if the moving object MO is a human body, depending on the difference in the basic posture to be evaluated. For example, if the moving object MO is an animal, the ears or tail of a dog may be set as feature points. In the case of a robot, if the working unit rotates relative to a base rather than a human form, the reference point of the working unit may be set as feature points. Furthermore, if the human body is normally lying down, the head HD and back B may also be set as feature points separately for the left and right sides in order to evaluate the rolling posture. In this sense, the joint position determination unit 21 can be expanded and understood as a feature point position determination unit 21.

[0024] [[Joint distance calculation part]] Next, an example of the joint distance calculation unit 22 of the evaluation unit 20 is shown in FIG. 13, which will be explained later. Prior to that explanation, the issues with joint distance calculation using a conventional stereo camera will be described with reference to FIGS. 7 to 11, and the joint distance calculation according to this embodiment will be described with reference to FIG. 12.

[0025] When performing joint distance calculations using distance cameras 12a, 12a1, and 12a2 as in Examples 2 and 3 of the measurement unit 10 described above, distance information D can be measured directly. Therefore, once the positions and viewing directions of optical cameras 11a, 11a1, 11a2, and 11a3 and distance cameras 12a, 12a1, and 12a2 are determined, feature points, such as joint positions, on image P or P1 and the joint positions on distance cameras 12a, 12a1, and 12a2 can be easily determined by coordinate transformation. However, when performing distance calculations using multiple optical cameras 11a1, 11a2, and 11a3 as in Example 1 of the measurement unit 10 described above, difficulties arise. This is because, in the past, stereo cameras using two optical cameras were often used for distance measurement, and the stereo cameras had significant limitations on measurement. These limitations are as follows: For the sake of convenience, the following description will refer to the two optical cameras that make up the stereo camera as camera A and camera B, but the structure is the same for a single stereo camera.

[0026] In conventional methods using stereo cameras, calculating the disparity between the images of the common part taken by the left and right cameras A and B requires a template matching method to determine whether the common part is the same object. Template matching is a method of identifying the position by performing image matching using an image processing algorithm to determine, for example, which position in the image of a certain area taken by camera A corresponds to in the image taken by camera B. From the positional deviation between the two images, the disparity angle between cameras A and B can be calculated, and the distance can be estimated. As a result, constraints (1) to (3) arise, as shown in Figures 7 to 9.

[0027] (1) When the camera spacing is small (see Figure 7 "Stereo Camera 1: When the cameras are close together") The range of the target area for matching varies greatly depending on the distance between Camera A and Camera B and the object. In other words, when Camera A and Camera B are brought closer together, the observable range expands, but the matching range (the amount of change in parallax angle) differs depending on the distance to the object. Because the distance to the object is not known in advance, attempting to measure a wide distance range requires matching over a wide range, which makes it easy for misrecognition to occur. Furthermore, even if a misrecognition occurs during matching, there is no way to determine whether it is a misrecognition or not, which is a fatal flaw in 3D shape measurement.

[0028] To supplement the example in Figure 7, if cameras A and B are moved closer together, say 10 cm apart, the parallax θ at 500 cm away is 10 / 500 = 0.02 radians ≒ 1°, assuming sinθ ≒ θ. If the viewing angles of cameras A and B are 60° and the number of horizontal pixels is 1000, a difference of 1° corresponds to approximately 20 pixels. On the other hand, the parallax θ at 50 cm away is similarly 10 / 50 = 0.2 radians ≒ 12°, or 200 pixels. A distance of 200 pixels means that the same image must be found 1 / 5 the distance of one image.

[0029] (2) When the camera distance is large (see Figure 8 "Stereo Camera 2: When the cameras are far apart") Although the parallax can be made large, the area for distance calculation becomes narrower. In other words, when camera A and camera B are separated, the matching range (change in parallax angle) becomes narrower, and the observable range also becomes narrower. Furthermore, it becomes impossible to measure objects in front of camera A or camera B.

[0030] (3) When the camera direction is changed (see Figure 9 "Stereo Camera 3: When the camera direction is changed") The shapes of the objects captured by Camera A and Camera B differ. In other words, when the orientation of Camera A and Camera B is changed, the matching range (change in parallax angle) expands to a certain extent, but the shapes of the same objects seen by Camera A and Camera B on the left and right become distorted, making matching difficult.

[0031] The constraints (1) to (3) above indicate the limitations of stereo cameras. In other words, when the distance and position of the target object are limited to a certain extent and the shape of the target object is known, this method is somewhat effective as a simple distance calculation method. However, when the distance to the target object is a wide range, from 50 cm to 500 cm, and the shape of the target object changes from moment to moment, it is extremely difficult to measure an object such as a worker at a work site.

[0032] Figure 10 shows an example that clearly illustrates the drawbacks of stereo cameras. When template matching is performed using a stereo camera on the black and white vertical striped pattern shown on the left side of Figure 10, numerous matching points (matching images) appear (for example, multiple images within the area enclosed by the dashed line in the figure), making measurement impossible. It is known that similar situations occur frequently, and it is clear that the distance calculation method using a stereo camera is only of limited use. As shown on the right side of Figure 10, when a template area is determined using camera A and an image corresponding to the same part is searched for in the image taken by camera B, any number of matching points can be found, so unless information about the distance to the template is known, the solution is indeterminate.

[0033] Therefore, when using a stereo camera, in an image with many straight lines, such as machine tools in a workshop, matching can occur anywhere, and since the distance is indefinite and the amount of parallax cannot be estimated in advance, the range for template matching cannot be determined, so it is predicted that matching errors (no match) and multiple matches will occur frequently.

[0034] Furthermore, Figure 11 shows an example of template matching for a circular object CO and a rectangular object RO using two cameras, camera A and camera B. If the circular object CO and rectangular object RO, which are positioned as shown on the left side of Figure 11, are photographed by camera A and camera B, respectively, the images seen by camera A and camera B will be as shown on the right side of Figure 11, and it can be seen that a matching error occurs (matching is not possible, i.e., distance cannot be calculated). This is because there is no information that the circular object CO and rectangular object RO are at different distances, and it is therefore not possible to mathematically distinguish whether the combination of the circular object CO and rectangular object RO is a single object or not.

[0035] In contrast to the above-described method using a stereo camera, this embodiment proposes a method of calculating distance using feature points on images obtained from optical cameras 11a1, 11a2, and 11a3. This embodiment is characterized in that when calculating distance using multiple optical cameras 11a1, 11a2, and 11a3, template matching is not performed, and distance reconstruction is performed from multiple images obtained by synchronized shooting using three or more optical cameras 11a1, 11a2, and 11a3 instead of a stereo camera.

[0036] The method is as follows. (a) The positions and viewing directions of the optical cameras 11a1, 11a2, and 11a3 are measured in advance. (b) The joint positions of the moving object MO are extracted using machine learning from multiple images P1, P2, and P3 acquired synchronously. (c) The line of sight direction of each optical camera at each joint position is calculated based on the positions and view direction of each optical camera 11a1, 11a2, 11a3. If a joint position is recognized on multiple screens in the vicinity area of ​​the intersection direction, it is assumed that the combination of those multiple screens indicates a specific joint position of a specific individual moving object MO. (d) The position of the joint in question in a generalized three-dimensional coordinate system is determined by applying the algorithm described below to the multiple images sampled as described above.

[0037] The above method makes it possible to effectively extract only the 3D coordinates of the parts necessary for posture evaluation of a moving object MO, and by measuring from multiple directions, it is possible to perform robust measurements that are less likely to be obstructed by obstructions than conventional stereo camera methods.

[0038] As an algorithm for calculating a three-dimensional coordinate position using images from multiple directions, an example of a specific algorithm for estimating the position of an object when the object is photographed by N optical cameras will be described below with reference to FIG. 12.

[0039] (§1 Setting) The setting conditions are as follows: The object is a point P_0 in the three-dimensional space R^3, but its coordinates (x, y, z) are unknown. The index of N optical cameras is k=1, 2, …, N. The position of the kth optical camera is P_k. However, the coordinate value of P_k is assumed to be known. Let V_k be the vector whose starting point is P_k and whose end point is P_0. In other words, V_k=P_0-P_k. ·Let the norm of a vector V be expressed as norm(V), and define a vector U_k with a norm of 1 as V_k / norm(V_k). Since P_0 is unknown, V_k is also unknown, but U_k is assumed to be known from the disparity data obtained as a result of matching. However, since U_k is obtained by optical camera photography, it is important to take into account the possibility of errors being mixed in. Let L_k be the line extending from P_k in the direction of U_k. If U_k had no error, L_1, L_2, ..., L_N would share P_0 as an intersection point, but since U_k has error, L_i and L_j may not necessarily have an intersection point for different indexes i and j.

[0040] (§2 Estimation method) Based on the above settings, the method for estimating the position of P_0 is as follows. ·Let vector P_0 be a vertical vector consisting of (x, y, z) coordinate values. Similarly, P_1, P_2, …, P_N and U_1, U_2, …, U_N are vertical vectors consisting of (x, y, z) coordinate values. ·Let X be any vector or matrix, and let its transpose be X ^* We will express this as follows. ·Let I be a 3x3 identity matrix. For k=1, 2, …, N, the 3-by-3 matrix Q_k is Q_k=I-U_k U_k ^* By definition, Q_k is a symmetric matrix with rank 2 (see below) and is non-negative definite. Moreover, Q_k is a projection matrix. That is, Q_k ^ 2=Q_k holds, where ^ The 2 means squared. Next, let R, a 3x3 matrix, be R=Q_1+Q_2+…+Q_N Generally, the determinant det(R) of R is a real number greater than or equal to zero, and it is possible that det(R) = 0, but in the following, we will assume that det(R) > 0. From the assumption that det(R) > 0, R has an inverse matrix inv(R). For k=1, 2, …, N, the vertical vector S_k is S_k=inv(R) Q_k P_k It is defined as: ·Then the estimate of P_0 is S_1+S_2+…+S_N.

[0041] The rank of Q_k mentioned above is as follows: ·Let G_k be a vertical vector with a norm of 1 that is orthogonal to U_k, and let H_k be a vertical vector with a norm of 1 that is orthogonal to both U_k and G_k. I=U_k U_k ^* +G_k G_k ^* +H_k H_k ^* Therefore, Q_k=G_k G_k ^* +H_k H_k ^* Therefore, the rank of Q_k is 2.

[0042] (§3 Derivation method) The estimated value of P_0 mentioned above was derived in the following way. ·R ^ Any point on 3 is represented by a three-dimensional vertical vector W, and R ^ Let any line in 3 be represented by L. ·Also, define the distance dist(L, W) between L and W as the length of the perpendicular line dropped from W to L. Then, for k=1, 2, …, N, dist(L_k, W) is the norm of the vector obtained by excluding the component parallel to U_k from vector W-P_k, that is, the norm obtained by projecting W-P_k onto a plane perpendicular to U_k, dist(L_k, W)=norm[(I-U_k U_k ^* )(W-P_k)]=sqrt[(W-P_k) ^* Q_k (W-P_k)] is. And we will write the sum of squares of dist(L_1, W), dist(L_2, W), …, dist(L_N, W) as f(W). In other words, f(W)=dist(L_1, W) ^ 2+dist(L_2, W) ^ 2+…+dist(L_N, W) ^ 2 Let's say. The estimated value of the object's position P_0 is defined as W that minimizes the value of f(W). Then, by completing the square (see below), S_1+S_2+…+S_N is an estimate of P_0.

[0043] To complete the square: Transforming the equation for f(W), we get f(W)=Σ_(k=1) ^ N (W-P_k) ^* Q_k (W-P_k) =(W-inv(R) Σ_(k=1) ^ N Q_k P_k) ^* R (W-inv(R) Σ_(k=1) ^ N Q_k P_k)+constant term =(W-Σ_(k=1) ^ N S_k) ^* R (W-Σ_(k=1) ^ N S_k)+constant term Therefore, f(W) is minimized when W=Σ_(k=1) ^ This is when N S_k.

[0044] As described above, an example of the joint distance calculation unit 22 of the evaluation unit 20 is as shown in Fig. 13. As shown in Fig. 13, the joint distance calculation unit 22 receives as input two-dimensional joint position coordinates TwD obtained from the optical cameras 11a, 11a1, 11a2, and 11a3 whose positions have been determined by the joint position determination unit 21, performs coordinate transformation 22a from the optical camera coordinate system to the distance camera coordinate system, and then performs three-dimensional joint position coordinate calculation 22b of the joint position in the distance camera coordinate system based on distance information D obtained from the distance cameras 12a, 12a1, and 12a2.

[0045] The algorithm for calculating the three-dimensional joint position coordinate ThD converts the joint distance data obtained by the distance cameras 12a, 12a1, 12a2, etc. into polar coordinates (zenith angle, azimuth angle, radius) based on the camera pixel count and field of view specifications, and then converts it into Cartesian coordinates (x, y, z) to obtain the three-dimensional joint position coordinate ThD. The bottom of Figure 13 shows examples of the three-dimensional joint position coordinates ThD on the optical camera 11a1 or 11a for the joint sites (here, the right shoulder SR, left shoulder SL, and right elbow ER) as feature points of the human body shown in Figure 6.

[0046] [[3D skeletal reconstruction unit]] Next, an example of the 3D skeleton reconstruction unit 23 of the evaluation unit 20 will be described with reference to Fig. 14. As shown in Fig. 14, the 3D skeleton reconstruction unit 23 performs calculation 23a of the inter-joint distance and angle to adjacent joints based on the joint position three-dimensional coordinates ThD, and then performs verification 23b of their validity.

[0047] Specifically, if the target moving object MO is a human body, the inter-joint distance is calculated, and the calculation result is evaluated for its validity as the inter-joint distance of a human body. If valid, it is adopted as skeletal data (inter-joint distance data LD). If invalid, that portion of the skeleton is rejected as unmeasurable. Similarly, assuming that the line connecting adjacent joints is a bone, the angle between the bone and its neighboring bone is calculated, and evaluated for its validity as the angle at which the bones meet based on the range of motion of the human bones. If valid, it is adopted as skeletal data (angle data AD). If invalid, that portion of the skeleton is rejected as unmeasurable. For example, the forward and backward tilt of the spine (back B) is measured for the left and right hips WL and WR. Normally, in a standing position, a forward tilt of more than 90 degrees is acceptable, but a backward tilt of more than 30 degrees increases the risk of falling. If an abnormal value is obtained, that portion of the skeleton is rejected as unmeasurable.

[0048] The lower part of Figure 14 shows, as inter-joint distance data LD, examples of lengths [cm] of parts (here, shoulder width S, waist width W, back height B (back B), right upper arm ArmR1, left upper arm ArmmL1, right thigh ThiR, left thigh ThiL) as characteristic points of the human body shown in Figure 6, and, as angle data AD, examples of angles [degrees] of opening angles (here, twisting angle θ1 of back B relative to waist width W, lateral bending angle θ2, forward bending angle θf, backward bending angle θb) shown in Figures 22 and 23 described below.

[0049] [[Posture Assessment Section]] Next, an example of the posture evaluation unit 24 of the evaluation unit 20 will be described with reference to FIG. 15. As shown in FIG. 15, the posture evaluation unit 24 selects a posture evaluation algorithm AL1, AL2, ... according to the state of the target moving object MO based on the three-dimensional joint position coordinates ThD, the inter-joint distance data LD, and the angle data AD, and performs posture evaluation 24a. The evaluation result is output as posture evaluation data PED. The lower part of FIG. 15 illustrates an example of the evaluation result when the posture evaluation data PED is divided into the upper body, the lower body, and the hands.

[0050] Various posture evaluation algorithms AL1, AL2, ... can be used depending on the posture evaluation criteria corresponding to the purpose of measuring the moving object MO. For example, if the automatic posture evaluation device 1 is used as a lower back pain monitoring system, a posture evaluation algorithm that specifically recognizes postures such as walking while carrying a large box or squatting to lift it is required. If it is used as a monitoring system, a posture evaluation algorithm that measures daily rhythms such as the number of hours spent sleeping, leaving the room, watching television, and regular meal times is required. If it is used as a system to verify work efficiency in a workplace where a series of assembly line processes are performed, a posture evaluation algorithm that checks the frequency of occurrence of specific postures is required.

[0051] In this way, since the automatic posture evaluation device 1 can accommodate various purposes of measuring a moving object MO, the three-dimensional joint position coordinates ThD, inter-joint distance data LD, and angle data AD used for posture evaluation are not limited to the examples shown in Figures 13 and 14. For example, the shoulders SL / SR, elbows EL / ER, wrists HL / HR, and each joint of the fingers may be added as feature points to generate three-dimensional joint position coordinates ThD, inter-joint distance data LD, and angle data AD specialized for upper limb movements, in which case a posture evaluation algorithm for only the upper limbs is used.

[0052] (Example of posture assessment) 16 and 17, a case where the Ovako Working Posture Analyzing System (OWAS) classification method used to evaluate the postural load of a worker is applied will be described below as an example of posture evaluation by the posture evaluation unit 24. In this embodiment, the automation of workload classification is achieved by a classification algorithm described later.

[0053] (Overview of the OWAS Act) First, we will quote the following excerpt from an overview of the OWAS method (source: "OWAS: Ovako Work Posture Analysis System" http: / / www.nrec.sakura.ne.jp / OWAS.htm. However, some diagram numbers have been revised).

[0054] [2] Various working posture assessment methods and OWAS Development of the OWAS began in the early 1970s by Karhu and Nasman, who worked at a Finnish steel company (Ovako Oy, now Fundia Wire), and Kuorinka and others at the Finnish Institute of Occupational Health. (Omitted) The agreement rate for posture discrimination between examiners was high, at over 90%, and the instrument was tested in over 20 different industries. Since the late 1980s, there have been an increasing number of reports of the use of the OWAS for research and work improvement. [3] Recording of working posture using OWAS As shown in the example in Figure 16, OWAS captures the working posture at a given time in four categories: back, upper limbs, lower limbs, and weight, and records them as a coded four-digit number (posture code). This posture code classification was determined taking into consideration the subjective evaluation of discomfort, the health effects of posture, and practicality.

[0055] In the OWAS method, as shown in FIG. 16, posture codes are organized as follows: "1. Back: Classified into four postures 1) to 4) (for example, 1) is "straight")," "2. Upper limbs: Classified into three postures 1) to 3) (for example, 1) is "both arms below the shoulders")," "3. Lower limbs: Classified into seven postures 1) to 7) (for example, 1) is "sitting")," and "4. Weight or force: Classified into three levels 1) to 3) (for example, 1) is "10 kg or less"). Of these, "4. Weight or force" indicates the weight of the object being worked on or the level of force required by the worker, and is therefore not subject to measurement in this embodiment. "4. Weight or force" is specified by the measurer.

[0056] Next, in the OWAS method, after recording the above postures, the degree of burden on the posture and the degree of need for improvement are classified into the following four levels (AC: Action Category). AC1: The musculoskeletal strain of this posture is not a problem. No improvement is necessary. AC2: This posture is harmful to the musculoskeletal system. It should be improved soon. AC3: This posture is harmful to the musculoskeletal system and should be corrected as soon as possible. AC4: This posture is extremely harmful to the musculoskeletal system and should be corrected immediately.

[0057] For this determination, the AC determination table shown in Figure 17 is used. In the AC determination table, as shown in Figure 17, "1. Back" and "2. Upper Limbs" are combined on the left side, and "3. Lower Limbs" and "4. Weight or Force" are combined on the upper side. There are 12 combinations, which combine each of the categories 1) to 4) of "1. Back" with the categories 1) to 3) of "2. Upper Limbs," and 21 combinations, which combine each of the categories 1) to 7) of "3. Lower Limbs" with the categories 1) to 3) of "4. Weight or Force," for a total of 252 categories. Each category is then assigned a determination of AC1 to AC4.

[0058] (Posture evaluation algorithm of this embodiment) Next, a posture evaluation algorithm according to this embodiment will be described with reference to Figs. 18 to 24. The posture evaluation algorithm applies the classification of the OWAS method to automate posture evaluation using the following algorithm. Fig. 18 defines the names of coordinate positions, lines, and angles, and Fig. 19 shows the target body parts ("1. Back," "2. Upper Limbs," and "3. Lower Limbs"), the classification of posture for each body part, the determination method, and default values ​​as an algorithm. The classification of the three body parts and posture for each body part in Fig. 19 is the same as the classification of the OWAS method shown in Fig. 16, but as mentioned above, "4. Weight or Force" is not included in the algorithm.

[0059] The posture evaluation algorithm is as shown in Figure 19. Among them, for "2. Classification of upper limbs: 1) to 3)" and "3. Classification of lower limbs: 1) to 6)", coordinate positions (for example, X, Y, Z coordinates of the left and right wrists HL, HR), line segments (for example, the right upper arm ArmR1 between the right shoulder SR and the right elbow ER), angles (for example, the angle θ determined by S0Z0-S0-W0 when the intersection of the perpendicular lines from the midpoint S0 of both shoulders to the floor is S0Z0) are used. S0Z0SW ), and the classification is determined by the magnitude relationship between related indices, so Figure 19 will be used for explanation. The other categories, "1. Classification of the back: 1) to 4)" and "3. Classification of the lower limbs: 7)" are explained below.

[0060] First, "1. Classification of back: 1) to 4)" will be explained with reference to Figures 20 to 24. As shown in Figure 20, the joints and line segments used to determine the bending of the back B are shoulder width S, both shoulders SL / SR, midpoint S0 of shoulder width S, back B, waist width W, both waists WL / WR, midpoint W0 of waist width W, and back B (corresponding to the spine) connecting midpoint S0 and midpoint W0.

[0061] The back B bending deals with the following four bendings: (i) Twist (b) Horizontal bending (c) Forward bending (D) Back bending

[0062] In defining the four bends, the x-axis represents the horizontal direction of the angle of view captured by optical cameras 11a, 11a1, 11a2, and 11a3; the y-axis represents the direction of light traveling from the lens centers of distance cameras 12a, 12a1, and 12a2 toward the subject (moving object O); and the z-axis represents the vertical direction of the angle of view captured by optical cameras 11a, 11a1, 11a2, and 11a3. For ease of understanding, the coordinate system is set with the approximate center of gravity GC of the person as the origin (x=0, y=0, z=0), as shown in FIG. 21. The approximate center of gravity GC may be, for example, the average of the coordinates of four points: left shoulder SL, right shoulder SR, left hip WL, and right hip WR, or may be the average of the coordinates of eight points, including left elbow EL, right elbow ER, left knee KL, and right knee KR.

[0063] (i) Twist As shown in FIG. 22, in FIG. 20, a plane perpendicular to the back B (S0-W0) and including the midpoint W0 is defined as plane P, a line segment S' is the projection line of shoulder width S onto plane P, and a line segment W' is the projection line of hip width W onto plane P. The angle formed by line segments S' and W' is defined as θ1. θ1 is the angle formed by WR'-W0-SR' when the projection point of right hip WR onto plane P is WR' and the projection point of right shoulder SR onto plane P is SR'. This θ1 is θ1>θ1 ひねりlimit This definition is intended to take into account general positions such as the lateral position and the prone position.

[0064] (b) Horizontal bending As shown in FIG. 23, in FIG. 20, a plane including the waist width W and the back B is defined as Q. On the plane Q, a line segment B" is a projection of the back B onto a perpendicular line that passes through the midpoint W0 and is perpendicular to the waist width W, and the projection point of the midpoint S0 is defined as S0". The angle formed by S0-W0-S0" is defined as θ2. This θ2 is expressed as follows: θ2>θ2 横曲げlimit When the spine is bent to the left or right, it is considered to be in a state of lateral bending.

[0065] (c) Forward bending, (d) Backward bending As shown in FIG. 24, in FIG. 20, the waist width W is projected from the z-axis direction onto the xy plane. The projected end points are WL' and WR', respectively, and their midpoint is W0'. The back B, which corresponds to the spine, is translated to create a line segment B' (W0'-S0') with W0' as its end point. The projection line of line segment B' onto a plane perpendicular to the line segment passing through the midpoint W0' of line segment WL'-WR' is defined as line segment Bf. If the angle formed by z'-W0'-S0f is an angle in the y-positive direction, it is called the forward bending angle θf. If the angle formed by z'-W0'-S0' is an angle in the y-negative direction, that is, the angle formed by z'-W0'-S0', it is called the backward bending angle θb. FIG. 24 illustrates the case of a forward bending angle θf. This θf or θb is, respectively, θf>θf 前曲げlimit When this happens, it is recognized as a forward bending state, θb>θb 後ろ曲げlimit When θf is defined as ±, the backward bending state may be replaced as follows, as shown in FIG. θf<θf 後ろ曲げlimit

[0066] Based on the above, "1. Back classification: 1) to 4)" is determined as follows, as shown in Figure 19. Classification 2) Judgment: θf > θf 前曲げlimit " or "θb>θb 後ろ曲げlimit (When θf is defined as ±, θf<θf 後ろ曲げlimit )', then back B is determined to be 'Category 2: bending forward or backward'. Classification 4) Judgment: Not classification 2), but "θ2 > θ2 横曲げlimit " and "θ1>θ1 ひねりlimit ", then back B is determined to be "Category 4: twisting and bending sideways or bending diagonally forward." · Judgment of Category 3): If not in Category 4), back B is judged as "Category 3: Twisting or bending sideways." · Judgment of Category 1): If it is not Category 3), back B is judged as "Category 1: Straight".

[0067] Next, regarding "3. Classification of Lower Limbs: 7)," the posture assessment algorithm shown in Figure 19 uses a different algorithm to determine "3. Classification of Lower Limbs: 7) Walking or Moving" in the OWAS method. This is because, in order to detect "walking or moving," it is necessary to measure and evaluate the time-series changes in the 3D skeletal diagram using continuous measurement of the movement of a moving object from a video and distance information D to the joints of the moving object MO synchronized with each frame of the video. In this case, it is necessary to recognize the movement of a single moving object MO between frames over a certain consecutive period of time.

[0068] To achieve this, a moving object MO is considered as a single point in space or a finite three-dimensional spatial region. For example, if a point on the head HD is considered as a representative point of the moving object MO and the movement of that point is sampled at a standard video sampling rate of 30 frames per second, "walking or moving" can be estimated from the movement time of the head HD in three-dimensional space approximately every 30 msec. In this case, the aforementioned approximate center of gravity GC of the human body may be used as the representative point. Furthermore, if the feet are visible, the ankles AnL / AnR may also be used as representative points. A typical human walks at a speed of 2-3 msec. When sampled at 30 frames per second, the distance traveled per frame is approximately 7-10 cm. Therefore, if the moving object moves by a distance of approximately 7-10 cm in a certain direction for a certain distance or more, it can be determined that the moving object is "walking or moving." Alternatively, a moving object MO can be considered as a specific spatial region based on multiple joint position information, and the moving object can be considered as walking in three-dimensional space. The "walking or moving" status can be determined by taking into account the skeletal movement during walking.

[0069] (3D skeleton reconstruction from multiple distance information) The 3D skeleton reconstruction unit 23 has been described above, but we will provide some additional information here. When joint distances of the same moving object MO are obtained by multiple distance cameras 12a1, 12a2 installed at different locations and with different viewing angles, the 3D skeleton of the individual is reconstructed from the multiple distance information D. In this case, for example, it is necessary to use the 3D joint position information of the individual obtained by each distance camera 12a1, 12a2 to recognize that the individual captured by each distance camera 12a1, 12a2 is the same individual. To do this, it is necessary to determine the identity of the positions in 3D space using the 3D coordinate position of the individual's head HD and the approximate center of gravity GC or center position of the moving object MO, as described above.

[0070] If the two are determined to be the same individual, the consistency of the positional relationships of each joint is confirmed. If distance information D is missing due to an obstruction or other reason, and the corresponding joint position is obtained by another distance camera 12a1, 12a2, the value obtained by the other distance camera 12a1, 12a2 is used. However, joints such as the shoulders SL / SR and hips WL / WR depend on the person's orientation relative to the camera, and errors occur depending on the thickness of the back and buttocks of the human body, so corrections are made as necessary. This reduces the amount of missing joint data due to obstructions and makes it possible to reconstruct a 3D skeleton with as high a degree of completeness as possible.

[0071] [Evaluation result display section] Next, examples of the evaluation result display unit 30 will be described with reference to Fig. 25 to Fig. 32. The evaluation result display unit 30 may employ various display methods based on the above-described posture evaluation and depending on the purpose for which the automatic posture evaluation device 1 is used, but they can be broadly classified into the following three types:

[0072] (1) Numerical information The numerical information displayed is three-dimensional joint position information (the numerical values ​​and units differ depending on the coordinate system, such as a distance camera system, a subject coordinate system, or a general coordinate system) for feature points of each part, including the joint parts. The numerical information includes, for example, the right shoulder SR, left shoulder SL, and right elbow ER shown in Fig. 13 as well as the head HD, right wrist HR, left wrist HL, left elbow EL, midpoint S0 of both shoulders (neck), right waist WR, left waist WL, right knee KR, left knee KL, right ankle AnR, and left ankle AnL shown in Fig. 6, and, if applicable, spatial position coordinates along the x-, y-, and z-axes of the right hand tip, left hand tip, right finger joints, left finger joints, right toes, left toes, etc.

[0073] The numerical information may also include the shoulder width S, waist width W, back B (height), right upper arm ArmR1, left upper arm ArmmL1, right thigh ThiR, and left thigh ThiL illustrated in Fig. 14, as well as the lengths (inter-joint distances) of the right forearm ArmR2, left forearm ArmmL2, right lower leg LlimR, and left lower leg LlimL illustrated in Fig. 6. Furthermore, the numerical information may also include the twist angle θ1, lateral bending angle θ2, forward bending angle θf, and backward bending angle θb (or -θf) illustrated in Fig. 14.

[0074] (2) Text information, sound information As described above, the purpose of measurement using the automatic posture evaluation device 1 is diverse, including back pain monitoring systems, watchdog systems, and work efficiency verification systems. For example, in the case of a back pain monitoring system, when a person tries to lift something while leaning forward with their lower body in an upright position, the text information and sound information may be displayed as "Caution: Back Pain" or an alarm may be sounded. In the case of a watchdog system, for example, to observe residents in a care facility, text such as "Sleeping," "Awake," "Moving," or "Lying on the Floor" may be displayed or an alarm may be sounded, provided the text does not violate privacy. In the case of a work efficiency verification system, general evaluation results indicating the posture of the "upper body," "lower body," and "hands" may be displayed, as shown in the upper part of the lower left section of Figure 1.

[0075] Figure 25 shows an example of the four-level AC classification results displayed in text using the aforementioned OWAS method in a work efficiency verification system. Figure 26 also shows an example of a chart displaying the progress of the OWAS AC classification over time. Figure 26 shows the progress of an example case where "4. Weight or force" shown in Figure 16 is 10-20 kg, with the vertical axis representing the AC classification (levels 1-4) and the horizontal axis representing time (in 30-second increments).

[0076] (3) 3D skeleton diagram display 3D skeleton diagram displays include xz plane projection, yz plane projection, xy plane projection, and bird's-eye view from any viewpoint. The following three coordinate systems can be considered for displaying these skeleton diagrams and coordinate positions:

[0077] -Display of 3D joint position information in the distance camera coordinate system FIG. 27 shows an example of three-dimensional position information as viewed from distance camera 12a, where the line of sight of distance camera 12a is the y-axis, the horizontal direction of the field of view of distance camera 12a is the x-axis, and the vertical direction is the z-axis.

[0078] -Display of 3D joint position information in the subject coordinate system When checking the posture of a human body or other object in three dimensions, it is very effective to define the approximate center of the human body and use that point as the origin of a three-dimensional coordinate space to observe a bird's-eye view of the three-dimensional skeleton from any point outside the body. The approximate center of the human body may be defined as the average of the three-dimensional coordinate values ​​of all measured joints in the human skeletal diagram shown in Figure 12. It may also be the average of the coordinates of limited joint locations, such as both shoulders and both hips. Figure 28 shows an example of an object coordinate system when the object is a human body. Figure 29 shows a three-dimensional plan view of the object skeleton in the object coordinate system as viewed from the x-, y-, and z-axes, along with an optical camera image. Similarly, Figure 30 shows an example of a bird's-eye view of the three-dimensional skeleton as viewed from any viewpoint in the object coordinate system, along with an optical camera image.

[0079] -Display of 3D joint position information in a general coordinate system When analyzing the three-dimensional working postures of workers in a large factory, especially when people flow analysis is included, it may be appropriate to display a three-dimensional skeletal diagram of the subject from any viewpoint, with any position in the factory as the coordinate origin. In such cases, this configuration can also be used for display. Figure 31 shows an example of a skeleton projected onto the xz plane in a general coordinate system.

[0080] Here, the means for recognizing the installation position of the range camera 12a, whose field of view overlaps with the image, and the elevation angle θ of the range camera 12a will be described with reference to Figure 32. One method for semi-automatically or fully automatically setting the installation position of the range camera, whose field of view overlaps with the image, is to specify three locations on the image that are not on a straight line and that are known in advance on a factory floor plan, and obtain distance data corresponding to those three points, thereby determining the location within the factory at which the distance measurement device is installed. In addition, the elevation angle θ of the range camera 12a can be determined by specifying the positions of several points on the floor surface F on the image, and from the distance, the elevation angle θ in degrees relative to the floor plane at which the range camera is installed can be determined.

[0081] The calibration method for determining the elevation angle θ of the range camera is as follows. (1) From the coordinate positions and distances of three points on the floor F specified on the screen, three-dimensional coordinates in the camera coordinate system are calculated using the method described above. (2) From the calculated 3D coordinates, calculate the normal vector n of the plane that passes through the three points. (3) The elevation angle θ between the normal vector n on the camera coordinate system and the Z axis of the general coordinate system is calculated.

[0082] The details are as follows. First, let the coordinates and distances of three points p1, p2, and p3 on the floor surface F be p1 = (ξ1,η1,d1), p2 = (ξ2,η2,d2), and p3 = (ξ3,η3,d3). Here, ξ represents the x coordinate on the image, η represents the y coordinate on the image, and d represents the distance. The coordinates and distance of the floor surface F are converted into the three-dimensional coordinates p , 1, p. , 2, p. , Convert to 3.

[0083] When three points on the floor surface F are defined as p1 = (x1, y1, z1), p2 = (x2, y2, z2), and p3 = (x3, y3, z3), point p , 1 and point p , Find the vector v1 = (x2-x1, y2-y1, z2-z1) that passes through point p , 1 and point p , Also find the vector v2 that passes through 3. Point p , 1, point p , 2, point p , The normal vector n of a plane passing through 3 can be calculated as the cross product of v1 and v2. Here, a method for calculating the normal vector n using three points has been described, but this embodiment is not limited to this. Any method for calculating the normal vector n may be used, and it may be calculated from multiple points. Methods for calculating the normal vector n from multiple points include, for example, a method using the least squares method or a method using the RANSAC (RANdom SAmple Consensus) algorithm.

[0084] Furthermore, the normal vector n=(n x ,n y ,n z The elevation angle θ between the Z-axis of the general coordinate system and the target point is calculated as follows:

number

[0085] This completes the calibration method for determining the elevation angle θ of the distance camera 12a. Furthermore, the equation for converting a point (x, y, z) in the camera coordinate system to a point (X, Y, Z) in the general coordinate system using the determined elevation angle θ of the distance camera is as follows:

number

[0086] In the calibration method described above, only the elevation angle θ was calculated assuming that the X and Y axes of the camera coordinate system and the X and Y axes of the world coordinate system are the same. Of course, it is also possible to calculate the azimuth, which is the angle between the X axis of the camera coordinate system and the X axis of the general coordinate system, and perform calibration using the elevation angle θ and azimuth. Also, although this example shows a method of manually specifying the floor surface F on the image, the floor surface F can also be automatically extracted from the image.

[0087] Furthermore, the method shown here is just one example, and any method that can match the camera coordinate system with a general coordinate system will do. One method for automatically recognizing the elevation angle θ of the range camera is to attach a three-axis acceleration sensor, for example.

[0088] Furthermore, calibration may be performed once when the camera is installed, or may be performed for each image capture.

[0089] [Automatic posture evaluation method] Although an embodiment of the automatic posture evaluation device 1 has been described above, the method employed in the automatic posture evaluation device 1 may also be configured as an automatic posture evaluation method. That is, the automatic posture evaluation method for a moving object MO includes a measurement step of measuring the moving object MO, an evaluation step of evaluating the measurement results of the measurement step, and an evaluation result display step of displaying the evaluation results of the evaluation step. The measurement step includes an image acquisition step of photographing the moving object MO, and a distance acquisition step of measuring the distance to the moving object MO, which is synchronized in time with the photographing in the image acquisition step and has a field of view overlapping with the photographed image. The evaluation step includes a feature point position determination step (joint position determination step) of extracting any feature point from the photographed image, a feature point distance calculation step (joint position calculation step) of calculating the spatial distance to the extracted feature point, a 3D skeleton reconstruction step of reconstructing the skeleton of the moving object MO, and a posture evaluation step of evaluating the posture of the moving object MO. The evaluation result display step displays the evaluation result using at least one of numerical information, text information, sound information, and a 3D skeleton diagram.

[0090] [Automatic posture evaluation program] Furthermore, this automatic posture evaluation method may be configured as an automatic posture evaluation program, with its execution contents programmed and executed by a computer. That is, the automatic posture evaluation program for a moving object MO causes a computer to execute a measurement process for measuring the moving object MO, an evaluation process for evaluating the measurement results of the measurement process, and an evaluation result display process for displaying the evaluation results of the evaluation process. The measurement process includes an image acquisition process for photographing the moving object MO, and a distance acquisition process for measuring the distance to the moving object MO, which is synchronized in time with the photographing by the image acquisition process and has a field of view overlapping with the photographed image. The evaluation process includes a feature point position determination process (joint position determination process) for extracting any feature point from the photographed image, a feature point distance calculation process (joint position distance calculation process) for calculating the spatial distance to the extracted feature point, a 3D skeleton reconstruction process for reconstructing the skeleton of the moving object MO, and a posture evaluation process for evaluating the posture of the moving object MO. The evaluation result display process displays the evaluation result using at least one of numerical information, text information, sound information, and a 3D skeleton diagram.

[0091] [Effects of the embodiment] The automatic posture evaluation device 1, the automatic posture evaluation method, and the automatic posture evaluation program according to this embodiment have been described. By configuring the automatic posture evaluation device 1, the automatic posture evaluation method, and the automatic posture evaluation program as described above, the posture load of a moving object can be quantitatively measured at regular intervals without contact, and the load can be evaluated.

[0092] More specifically, the automatic posture evaluation device 1, the automatic posture evaluation method, and the automatic posture evaluation program enable non-contact measurement of moving objects MO (including human bodies such as workers, as well as animals and robots), and also enable measurement to be performed over long periods of time without relying on the individual, thereby reducing the labor required for the person performing the measurement by automating the measurement work. It is also possible to specify the setting of the threshold value for the load to be measured (for example, the threshold value for the angle of forward lean of the back). Furthermore, because continuous measurement is possible, it is also possible to realize functions such as issuing a warning when a posture that is likely to cause back pain is adopted. [Explanation of symbols]

[0093] 1...Automatic posture evaluation device 10...Measuring unit 11...Image acquisition unit 11a, 11a1, 11a2, 11a3...Optical camera 12...Distance acquisition section 12a, 12a1, 12a2...Distance camera 20...Evaluation Department 21... Joint position determination unit (feature point position determination unit) 22...Joint distance calculation unit (feature point distance calculation unit) 23...3D skeletal reconstruction unit 24... Posture evaluation section 30...Evaluation result display section MO...moving objects A, B...Stereo camera CO...Circular object RO…Rectangular object

Claims

1. An automatic posture evaluation device for a moving object, comprising: a measuring unit that measures the moving object; an evaluation unit that evaluates the measurement results of the measurement unit; an evaluation result display unit that displays the evaluation result of the evaluation unit, The measurement unit an image acquisition unit that captures an image of the moving object; a distance acquisition unit that is synchronized in time with the image capture by the image capture unit, has a field of view that overlaps with the captured image, and measures the distance to the moving object; The evaluation unit a feature point position determination unit that extracts any feature point from the captured image; a feature point distance calculation unit that calculates a spatial distance to the extracted feature point; a three-dimensional skeleton reconstruction unit that reconstructs the skeleton of the moving object; a posture evaluation unit that evaluates a load on the posture of the moving object, the posture evaluation unit evaluates a load on the posture of the moving object based on joint positions, inter-joint distance data, line segment and angle data obtained from the three-dimensional skeleton of the moving object reconstructed by the three-dimensional skeleton reconstruction unit; the evaluation result display unit displays the evaluation result of the posture load of the moving object using at least one of numerical information, text information, sound information, and a three-dimensional skeletal diagram. An automatic posture evaluation device characterized by:

2. the image acquisition unit captures a series of moving images of the moving object; the distance acquisition unit is synchronized with each frame image of the captured moving image, has a field of view overlapping with the captured frame images, and measures the distance to the moving object; 2. The automatic posture evaluation device according to claim 1.

3. the image acquisition unit takes a plurality of images of the moving object from different fields of view in a time-synchronized manner; the distance acquisition unit is synchronized with the image capture in time, has a field of view overlapping with the captured images, and measures a plurality of distances to the moving object from different viewpoints; 2. The automatic posture evaluation device according to claim 1.

4. When the distance acquisition unit includes a distance camera that measures distance, the distance acquisition unit semi-automatically or fully automatically recognizes the elevation angle of the distance camera with respect to the space of the object to be photographed.

4. The automatic posture evaluation device according to claim 1, wherein the posture evaluation device is a device for evaluating a posture of a person.

5. the image acquisition unit captures a plurality of images of the moving object from different fields of view, the distance acquisition unit measures the distance to the moving object from the plurality of captured images; 2. The automatic posture evaluation device according to claim 1.

6. A method for automatically assessing the posture of a moving object, comprising: a measuring step of measuring the moving object; an evaluation step of evaluating the measurement result of the measurement step; an evaluation result display step of displaying the evaluation result of the evaluation step, The measuring step includes: an image acquisition step of photographing the moving object; a distance acquisition step of measuring a distance to the moving object, the distance acquisition step being synchronized in time with the image capturing step and having a field of view overlapping with the captured image, The evaluating step includes: a feature point position determination step of extracting any feature point from the captured image; a feature point distance calculation step of calculating a spatial distance to the extracted feature point; a three-dimensional skeleton reconstruction step of reconstructing a skeleton of the moving object; a posture evaluation step of evaluating a load on the posture of the moving object, the posture evaluation step evaluates a load on the posture of the moving object based on joint positions, inter-joint distance data, line segment and angle data obtained from the three-dimensional skeleton of the moving object reconstructed in the three-dimensional skeleton reconstruction step; the evaluation result display step displays the evaluation result of the posture load of the moving object using at least one of numerical information, text information, sound information, and a three-dimensional skeleton diagram. An automatic posture evaluation method characterized by:

7. An automatic posture evaluation program for a moving object, comprising: On the computer, a measurement process for measuring the moving object; an evaluation process for evaluating the measurement results of the measurement process; an evaluation result display process for displaying the evaluation result of the evaluation process; The measurement process includes: an image acquisition process for capturing an image of the moving object; a distance acquisition process that is synchronized in time with the photographing by the image acquisition process and has a field of view that overlaps with the photographed image, and that measures the distance to the moving object; The evaluation process includes: a feature point position determination process for extracting any feature points from the captured image; a feature point distance calculation process for calculating a spatial distance to the extracted feature point; a three-dimensional skeleton reconstruction process for reconstructing the skeleton of the moving object; a posture evaluation process for evaluating a load on the posture of the moving object, the posture evaluation process evaluates a load on the posture of the moving object based on joint positions, inter-joint distance data, line segment and angle data obtained from the three-dimensional skeleton of the moving object reconstructed by the three-dimensional skeleton reconstruction process; the evaluation result display process displays the evaluation result of the posture load of the moving object using at least one of numerical information, text information, sound information, and a three-dimensional skeleton diagram. An automatic posture evaluation program characterized by:

Images