Method and device for estimating the positions of human key points

Abstract

The invention relates to a method and a device for estimating the positions of three-dimensional human key points S1 - SN of a human in a space, comprising the following steps: an image recording unit (8) recording at least one two-dimensional image (1) of the space; a data processing unit (3) receiving the image (1); a first image processing unit (4) estimating the three-dimensional positions of the key points S1 - SN of the human from the two-dimensional image.

Description

[0001] Method and apparatus for estimating the positions of human key points

[0002] The invention relates to a method and a device for estimating the positions of human key points in a three-dimensional space.

[0003] Methods for determining the three-dimensional positions of key points of the human body, particularly the positions of the head, eyes, and various human joints, are known from the prior art. Such methods are used in vehicles to detect the positions and body postures of vehicle occupants.

[0004] For this purpose, image acquisition units in the form of depth cameras or time-of-flight (ToF) cameras are used. Such cameras provide a two-dimensional image of the recording area and use dedicated sensors to measure the distance to the camera for each pixel (hereinafter referred to as pixel depth). The pixel depths are superimposed on the two-dimensional image and displayed, for example, as a grayscale or pixel intensity. If necessary, the pixel depths can also be provided in other forms.

[0005] When measuring pixel depths, however, the problem arises that certain key points on the human body are often obscured, particularly in vehicles. For example, the pixel depths of key points that are obscured by body extremities, seat belts, or clothing cannot be measured correctly by the sensors. The object of the invention is, among other things, to solve this problem and to provide a method and a device for estimating the positions of human key points that provides the positions with greater accuracy. A further object is to supplement existing methods for determining the three-dimensional positions of key points on the human body in such a way that the accuracy of the determined key points can be evaluated using a metric, so that a measure of the accuracy of the key points can be provided for further use of the key points.

[0006] These and other objects are achieved according to the invention by a method according to claim 1.

[0007] A method according to the invention is designed to estimate the three-dimensional positions of key points of a person in a space. It comprises the following steps:

[0008] In a first step, an image acquisition unit captures at least one image of the room. Depth information of the room is also recorded. The image acquisition unit can be a conventional camera that captures a two-dimensional image, with the depth information being determined by a separate sensor. However, the image acquisition unit can also be a ToF camera, which simultaneously captures the image and the depth information. The depth information can also be generated by a stereo camera. The image and the depth information are subsequently received by a data processing unit.

[0009] The data processing unit can be designed as a microcontroller or microcomputer and can comprise a central processing unit (CPU), a volatile semiconductor memory (RAM), a non-volatile semiconductor memory (ROM, SSD hard drive), a magnetic memory (hard drive) and / or an optical memory (CD-ROM), as well as interface units (Ethernet, USB), and the like. The components of such data processing units are generally known to those skilled in the art. In the next step, an estimation of the two-dimensional positions Si is carried out by a first image processing unit. (2D) - SN (2D)The key points of the person from the captured two-dimensional image. For this purpose, well-known pattern recognition algorithms, particularly from the field of machine learning, can be used. In particular, a neural network can be used, which has been trained to estimate the key positions of one or more people from 2D images. The key points can be predefined points on the human body, in particular wrists, arm joints, elbow joints, shoulder joints, knee joints, ankle joints, hip joints, as well as the center of the head, the sternum, the eyes, the ears, the nose, and / or the mouth of a person.

[0010] The result of the estimation is a list of two-dimensional positions Si (2D) of the detected keypoints in the form Si = [xi, ys], where i = 1 ... N and N is the number of detected keypoints.

[0011] In the next step, an extraction unit extracts the measured pixel depths Ti - TN at the detected two-dimensional key points Si (2D) - SN (2D) , and combines the key points Si (2D) - SN (2D) with the pixel depths Ti - TN to form three-dimensional measuring points Mi - MN of the key points.

[0012] The pixel depths Ti - TN denote the respective distances of the key points Si (2D) - SN (2D)from a camera plane, so that the measurement points Mi - MN contain not only the estimated positions of the key points in two dimensions but also the actually measured depth of these key points, i.e., the actual third dimension. To obtain the pixel depths, it can be provided that depth information is measured when the image is captured by the image acquisition unit and stored in the image. The depth information can be determined, for example, by echo-based sensors that are provided in addition to the image acquisition unit. These sensors provide depth information for each pixel of the image in a 2D grid. Since each pixel in the image is assigned a distance value, for each two-dimensional key point Si (2D) - SN (2D)a depth information can be determined. However, it can also be provided that the depth information is determined in the form of a second image by a second image recording unit, wherein for the extraction of the pixel depths Ti - TN at the key points Si (2D) - SN (2D) This second image is used. In this case, it must be ensured that the second image acquisition unit captures essentially the same situation as the first image acquisition unit, so that the depth information of the images matches. For example, it can be provided that the first image acquisition unit and the second image acquisition unit are activated simultaneously and have the same image section.

[0013] The depth information can be represented using different intensities, so that the depth information can also be treated as 2D image information. To extract the pixel depths Ti - TN at the key points Si(2D) - SN (2D) The depth information stored in the corresponding pixels in the image can thus be used directly.

[0014] The accuracy of the three-dimensional measurement points determined in this way depends on the accuracy of the captured image and the image processing unit used. However, the results for those key points that are obscured in the image by obstacles such as clothing or the seat belt will inevitably be inaccurate. These key points will inevitably have measurement points with inaccurate or incorrect pixel depths.

[0015] To assess and improve the accuracy of the measurement points, the following steps are carried out:

[0016] First, a second image processing unit estimates the three-dimensional positions Si - SN of the key points from the captured image. Depth information is not taken into account, but only the two-dimensional image. Image processing algorithms specifically designed for this application are known. Methods from the field of machine learning are often used for this purpose, for example, neural networks trained to recognize the three-dimensional positions of a person's relevant key points in two-dimensional images. Typically, the training data for the neural networks of the first image processing unit and the second image processing unit differ. However, any other method from the field of machine learning and artificial intelligence can be used instead of a neural network.

[0017] In a further step, a comparator unit compares the three-dimensional measurement points Mi - MN with the positions Si - SN of the estimated key points, and calculates a quality metric Qi - QN based on the comparison. The quality metric can be calculated, for example, in the form Qi = | |Mi - Si| |, i.e., by calculating the Euclidean distance of the respective x-, y-, and z-coordinates.

[0018] Both the image processing units, the extraction unit, and the comparator unit can be provided as separate hardware units, or preferably as software modules in the RAM or ROM of the data processing unit. However, these units can also be provided externally, for example, on a server on the Internet, to which the required data, in particular the captured image and the captured depth information, are transmitted.

[0019] According to the invention, those measurement points Mi - MN for which the calculated quality metrics Qi - QN exceed a predetermined threshold value can be corrected using individual correction vectors Ki - KN, so that corrected measurement points Mi' are obtained. The threshold value can be, for example, 5 cm, 10 cm, or 15 cm.

[0020] According to the invention, it can be provided that the measurement points Mi - MN, the positions Si - SN of the estimated key points, and optionally also the corrected measurement points MT - MN' are each converted into a topological three-dimensional data model, in particular into a graph. The known three-dimensional physiology of the human body can be used to create the topological data model. For example, the detected position of an elbow joint can be connected as a key point in the form of a node in a graph to the positions of the wrist and shoulder joint via edges.

[0021] According to the invention, the correction vectors Ki for those measurement points Mi for which the quality metric Qi exceeds a predetermined threshold are calculated based on the three-dimensional estimated value Sj of at least one neighboring key point and taking into account the topology of the data models. For example, a neighboring key point can be determined by searching for the nearest neighboring node of the faulty measurement point Mi for which the quality metric Qj does not exceed the threshold.

[0022] According to the invention, it can be provided that the correction vectors Ki are calculated by adding to the measuring point Mi the difference between the estimated position Si and the estimated position Sj of the neighboring key point, so that Mi' = Mj + Si - Sj. An erroneous measuring point is thus corrected by that correction vector which would lead from the neighboring key point to this key point in the three-dimensional estimation.

[0023] According to the invention, it can be provided that the estimated three-dimensional positions Si - SN, the corrected measuring points Mi' - MN' and / or the measuring points Mi - MN are provided by the data processing unit in electronically readable form, for example in the form of tables or graphs.

[0024] The invention further relates to a computer-readable storage medium comprising instructions that cause a data processing unit to execute a method according to the invention.

[0025] The invention further relates to a device for estimating the positions of human key points in a three-dimensional space from at least one image and measured depth information, comprising at least one image acquisition unit for capturing the two-dimensional image and a device for measuring depth information, as well as a data processing unit designed to carry out a method according to the invention. It can be provided that the first image processing unit and / or the second image processing unit is part of the data processing unit and is preferably designed as a K1-based model, for example, as a neural network.

[0026] The invention further relates to a vehicle comprising a device according to the invention.

[0027] Further features of the invention emerge from the claims, the embodiments and the figures.

[0028] The invention is explained below using an exemplary, non-exclusive embodiment.

[0029] Fig. 1 shows a schematic example of a device according to the invention; Figs. 2a - 2c show schematic examples of the data models generated when carrying out a method according to the invention.

[0030] Fig. 1 shows a schematic example of an inventive device for estimating the positions of human key points Si - SN in three-dimensional space. The device comprises an image acquisition unit 8 configured to acquire a two-dimensional image 1 and to measure depth information 2. Furthermore, the device comprises a data processing unit 3 with a first image processing unit 4, a second image processing unit 5, an extraction unit 6, and a comparator unit 7.

[0031] The first image processing unit 4 comprises a neural network trained to extract human keypoints in two dimensions from a two-dimensional image. Image data with two-dimensionally annotated keypoints were used to train this network.

[0032] The second image processing unit 5 comprises a neural network trained to extract human keypoints in three dimensions from a two-dimensional image without using depth information. Image data with precisely three-dimensionally annotated keypoints was used to train this network.

[0033] First, the first image processing unit 4 is applied to the image 1 so that two-dimensional positions Si (2D) - SN (2D) of key points of the human in Figure 1. The key points are in the form of Cartesian coordinates ui, Vi.

[0034] Since there is a corresponding depth information or a corresponding 3D position for each pixel in image 1, for each of the key points estimated in 2D, Si (2D) A measured depth, i.e. a z-coordinate, can also be determined, which is a measurement and not an estimate. This is done in the extraction unit 6, which calculates the measured pixel depths Ti - TN at the positions Si (2D) - SN (2D) The key points are extracted from image 1 and three-dimensional measurement points Mi - MN are generated. Using known camera parameters, a position consisting of ui, Ui, and Zi can be determined from the estimated two-dimensional coordinates ui, Vi, and the depth value Zi. The positions of the measurement points Mi thus each contain the estimated two-dimensional coordinates Xi, yi, and the measured depth Zi of the key points.

[0035] The second image processing unit 5 is then applied to image 1, but without considering the depth information. This second image processing unit provides estimated three-dimensional positions Si - SN of the person's key points from image 1 in the form of x, y, and z coordinates. These are usually not identical to the previously generated measurement points Mi - MN.

[0036] Subsequently, in a comparator unit 7, the measurement points Mi - MN are compared with the estimates Si - SN, and a quality metric Qi - QN is calculated based on the comparison. It is assumed that, in the event of occlusion, the estimated position Si of a key point deviates significantly from the measured position Mi of this key point. If the distance between the two values ​​Si and Mi is below a predetermined threshold, for example, below 10 cm or below 5 cm, the measurement Mi is marked as correct because it can be assumed that this key point is not occluded.

[0037] If the distance between the two values ​​exceeds this threshold, a corrected version of the measurement Mi is used. The measurement is corrected using a custom correction vector Ki by repositioning it relative to the nearest unoccluded keypoint.

[0038] To perform this correction, the measured positions Mi and the estimated positions Si of the key points are converted into a topological data model, namely the graphs 9, 9'. Schematic representations of these graphs 9, 9' are shown in Figures 2a and 2b, with the hidden surfaces indicated by bold polygons. The data models of the graphs 9, 9' are predefined based on a human skeleton model, with the predefined key points defined as nodes and connected by edges. For example, the right wrist is connected to the right elbow, the right elbow to the right shoulder, and the right shoulder, in turn, is connected to the right hip and the left shoulder via edges.

[0039] In the concrete example, the measured positions of the key points M?, Ms, M and M15 are obscured, which can be recognized by the significant differences between the measured values ​​M?, Ms, M and Mn and the three-dimensional estimated positions S7, Ss, Sn and Sn.

[0040] To correct a three-dimensional measured value Mi that has been identified as incorrect, the nearest correctly measured point Mj is identified along the edges of the graph, e.g. the wrist in the case of an incorrectly identified elbow joint. In the example in Figure 2a, this would be the correctly measured point M2 (shoulder joint) for point M7 (elbow joint), so that i=7 and j=2. The relative positioning of the estimated values ​​Si and Sj is then calculated and stored as a correction vector dij = Si - Sj. In the specific example from Fig. 2b, the correction vector essentially corresponds to the estimated course of the upper arm. To now obtain a corrected measuring point M7', this correction vector is added to the measured value of M7, so that M7' = M7 + d72.

[0041] If there are multiple nearest keypoints, a correction value is calculated for each and the average is formed. A schematic representation of the resulting corrected graph 9" is shown in Figure 2c, where it can be seen that the measured positions of the hidden keypoints have been corrected.

[0042] However, the invention is not limited to this described embodiment, but also includes further embodiments of the present invention within the scope of the following patent claims.

[0043] List of reference symbols

[0044] 1 , 1 ' Two-dimensional image

[0045] 2 Depth information

[0046] 3 Data processing unit

[0047] 4 First image processing unit

[0048] 5 Second image processing unit

[0049] 6 Extraction unit

[0050] 7 Comparator unit

[0051] 8, 8' image acquisition unit

[0052] 9, 9' Graph

Claims

Patent claims 1 . Method for estimating the three-dimensional positions of key points of a person in a room, comprising the steps: a. Recording, by at least one image recording unit (8), at least one image (1) and depth information (2) of the room, b. Receiving, by a data processing unit (3), the image (1) and the depth information (2); c. Estimating, by a first image processing unit (4), the two-dimensional positions Si (2D) - SN (2D) the key points of the person from the image (1 ); d. extraction, by an extraction unit (6), of pixel depths Ti - TN at positions Si (2D) - SN (2D) , and combination of the estimated positions Si (2D) - SN (2D)with the pixel depths Ti - TN for forming three-dimensional measurement points Mi - MN of the key points, characterized in that the following steps are carried out to estimate and improve the accuracy of the measurement points Mi - MN: e. estimation, by a second image processing unit (5), of the three-dimensional positions Si - SN of the key points from the image (1) without using the depth information, f. comparison, by a comparator unit (7), of the three-dimensional measurement points Mi - MN with the estimated three-dimensional positions Si - SN, and calculation of a quality metric Qi - QN on the basis of the comparison carried out, for example in the form Qi = | |Mi - Ss| |.

2. Method according to claim 1, characterized in that when the image (1) is recorded, depth information (2) is measured by sensors and stored in the image (1), and for extracting the pixel depths Ti - TN at the key points Si (2D) - SN (2D)the depth information (2) stored in the image (1) is used.

3. Method according to claim 1, characterized in that a second image (T) of the room is recorded by a second image recording unit (8'), which second image has measured depth information (2), wherein for the extraction of the pixel depths Ti - TN at the key points Si (2D) - SN (2D) the second image (T) is used.

4. Method according to one of claims 1 to 3, characterized in that those measuring points Mi - MN for which the calculated quality metrics Qi - QN exceed a predetermined threshold value are corrected by individual correction vectors Ki, so that corrected measuring points Mi' are obtained.

5. Method according to one of claims 1 to 4, characterized in that the measuring points Mi - MN, the positions Si - SN of the estimated key points, and optionally also the positions Mi' of the corrected measuring points are converted into a topological three-dimensional data model, in particular into a graph (9, 9', 9").

6. Method according to claim 5, characterized in that the correction vectors Ki for those measuring points Mi for which the quality metric Qi exceeds a predetermined threshold are calculated on the basis of the estimated value of the position Sj of at least one neighboring key point and taking into account the topology of the data models (9, 9').

7. Method according to claim 6, characterized in that the correction vectors Ki are calculated by adding to the measured position Mi the difference between the estimated position Si and the estimated position Sj of the adjacent key point, so that Mi' = Mj + Si - Sj.

8. Method according to one of claims 1 to 7, characterized in that the measuring points Mi - MN, the estimated positions Si - SN, and / or the corrected measuring points MT - MN' are provided by the data processing unit (3) in electronically readable form, for example in the form of tables or graphs.

9. Method according to one of claims 1 to 8, characterized in that the key points are predefined points of the human body, in particular wrists, arm joints, elbow joints, shoulder joints, knee joints, ankle joints, hip joints, as well as the center of the head, sternum, eyes, ears, nose, and / or mouth of a person.

10. A computer-readable storage medium comprising instructions that cause a data processing unit (1) to execute a method according to one of claims 1 to 9.

11. A device for estimating the positions of human key points in a three-dimensional space from at least one image (1, T) and measured depth information (2), comprising a. at least one image recording unit (8) for recording the image (1, T) and a device for measuring depth information (2), and b. a data processing unit (3) designed to carry out a method according to one of claims 1 to 9.

12. Device according to claim 11, characterized in that the first image processing unit (4) and / or the second image processing unit (5) is part of the data processing unit (3) and is preferably designed as a Kl-based model, for example as a neural network.

13. A vehicle comprising a device according to any one of claims 11 or 12.

Images