Methods for analyzing a person's posture and / or movement
A method using a pre-trained neural network to create a three-dimensional skeletal model from two-dimensional image data and depth information provides a compact representation for analyzing vehicle occupant posture and movement, addressing the inefficiencies of existing solutions by reducing data and computing needs.
Patent Information
- Application Number
- DE102021200306
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-01-14
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2041-01-14
AI Technical Summary
Existing solutions for analyzing the behavior of vehicle occupants, such as fatigue detection or seat belt adjustment, lack a generic and compact representation suitable for a wide variety of applications, requiring high computing power and generating large data volumes that are difficult to process locally or transmit efficiently.
A method that uses a pre-trained artificial neural network to create a two-dimensional skeletal model from two-dimensional image data, combined with depth information to generate a three-dimensional skeletal model, where keypoints are assigned to body features, and detection values indicate detection status, allowing for a compact representation of posture and movement analysis.
This approach reduces data volume and computing requirements, enabling efficient local processing and transmission, while maintaining high information content for posture and movement analysis, suitable for various applications including driver assistance systems.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The present invention relates to a method for posture and / or movement analysis of a person, in particular of a vehicle occupant, wherein two-dimensional image data and depth information about the person are determined, wherein a two-dimensional skeleton model of the person is created from the two-dimensional image data by means of an artificial neural network.
[0002] Furthermore, the present invention relates to a motor vehicle comprising a computing device designed to carry out a method for posture and / or movement analysis of a person and a computer program product.
[0003] Modern vehicles use driver assistance systems based on analyzing the behavior of their occupants. Attention assistants, for example, detect driver fatigue. Fatigue detection is intended to reduce the risk of accidents caused by drowsy drivers and prevent dangerous microsleep.
[0004] Describing and analyzing the behavior of vehicle occupants has traditionally required a complete recording of user inputs and a complex evaluation process. It is known to analyze the behavior of vehicle occupants by analyzing video sequences. However, analyzing video sequences requires high computing power, as a characterization of the vehicle occupants' behavior must first be generated from the pixel data. Video recordings also generate large amounts of data that either cannot be processed locally in the vehicle due to limited computing power or require a broadband data connection for transmission to the cloud.
[0005] Machine learning methods are used to describe and analyze the behavior of vehicle occupants. These methods classify situations or temporal sequences of user actions by capturing sensor signals. During the training of a so-called classifier, the categories are defined by learning examples. For example, fatigue detection is based on the evaluation of the driver's steering statistics. Using a suitably trained neural network, the driver's fatigue can be determined based on the steering statistics.
[0006] From the publication "Action and object interaction recognition for driver activity classification", Weyers, Patrick et al., 2019 IEEE Intelligent Transportation Systems Conference (ITSC), IEEE, 2019, pp. 4336-4341, a method for posture and / or movement analysis of a person is known in which a two-dimensional skeleton model of a person is created from two-dimensional image data, and a three-dimensional skeleton model of the person is calculated by combining the two-dimensional skeleton model with depth information.
[0007] DE 10 2018 206 777 A1 discloses a depth evaluation for locating a vehicle occupant in a vehicle interior, comprising at least a first input interface for obtaining first image data of a recording of the vehicle occupant and an artificial neural network, wherein the depth evaluation device is designed to execute the artificial neural network when the artificial neural network is loaded into a memory of the depth evaluation device, wherein the artificial neural network is designed to obtain first depth information from the first image data at least as a function of the first image data, and wherein the depth evaluation device is further designed to obtain a first surface model of the vehicle occupant as a function of the first depth information and to calculate the localization of the vehicle occupant in the vehicle interior as a function of the first surface model.
[0008] DE 10 2018 207 977 A1 discloses an interior monitoring system for seat belt adjustment. For this purpose, a driver assistance system for a vehicle is provided, comprising a control unit configured to determine the state of a vehicle occupant using a neural network and to control a seat belt system for positioning or securing the vehicle occupant depending on the detected state of the vehicle occupant.
[0009] Previously known solutions for describing user behavior, especially of vehicle occupants, typically provide a tailored solution for a specific problem, such as fatigue detection or seatbelt adjustment monitoring. However, the state of the art lacks a generic yet compact description of user behavior.
[0010] The present invention is based on the object of providing a method for posture and / or movement analysis of a person, in particular of a vehicle occupant, which provides a compact representation of the data to be analyzed and is suitable for a wide variety of applications.
[0011] To achieve the object underlying the invention, a method for posture and / or movement analysis of a person, in particular of a vehicle occupant, is proposed, wherein two-dimensional image data and depth information about the person are determined, wherein a two-dimensional skeleton model of the person is created from the two-dimensional image data by means of an artificial neural network, wherein a three-dimensional skeleton model of the person is calculated by combining the two-dimensional skeleton model with the depth information, wherein keypoints of the two-dimensional skeleton model and / or the three-dimensional skeleton model are assigned to body features of the person, wherein a detection value is determined for each keypoint assigned to a body feature, wherein the detection value indicates whether the keypoint assigned to the body feature was detected, wherein it is further providedthat a person's movement and / or posture is analyzed by evaluating only the recorded values.
[0012] The two-dimensional image data can be provided by a camera device and contain two-dimensional information about the person's posture and / or movement. The depth information can also be provided by the camera device and / or by a depth sensor configured for this purpose. The depth information relates in particular to the distance of the person, or the distances of the person's body parts or body regions, from the camera device and / or the depth sensor.
[0013] According to the invention, a first step in the process involves using an artificial neural network to create a two-dimensional skeletal model of the person, or rather the person's body, from the two-dimensional image data. In a skeletal model of a person, so-called keypoints, which are assigned to the person's eyes, nose, ears, or joints, for example, are connected to each other in an anatomically correct manner using edges.
[0014] Methods for determining a two-dimensional skeleton model from two-dimensional image data using an artificial neural network are known to the person skilled in the art.
[0015] According to the invention, it is now further provided that depth information about the person is determined parallel to the two-dimensional image data, and that after the creation of the two-dimensional skeleton model of the person, in a second method step a three-dimensional skeleton model of the person is calculated by combining the two-dimensional skeleton model with the depth information.
[0016] In other words, after the calculation of the two-dimensional skeleton model, the keypoints of the two-dimensional skeleton model are supplemented or extended by the depth information, so that for each keypoint a point in three-dimensional space or, in the case of a temporal sequence of two-dimensional image data, a trajectory in three-dimensional space can be determined.
[0017] Since the three-dimensional skeleton model comprises a relatively small number of 3D points, the 3D keypoints, which can be used as a basis for further analysis, a very compact representation of the person's posture and / or movement is provided.
[0018] Preferably, the method is used for posture and / or movement analysis of a person in a driver assistance system in a motor vehicle.
[0019] Preferably, the artificial neural network is an artificial neural network pre-trained with two-dimensional training image data.
[0020] A particular advantage is that pre-trained artificial neural networks can be used to create the two-dimensional skeleton model. To calculate the three-dimensional skeleton model, the two-dimensional skeleton model is combined with the depth information. This eliminates the need for the time-consuming training of a new artificial neural network to determine a three-dimensional skeleton model.
[0021] The pre-trained artificial neural network is designed to create a two-dimensional skeletal model of the person from the two-dimensional image data. Only after the two-dimensional skeletal model has been created using the pre-trained artificial neural network is the depth information obtained in parallel combined with the two-dimensional skeletal model to calculate a three-dimensional skeletal model.
[0022] By using a pre-trained artificial neural network, insights from so-called transfer learning are applied, which minimizes the implementation effort for a corresponding artificial neural network.
[0023] It is preferably provided that the camera device for determining the two-dimensional image data and / or the depth information, and optionally the depth sensor, have a predefined fixed spatial arrangement and orientation in the spatial volume to be monitored, for example in a vehicle interior.
[0024] It is further advantageous that the two-dimensional image data and the depth information form a temporal sequence of image data, in particular a video sequence, and a temporal sequence of depth information.
[0025] In other words, the two-dimensional image data can comprise a plurality of two-dimensional images, in particular in the form of a video sequence, and the depth information forms a temporal sequence of depth information associated with the respective two-dimensional image data. By combining a video recording and temporally correlated depth information, a movement analysis of a person, in particular a vehicle occupant, can be enabled.
[0026] It can be further advantageously provided that the two-dimensional image data are determined by a camera device, in particular by a time-of-flight camera.
[0027] It can further preferably be provided that the depth information is determined by the camera device.
[0028] The camera device can be a known video camera. If the method is used in a motor vehicle, the camera device is preferably directed into the vehicle interior to record persons present therein. It is further preferred that the camera device be a time-of-flight camera. A time-of-flight camera simultaneously provides the depth information used in the method in addition to the two-dimensional image data. A further advantage of using a time-of-flight camera is that the two-dimensional image data and the depth information can be recorded even in complete darkness in the vehicle interior, since time-of-flight cameras typically emit infrared light.Another advantage of using a time-of-flight camera to acquire the two-dimensional image data and depth information is that both the two-dimensional image data and the depth information are acquired from the same recording position.
[0029] It can thus be provided that the depth information is determined by the camera device. Alternatively or additionally, however, it can also be provided that the depth information is determined by combining the two-dimensional image data with additional image data, in particular with additional two-dimensional image data, from another camera device, and / or that the depth information is determined by a depth sensor, in particular by a time-of-flight camera or a radar device.
[0030] If the depth information is determined alternatively or additionally by combining the two-dimensional image data with additional image data, in particular with additional two-dimensional image data, from another camera device, depth information is extracted from the double set of two-dimensional image data according to the stereo camera principle.
[0031] Alternatively or in addition to the depth information determined by the camera device, a depth sensor can be provided. The depth sensor can also be a time-of-flight camera, possibly an additional one. However, the depth sensor can also be a radar device or any other suitable depth sensor.
[0032] It is preferable to ensure that the camera device and / or the depth sensor have fixed positions in the vehicle interior as well as fixed positions relative to each other.
[0033] According to the invention, key points of the two-dimensional and / or three-dimensional skeleton model are assigned to body characteristics of the person.
[0034] The set of body features to be considered is predetermined and encoded in the artificial neural network. When creating a two-dimensional skeleton model from two-dimensional image data, so-called keypoints are assigned to the human body features, such as eyes, nose, ears, and joints, which are then connected anatomically correctly using edges. The two-dimensional or three-dimensional skeleton model thus includes keypoints that are assigned to the person's body features and thus provide an anatomically compact representation of the person's body.
[0035] Furthermore, the invention provides that a detection value is determined for each keypoint assigned to a body feature, wherein the detection value indicates whether the keypoint assigned to the body feature has been detected.
[0036] The keypoints, or the body features associated with the keypoints, are predefined. The artificial neural network attempts to detect the keypoints for all body features to be considered, such as eyes, ears, and joints. It may happen that individual body features are not detected due to a movement or a certain posture of the person relative to the camera device and / or the depth sensor, as they are obscured by the person's body. Accordingly, the detected and non-detected keypoints are each assigned a detection value, which indicates whether a specific keypoint or the associated body feature was detected.
[0037] The detection value can be a binary detection value, for example, 0 or 1, where 0 preferably means that a keypoint or body feature was not detected, and 1 preferably means that a keypoint or body feature was detected. Alternatively, it is also possible for the detection value to be a probability between 0 and 1 with which the respective keypoint was detected.
[0038] One advantage of reducing the three-dimensional skeleton model to the acquisition values is a further reduction in the amount of data to be transmitted. If the three-dimensional trajectories of the keypoints are used to evaluate the person's posture or movement, at least three numerical values corresponding to the three spatial coordinates must be transmitted and evaluated for each time point and each keypoint. However, if only acquisition values are used for the evaluation, the amount of data to be evaluated or transmitted is further reduced.
[0039] Preferably, the detection values of the keypoints are summarized in a vector.
[0040] According to the invention, it is provided that a movement and / or posture of the person is analyzed by evaluating exclusively the detection values, in particular the vector of the detection values.
[0041] In a variant not covered by the invention, the complete 3D information of the keypoints or the trajectories of the keypoints can be evaluated in three-dimensional space. However, according to the invention, the analysis of the detection values, in particular the vectors of the detection values, is provided. In this case, an even more compact representation of the person's posture or movement is used for the evaluation.
[0042] It's particularly worth noting that the detection values themselves contain a high level of information regarding a person's posture or movement. Thus, the detection values alone are sufficient to draw conclusions about a person's posture or movement, especially a vehicle occupant.
[0043] It can be further advantageous to provide that the determination of the movement and / or posture of the person is carried out by means of a second artificial neural network.
[0044] The second artificial neural network is thus preferably designed to determine a movement and / or a posture of the person from the detection values.
[0045] However, instead of a second artificial neural network, other known methods such as object tracking methods or analyses using optical flow methods can also be used.
[0046] If a second artificial neural network is used, it is preferably provided that the second artificial neural network is an artificial neural network trained with three-dimensional test skeleton models, in particular with keypoints and / or edges of three-dimensional test skeleton models, or with test acquisition values.
[0047] In other words, the second artificial neural network can be trained by repeatedly presenting three-dimensional test skeleton models or acquisition values derived from them, whereby the three-dimensional test skeleton models are labeled with labels characterizing postures or movements for this purpose. The second artificial neural network is thus trained as a classifier that has learned behaviors such as normal driving situations with the hands on the steering wheel, distraction due to averted gaze, microsleep due to a hunched back posture, and other movements or poses.
[0048] The second artificial neural network can be a convolutional neural network, which classifies a sequence of keypoints or a sequence of detection vectors over time. Especially when the method is used for motion analysis, the implementation of a recurrent neural network is also suitable, which, by definition, is particularly suitable for evaluating temporal sequences.
[0049] Preferably, it is further provided that the calculation of the three-dimensional skeleton model, in particular the 3D positions of the keypoints and / or the edges, and / or the determination of the movement and / or posture of the person is carried out by means of a local computing device and / or in a cloud.
[0050] Due to the compact feature description provided by the keypoints, the evaluation can be carried out locally, for example in a vehicle, or, with minimal data transmission effort, in a cloud.
[0051] Additionally, the keypoints can also be transmitted via the cloud to other systems that benefit from this data. The three-dimensional skeleton model consists of a relatively small number of 3D points, which allows for rapid classification due to the limited amount of data and also facilitates the outsourcing of computations to the cloud.
[0052] Furthermore, the use of three-dimensional skeleton models and the data derived from them ensures a high degree of data protection, since it is difficult to draw conclusions about a person's identity from the three-dimensional skeleton models and, in particular, from the data derived from them.
[0053] A further solution to the problem underlying the invention consists in a motor vehicle, comprising a computing device and at least one camera device for determining two-dimensional image data, wherein the camera device is additionally designed to determine depth information or wherein a depth sensor is provided for determining depth information, wherein it is further provided that the computing device is designed to carry out a method described above.
[0054] The camera device can be designed as a time-of-flight camera, which provides both the two-dimensional image data and the depth information.
[0055] The motor vehicle can be an internal combustion vehicle, a battery-powered vehicle, an electric vehicle or a hybrid electric vehicle.
[0056] Furthermore, the motor vehicle can be an automatic, automated or autonomous motor vehicle.
[0057] Furthermore, it can be provided that the method implemented by the computing device is used within the framework of a driver assistance system or a system for automatic driving or automated driving or autonomous driving.
[0058] Yet another solution consists in a computer program product comprising instructions which, when the program is executed by a computing device, in particular a computer, cause the device, in particular a computer, to carry out a previously described method.
[0059] The invention is explained in more detail below with reference to the figures.
[0060] They show: Fig. 1 shows a block diagram of a method for analyzing the posture and / or movement of a person, Fig. 2 Two-dimensional image data from a video sequence of a person with a two-dimensional skeleton model superimposed on the person’s body, and Fig. 3 a temporal sequence of vectors from acquisition values.
[0061] Fig. 1 shows a block diagram of a method 100 for posture and / or movement analysis of a person 19 ( Fig. 2), in particular of a vehicle occupant 24.
[0062] In a first method step, two-dimensional image data 11 as well as depth information 12 relating to a person 19, for example a vehicle occupant 24 ( Fig. 2) in a vehicle interior. The two-dimensional image data 11 and the depth information 12 are temporarily stored in a data memory 13.
[0063] From the two-dimensional image data 11, an artificial neural network 14 first creates a two-dimensional skeleton model 15 ( Fig. 2) of the person 19. The two-dimensional skeleton model 15 comprises so-called keypoints 16, which are assigned to body features such as the person's eyes, ears, nose, and joints, as well as edges 17 that anatomically correctly connect the keypoints 16. Subsequently, the keypoints 16 of the two-dimensional skeleton model 15 are combined with the depth information 12 to form a three-dimensional skeleton model 20 using a mapping algorithm 18. The keypoints 21 of the three-dimensional skeleton model 20 describe the spatial coordinates of the body features of the person 19.
[0064] The keypoints 21 of the three-dimensional skeleton model 20 can then be evaluated by means of a further artificial neural network 22 for the posture and / or movement analysis of the person 19.
[0065] Fig. 2 shows a still image 23 from a video sequence of a person 19, in particular a vehicle occupant 24. The still image 23 comprises two-dimensional image data 11. A two-dimensional skeleton model 15 is superimposed on the still image 23. The two-dimensional skeleton model 15 comprises keypoints 16 assigned to the body features, which are connected to one another via edges 17.
[0066] Fig.3 shows a compact representation 25 of keypoints 21 of the three-dimensional skeleton model 20 across a sequence of approximately 160 images captured by the time-of-flight camera 10. Each row of the representation corresponds to a keypoint 21. In the example, the first four keypoints 21 are assigned to the left ear (LEar), the right ear (REar), the left eye (LEye), and the right eye (REye). For each keypoint 21, a detection value 26 with the value 1 indicates whether the body feature corresponding to the keypoint 21 was detected, represented by a dark line. If the body feature was not detected, the detection value 26 receives the value 0, represented by the absence of a dark line. The temporal sequence of the detection values 26 contains a multitude of information that can be used to assess the movement or posture of a person 19.For example, in the area marked by rectangle 27 between images 40 and 60, it can be seen that the body features of the right hemisphere of person 19 were not recorded. From this, the second artificial neural network 22 can conclude that person 19 has turned to the right. In the context of a driver assistance system, this can be interpreted as a distraction of a vehicle occupant 24 from the traffic situation. List of reference symbols 100 procedures 10 Time-of-Flight camera 11 Two-dimensional image data 12 depth information 13 Data storage 14 Artificial neural network 15 Two-dimensional skeleton model 16 Keypoint 17 edge 18 Mapping algorithm 19 people 20 Three-dimensional skeleton model 21 Keypoint 22 Artificial neural network 23 Still image 24 vehicle occupants 25 Compact Representation 26 Recording value 27 Rectangle
Claims
[1] Method (100) for posture and / or movement analysis of a person (19), in particular of a vehicle occupant (24), wherein two-dimensional image data (11) and depth information (12) relating to the person (19) are determined, wherein a two-dimensional skeleton model (15) of the person (19) is created from the two-dimensional image data (11) by means of an artificial neural network (14), wherein a three-dimensional skeleton model (20) of the person (19) is calculated by combining the two-dimensional skeleton model (15) with the depth information (12), wherein keypoints (16, 21) of the two-dimensional skeleton model (15) and / or the three-dimensional skeleton model (20) are assigned to body features of the person (19), wherein for each keypoint (16, 21) assigned to a body feature, a detection value (26) is determined, wherein the detection value (26) indicates whether the Keypoint (16, 21) was recorded, characterized bythat a movement and / or posture of the person (19) is analyzed by evaluating only the detection values (26). [2] Method (100) according to claim 1, characterized by that the artificial neural network (14) is an artificial neural network (14) pre-trained with two-dimensional training image data, and / or that the two-dimensional image data (11) and the depth information (12) form a temporal sequence of image data (11), in particular a video sequence, and of depth information (12). [3] Method (100) according to one of the preceding claims, characterized by that the two-dimensional image data (11) are determined by a camera device, in particular by a time-of-flight camera (10). [4] Method (100) according to claim 3, characterized bythat the depth information (12) is determined by the camera device, and / or that the depth information (12) is determined by combining the two-dimensional image data (11) with additional image data, in particular with additional two-dimensional image data, from a further camera device, and / or that the depth information (12) is determined by a depth sensor, in particular by a time-of-flight camera (10), or a radar device. [5] Method (100) according to one of the preceding claims, characterized by that the determination of the movement and / or posture of the person (19) is carried out by means of a second artificial neural network (22), wherein the second artificial neural network (22) is preferably an artificial neural network (22) trained with three-dimensional test skeleton models, in particular with keypoints and / or edges of three-dimensional test skeleton models, or with test detection values. [6] Method (100) according to one of the preceding claims, characterized by that the calculation of the three-dimensional skeleton model (20), in particular the 3D positions of the keypoints (21) and / or the edges, and / or the determination of the movement and / or posture of the person (19) is carried out by means of a local computing device and / or in a cloud. [7] Motor vehicle, comprising a computing device and at least one camera device for determining two-dimensional image data (11), wherein the camera device is additionally designed to determine depth information (12) or wherein a depth sensor is provided for determining depth information (12), characterized by that the computing device is designed to carry out a method (100) according to one of claims 1 to 6. [8] Computer program product comprising instructions which, when the program is executed by a computing device, in particular a computer, cause the device, in particular a computer, to carry out the method (100) according to one of claims 1 to 6.
Citation Information
Patent Citations
Internal monitoring for seatbelt adjustment
DE102018207977A1
Depth analysis for locating a vehicle occupant in a vehicle interior
DE102018206777A1
Cited By
Method and apparatus for processing data associated with a vehicle
DE102024207288A1
System, passenger car with the system, method for operating the system and computer program
DE102025106658A1
