Method for generating sensor data for the movement of a living being

Abstract

The invention relates to a method for generating sensor data for the movement of a living being, in particular a human being, having the following steps: - providing raw data for the movement of the living being; - generating a model of the living being from the raw data; and - simulating the detection of the movement of the living being by means of a wave-based detection means using the model and / or the raw data and a simulated detection means in a simulated environment, wherein a process of generating sensor data for the movement is carried out.

Description

[0001] Method for generating sensor data of a movement of a living being

[0002] The present invention relates to a method for generating sensor data of a movement of a living being, in particular a human.

[0003] It is known from the state of the art to capture a person's movement with a stereo camera, from which key points are extracted as ground truth. The disadvantage of this approach is the low level of precision achieved, which is insufficient for generating a realistic moving body model.

[0004] It is also common to capture a person's movement using motion capture to create a digital model of a moving person. However, this often only captures the movement of individual body parts, not capturing the person's entire movement or the contribution of these body parts to the overall movement, thus failing to create a complete, realistic, digital model of the entire movement.

[0005] It's also known to create a model of a human consisting solely of ellipsoids based on motion capture data. Because the model consists solely of ellipsoids, it's not realistic and is generally only suitable for radar observations.

[0006] Basically, it is known that a “motion capture” measurement can capture entire or individual movements without any problems.

[0007] In the known methods from the state of the art, the data cannot be sufficiently annotated, augmented and / or edited, especially for further processing.

[0008] Against this background, the present invention is based on the object of providing a method which is improved compared to the prior art and which, in particular for further use by means of artificial intelligence, generates sensor data which can be advantageously annotated, augmented and / or edited.

[0009] This object is achieved by the method having the features of independent claim 1. Advantageous developments of the invention are the subject of the dependent claims.

[0010] Accordingly, the invention provides a method for generating sensor data of a movement of a living being, in particular a human, comprising the following steps:

[0011] - Providing raw data of a living being's movement;

[0012] - Creating a model of the organism from the raw data;

[0013] - Simulating a detection of the movement of the living being by a wave-based detection means using the model and / or the raw data and a simulated detection means in a simulation environment, whereby sensor data of the movement is generated. The sensor data is preferably or includes simulated sensor data and / or real sensor data. Simulated sensor data can preferably also be referred to as virtual sensor data or include virtual sensor data.

[0014] The simulation is preferably performed on a computer. The method is preferably a computer-implemented method.

[0015] Preferably, an artificial variation of the data can be carried out, in particular in order to generate large data sets.

[0016] Preferably, the simulated sensor data is accurately annotated.

[0017] Preferably, mapping of annotation from the simulation to real measurements is done.

[0018] Preferably, simulated but realistic sensor data from a wave-based detection device or sensor, in particular radar, lidar, camera, ultrasound, which detects a moving living being, are generated.

[0019] Preferably, the scattering and / or reflection properties, i.e. the exact geometry and surface properties, such as surface geometry, relief and / or reflectivity and / or one or more Doppler signatures of the moving organism are modeled.

[0020] The invention has the particular advantage that the generated simulated sensor data can be perfectly annotated and the data is therefore perfectly suited for training artificial intelligence.

[0021] The sensor data preferably are or include, in particular, simulated sensor data from a wave-based detection device, in particular for detecting the movement of a living being. The simulated sensor data from wave-based detection devices can also be or include sensory data from wave-based detection devices.

[0022] The sensor data preferably is or includes, in particular, realistic radar data of a movement. The sensor data includes, for example, a simulated signal that corresponds to a real signal. The person skilled in the art thus preferably evaluates the simulated signal as if the simulated sensor had actually seen a moving living being or something similar. The simulated signal preferably exhibits all the characteristics associated with real sensor data, e.g., color in a camera image, distance and, if applicable, angle in radar data, etc.

[0023] Preferably, the sensor data are used at least partially in a real detection means and / or a real system, in particular to improve detection by means of the detection means.

[0024] The model is preferably a digital model and / or a digital twin of the real living being. The model can be static or dynamic, i.e., it can include data about movement.

[0025] In other words, it is preferably provided to simulate realistic sensor measurement data of moving living beings, in particular humans. The sensor data can include, in particular, simulated sensor measurement data and thus be or include simulation data and / or movement data.

[0026] Preferably, the detection means and / or the simulated detection means is or comprises a wave-based sensor, in particular a camera, an optical camera, a depth and / or spectral camera, a radar, a radiometer, a lidar, a laser scanner and / or an ultrasonic sensor.

[0027] It is preferably provided that the simulation is carried out by means of a ray tracing simulation and / or that the simulation comprises a simulated signal processing of simulated signals of the simulated detection means, wherein the signal processing preferably comprises an estimation of a distance, a speed and / or an angle.

[0028] Preferably, the simulation environment is or comprises a static environment, in particular a replica of an environment in which the raw data of the living being have been recorded.

[0029] It is preferably provided that the raw data is provided with a method for motion detection by means of a marker, an acceleration sensor and / or an inertial measuring unit, wherein the model is preferably created by means of key point extraction from data obtained with the method for motion detection and / or that the raw data is provided by means of an existing data set.

[0030] Preferably, the movements of at least one real person are captured using a motion capture method, with tracking preferably being performed using a marker. The motion capture method is preferably a motion capture method.

[0031] Preferably, standard arrangements or special arrangements known to a person skilled in the art can be selected for the marker position. Multiple markers can also be used. A suitable environment or a suitable motion capture setup, such as a laboratory, is preferably used for the motion capture method. The motion capture method can preferably be supplemented with measurement data from acceleration sensors or inertial measuring units designed to measure acceleration and angular rate.

[0032] Alternatively or additionally, existing motion capture data and / or key points derived or extracted from available data sets, e.g. from internationally available databases, can also be used.

[0033] It is preferably provided that the model comprises or is a skeleton model and / or a body shell model, wherein the body shell model preferably comprises or is a mesh, in particular a triangular mesh.

[0034] Preferably, the method comprises the further step:

[0035] - Editing and / or augmenting the model, the raw data and / or the sensor data, wherein preferably a variation and / or change of a movement of the model takes place, in particular in order to enrich the raw data and / or the sensor data with further data obtained thereby.

[0036] The model and / or sensor data can also be annotated.

[0037] Preferably, a variation based on biomechanical models is used to derive virtual creatures from the real measured data in order to generate large data sets with a very high diversity.

[0038] A skeleton and / or body shell model of the moving creature can be created from the motion capture data, for example, using keypoint extraction. Keypoint extraction can preferably include or be a keypoint extraction. The model, which is preferably in the form of a point cloud, can be provided with a digital body shell, which can be implemented, for example, in the form of a triangular mesh frequently used in computer vision.

[0039] Both the skeleton and / or the body shell model and / or the digital body shell can be edited or augmented to slightly vary or change the movement, to synthetically enlarge a limited motion capture dataset or the raw data. "Augmentation" preferably refers to enriching data with additional data. "Editing" preferably refers to changing data.

[0040] Preferably, the model is created with the aid of or using data from a 3D scanner, in particular to measure a body shell and / or proportions of the living being.

[0041] Preferably for verification and / or for a more realistic digitization of the movement, suitable 3D scanners can be used when creating the skeleton and / or body shell models, in particular when fitting the real and / or digital body shell or when editing and / or augmenting, which measure the real body shell and / or proportions of the moving creature.

[0042] Preferably, the method comprises the further step:

[0043] - Fusing the model, the raw data and / or the sensor data with data from a camera, preferably a depth camera, in particular to generate additional optical data.

[0044] Preferably, the body shell model can be fused with the data from one or more cameras, e.g. depth cameras, in order to include additional optical information in the raw data.

[0045] Preferably, the method comprises the further step:

[0046] - Automated and / or manual texturing, in particular of a triangular mesh, of the model, in particular with a material model, preferably clothing, skin, hair, scar tissue and / or a combination thereof, wherein the texturing is preferably carried out taking into account a measurement, in particular by means of a camera.

[0047] Preferably, the model, preferably the triangle mesh, can be textured, in particular to complement material models for clothing, skin, hair, scar tissue, etc. Preferably, material-specific reflection properties, e.g., for radar, lidar, and / or for a spectral camera, can be taken into account. Texturing can be performed manually and / or automatically, in particular taking the camera reference into account and / or with the aid of other suitable measurement technology.

[0048] Preferably, the method comprises the further step:

[0049] - Annotating and / or labeling at least part of the model and / or at least part of the raw data and / or at least part of the sensor data.

[0050] Preferably, an annotation or label can be added to the model or to a movement. The annotation or label preferably includes all information relevant for later use. Preferably, a label can be assigned to an entire movement, e.g., "person running," "person falling," and / or "person waving." Alternatively or additionally, individual sections of the model or sections or areas of the mesh can be labeled. For example, a walking person can be assigned the labels "left arm," "right arm," etc.

[0051] The order of the steps comprising editing and / or augmenting, merging, texturing and annotating and / or labeling can preferably be changed as desired.

[0052] Preferably, the method comprises the further step:

[0053] - Training an artificial intelligence with the raw data and / or the sensor data and / or with real sensor data and / or with only a part of the raw data and / or the sensor data.

[0054] Preferably, the artificial intelligence is trained using sensor data, which can be either raw or processed. The sensor data preferably comes from the simulation and can be well annotated in the simulation. Ideally, only this annotation enables meaningful and effective training of the artificial intelligence.

[0055] By optionally editing the model beforehand, especially the body model, the training data set, preferably the sensor data, can be diversified and expanded. This preferably improves the training of the artificial intelligence, and the artificial intelligence results are preferably more accurate and generally applicable, especially in more or different situations.

[0056] In another embodiment of the invention, real sensor data, rather than the simulated sensor data, is used for training the artificial intelligence. In this case, it can be provided that, in particular, only annotations from the simulated sensor data or the simulated model and / or body model are used, i.e., in particular not the (preferably entire) simulated sensor data, are used to derive annotations in the real measurement signal.

[0057] Preferably, it is provided that aspects that are impossible in reality or contradict the laws of physics are taken into account during training, in particular by eliminating reflections and / or artifacts, freezing a movement and / or breaking a movement down into individual components.

[0058] Preferably, the method is used for fall detection of living beings, for counting living beings, in medical technology for recording movements in diagnostics and therapy, in security technology, in automotive technology, in the context of gesture recognition, in the context of a medical technology solution for the diagnosis and therapy of diseases that affect movement, in the context of an industrial application, in a film production, and / or in a computer game. Preferably, the model or mesh, in particular the "triangle mesh," in particular with the respective optional configurations, can be loaded into a simulator, in particular a ray tracing simulator or a similar tool, as a realistic model and / or as a digital twin of a real living being. The ray tracing simulator can also be referred to as a "ray tracing simulator."

[0059] The simulator can be designed to map and / or simulate the sensor-specific properties of camera, radar, lidar, and / or ultrasound systems. This allows realistic measurement data for the digitized movement of the living being to be simulated, particularly for these sensor systems.

[0060] In this ray tracing simulator, signal processing can be performed in addition to or after the simulation of the sensor-specific measurement data, e.g. for radar: an estimation of distance, speed and / or angle.

[0061] Since moving living beings are usually embedded in a static environment, except for, for example, in free fall, in space or underwater, it is preferably intended to integrate the model of the moving person in the simulator into a static simulation environment or simulation world. The simulation environment can be a realistic representation of the "motion capture setup", in particular a measuring room and / or laboratory. Alternatively or additionally, the simulation environment can include or be a freely selectable realistic, in particular digitized and / or digitally recreated, and / or unrealistic static environment. The simulation environment can, for example, include pedestrians in an urban environment.

[0062] The data sets generated using the method, in particular the sensor data, can preferably be used to train artificial intelligence with realistic simulation data sets, which are preferably part of the sensor data. By automatically generating annotations as well as editing and / or augmenting the model, large data sets that are very well suited for training artificial intelligence can be generated. Since the model is digital, even aspects that are impossible or unphysical in reality can be taken into account in the training. This includes, for example, eliminating disruptive reflections or artifacts, freezing the movement and / or breaking the movement down into individual components. For example, the annotations “left arm” and “right arm” can be used for a pedestrian.

[0063] Since in the ray tracing simulator all further processing steps in the signal processing can be carried out starting from the simulated signals of the simulated detection device or the simulated raw data of the simulated sensor, all possible data of the respective processing steps of the signals of the simulated detection device or the simulated sensor can also be used as input data for training the artificial intelligence.

[0064] Preferably, an automatically annotated data set is generated, e.g. a data set enlarged by editing and / or augmentation, which is in particular a component of the sensor data, for the training of an artificial intelligence, e.g. for object classification, etc.

[0065] Preferably, an "advanced" and / or "enhanced digital twin" is generated, which is a specific component of the model. For example, in the context of prosthetics, missing body parts can be added and / or a simulation can be performed with better, impossible, and / or unphysical simulated detection devices or sensors.

[0066] Preferably, a comparison of different simulated detection means or simulated sensor systems or simulated sensor configurations can be carried out. Using a digitized movement in the model, different sensor configurations, e.g., antenna positions of a radar, different modulation types, e.g., different signal forms of the simulated sensor system, and / or different sensor positions or arrangements, e.g., several individual sensors distributed in space, can be simulated and / or compared during the simulation. Since the movements of living beings are typically not perfectly reproducible, this is not possible with the required precision in real measurements. A digitized movement of a model in a simulation, on the other hand, can be reproduced an unlimited number of times.

[0067] Preferably, the simulation involves freezing and / or slowing down movement sequences, in particular for more precise analysis with different recording devices or sensors that, for example, cannot measure at the required measuring rate.

[0068] Preferably, support can be provided in the development of sensor systems, in particular by means of hardware and / or software-in-the-loop, which can be used to measure movements.

[0069] The labels available or determined during the simulation, such as "left arm," can preferably be used to annotate simulated data. These labels can preferably be used alternatively or additionally to annotate real, non-simulated measurement data. This is done, for example, by "mapping" the sensor data onto real measurement data or overlaying it with it.

[0070] Preferably, the key points can also be used as "ground truth," i.e., predefined output, for training the artificial intelligence. This is particularly advantageous for simulating radar data.

[0071] Preferably, the method can be used in one or more of the following applications:

[0072] - diagnostics and therapeutics, such as gait analysis, Parkinson's therapy, arthritis therapy, palliative medicine and / or prosthetics, etc.);

[0073] - Fall detection of living beings, especially of the elderly;

[0074] - Counting living beings or people in a room;

[0075] - Security technology, such as in a body scanner, motion detection and / or intrusion detection, etc.; - Automotive sensor technology and / or traffic monitoring, e.g. for detecting a vulnerable road user;

[0076] - Gesture recognition, e.g. in infotainment systems, industrial plants and / or autonomous driving;

[0077] - Industrial applications, e.g. in human-machine systems;

[0078] - Film production, e.g. for “special effects”;

[0079] - Computer games, e.g. for realistic avatar movements;

[0080] - Augmented Reality.

[0081] In many of these applications, realistic “digital twins” can also be created and then improved during simulation or in the simulation world to create an “advanced” and / or “enhanced digital twin”.

[0082] The invention also relates to a system, in particular for carrying out a method according to the invention, with means designed to carry out the steps of the method according to the invention, in particular with a simulation unit designed to generate sensor data of the movement.

[0083] It should be noted here that the terms "a" and "an" do not necessarily refer to exactly one of the elements, although this represents a possible embodiment, but can also refer to a plurality of the elements. Likewise, the use of the plural also includes the presence of the element in question in the singular, and conversely, the singular also encompasses several of the elements in question. Furthermore, all features of the invention described herein can be combined with one another as desired or claimed in isolation from one another.

[0084] Further advantages, features, and effects of the present invention will become apparent from the following description of preferred embodiments with reference to the figures, in which identical or similar components are designated by the same reference numerals. Figure 1 shows a flowchart of an embodiment of a method according to the invention.

[0085] Fig. 2: a perspective view of a human in an environment and a perspective view of a model of the human in a simulation environment and a diagram showing measured values ​​of a radar image of the human and a simulated radar image of the model, respectively.

[0086] Fig. 3: a frontal view of a human and a frontal view of a model of the human.

[0087] Fig. 4: a perspective view of a skeleton model and a perspective view of a human body shell model.

[0088] Fig. 5: a frontal view of a human body shell model.

[0089] Fig. 6: a frontal view of a model of a human in a simulation environment and three representations of measured values ​​of a simulated radar image of the model in a diagram.

[0090] Fig. 7: a frontal view of a model of a human in a simulation environment and three representations of measured values ​​of a simulated radar image of the model in a diagram.

[0091] Fig. 1 shows an exemplary sequence of an embodiment of a method according to the invention.

[0092] In step S1, a motion capture measurement is performed, preferably to generate raw data. In this case, a marker arrangement according to a required model is preferably used in additional step E1. In step S2, a model is created, which preferably comprises a skeleton model and / or a body shell model. In step S3, the digital body shell of the body shell model is formed as a mesh, in particular a triangular mesh. In step S4, the model is edited and / or augmented. Steps S2 to S4 can each, or together with additional step E2, include an additional measurement of a real body shell and its proportions using a 3D scanner.

[0093] In step S5, fusion takes place with a camera, in particular a depth camera.

[0094] In step S6, the mesh is textured with a material model.

[0095] In step S7, at least part of the model is annotated and / or labeled.

[0096] In step S8, simulation is performed using a ray tracing simulation and / or the model is integrated into a ray tracing simulator or a ray tracing simulator to generate sensor data. The simulation is preferably performed with additional step E3 in a simulation environment that includes or is a virtual and / or digital twin of a real environment or the real world. The simulation environment can be synthetically generated or originate from an external source.

[0097] Steps S4 to S7, S9 and the additional steps E2 and E3 are optional.

[0098] The order of all steps can be changed as desired.

[0099] In Fig. 2, the top left depicts a real environment U or a "motion capture setup" in which a walking person M is located. The top right diagram shows a measured radar signal from a Doppler radar of the real environment U and the person M, with the distance or "range" from the radar shown on the ordinate and the signal strength of the radar signal shown on the abscissa.

[0100] In the illustration at the bottom left in Fig. 2, a body shell model MM with a digital body shell, which was determined from motion capture data of the human M, is shown in a simulation environment SU.

[0101] The diagram at the bottom right shows a simulated radar signal from a Doppler radar of the body shell model MM, with the distance or range from the radar shown on the ordinate and the signal strength of the radar signal shown on the abscissa.

[0102] The simulated radar signal is very similar to the actual measured radar signal in all crucial respects, which means that the simulation can be classified as “realistic”.

[0103] In Fig. 3, a “motion capture” measurement or a setup with a walking person M to which markers are attached is shown at the top, and extracted positions of the markers, which are part of the raw data from step S1, are shown in a marker model MAM at the bottom.

[0104] Fig. 4 shows a skeleton model SM derived from the markers and a body envelope model KHM derived from them. The preferred approach is to convert the skeleton model SM into a body envelope model KHM.

[0105] Fig. 5 shows an edited body shell model KHM. The body shell model KHM is preferably edited, in particular to model tall, short, fat, and / or thin people. Fig. 6 shows the model MM of a human in a simulation environment SU, with the first diagram from the left depicting the entire simulated radar signal during a simulated radar detection of the model MM.

[0106] The second diagram from the left in Fig. 6 shows the simulated radar signal without arms of the MM model during a simulated radar detection of the MM model.

[0107] The third diagram from the left in Fig. 6 shows the simulated radar signal of only the arms of the MM model during a simulated radar detection of the MM model.

[0108] In Fig. 6, the arms of the MM model are annotated separately. This allows the arms and the body without arms to be simulated separately, enabling a more detailed analysis of the signal components.

[0109] Fig. 7 shows the model MM of a human in a simulation environment SU, with the first diagram from the left showing the entire simulated radar signal during a simulated radar detection of the model MM.

[0110] In the second diagram from the left in Fig. 7, the simulated radar signal of only the multiple reflections of the MM model is shown in a simulated radar detection of the MM model.

[0111] The third diagram from the left in Fig. 7 shows the simulated radar signal without multiple reflections of the MM model during a simulated radar detection of the MM model.

[0112] In Fig. 7, multiple reflections are annotated separately. This allows them to be simulated separately, enabling a more precise analysis of the signal components, as above. This precise separation of annotations, as in Fig. 6 and Fig. 7, is only possible in simulation and cannot be achieved with real measurement data. In real measurements, only the superimposed signal components can be captured as a complete measurement signal.

[0113] In the diagrams in Fig. 6 and 7, the distance or “range” of the simulated radar is shown on the ordinate and the signal strength of the simulated radar signal is shown on the abscissa.

Claims

Patent claims 1 . A method for generating sensor data of a movement of a living being, in particular a human, comprising the following steps: - Providing raw data of a living being's movement; - Creating a model of the organism from the raw data; - Simulating a detection of the movement of the living being by a wave-based detection means using the model and / or the raw data and a simulated detection means in a simulation environment, whereby sensor data of the movement is generated.

2. Method according to claim 1, characterized in that the detection means and / or the simulated detection means is or comprises a wave-based sensor, in particular a camera, an optical camera, a depth and / or spectral camera, a radar, a radiometer, a lidar, a laser scanner and / or an ultrasonic sensor.

3. Method according to one of the preceding claims, characterized in that the simulation environment is or comprises a static environment, in particular a replica of an environment in which the raw data of the living being have been recorded.

4. Method according to one of the preceding claims, characterized in that the provision of the raw data is carried out with a method for motion detection by means of a marker, an acceleration sensor and / or an inertial measuring unit, wherein the creation of the model is preferably carried out by means of a key point extraction from data obtained with the method for motion detection and / or that the provision of the raw data is carried out by means of an existing data set.

5. Method according to one of the preceding claims, characterized in that the model comprises or is a skeleton model and / or a body shell model, wherein the body shell model preferably comprises or is a mesh, in particular a triangular mesh.

6. Method according to one of the preceding claims, characterized in that the method comprises the further step: - Editing and / or augmenting the model, the raw data and / or the sensor data, wherein preferably a variation and / or change of a movement of the model takes place, in particular in order to enrich the raw data and / or the sensor data with further data obtained thereby.

7. Method according to one of the preceding claims, characterized in that the creation of the model is carried out with the aid of data from a 3D scanner, in particular in order to measure a body shell and / or proportions of the living being.

8. Method according to one of the preceding claims, characterized in that the method comprises the further step: - Fusing the model, the raw data and / or the sensor data with data from a camera, preferably a depth camera, in particular to generate additional optical data.

9. Method according to one of the preceding claims, characterized in that the method comprises the further step: - Automated and / or manual texturing, in particular of a triangular mesh, of the model, in particular with a material model, preferably clothing, skin, hair, scar tissue and / or a combination thereof, wherein the texturing is preferably carried out taking into account a measurement, in particular by means of a camera.

10. Method according to one of the preceding claims, characterized in that the method comprises the further step: - Annotating and / or labeling at least part of the model and / or at least part of the raw data and / or at least part of the sensor data.

11. Method according to one of the preceding claims, characterized in that the method comprises the further step: - Training an artificial intelligence with the raw data and / or the sensor data and / or with real sensor data and / or only with a part of the raw data and / or sensor data.

12. Method according to claim 11, characterized in that aspects which are impossible in reality or which contradict the laws of physics are included in the training, in particular by eliminating reflections and / or artifacts, freezing a movement and / or breaking a movement down into individual components.

13. Method according to one of the preceding claims, characterized in that the method is used for fall detection of living beings, for counting living beings, in security technology, in automotive technology, in the context of gesture recognition, in the context of a medical technology solution for the diagnosis and therapy of diseases that affect movement, in the context of an industrial application, in a film production and / or in a computer game.

14. System, in particular for carrying out a method according to one of the preceding claims, with means designed to carry out the steps of the method, in particular with a simulation unit designed to generate sensor data of the movement.

Images