Training method, method and system for generating synthetic measurement data

Abstract

The present disclosure relates to a method for training a system for generating synthetic measurement data. The disclosure also relates to a system for generating the synthetic measurement data, as well as to the generation of the synthetic measurement data.

Description

[0001] The present disclosure relates to a method for training a system for generating synthetic measurement data.

[0002] The disclosure also relates to a system for generating the synthetic measurement data, as well as to a method for generating the synthetic measurement data. The disclosure also relates to a computer program implementing one of the aforementioned methods, a machine-readable data carrier and / or a downloadable product comprising such a computer program, and one or more computers comprising the aforementioned computer program. State of the art

[0003] Advanced driver assistance systems (ADAS) and autonomous driving (AD) require an accurate representation of the vehicle's surroundings. Machine learning (ML) methods are often used for this purpose. State-of-the-art technology involves using sensors to detect the surroundings and generating synthetic measurement data based on the sensor data.

[0004] The present disclosure provides a method that can be applied to multimodal measurement data. Disclosure of the invention

[0005] One embodiment relates to a method for training a system for generating synthetic measurement data, the method comprising: Providing first measurement data of at least one first measurement modality and second measurement data of a second measurement modality, wherein the first and second measurement data comprise data of an observation space acquired by the first and second measurement modalities, and wherein the first and second measurement modalities acquire the observation space from mutually different positions, and wherein the measurement modalities are sensors of a different measurement principle; Predicting first sensor-specific features based on the first measurement data using a first sensor-specific neural network; Predicting second sensor-specific features based on the second measurement data using a second sensor-specific neural network; Predicting sensor-agnostic features based on the first and second measurement data using a generic neural network; Generating first synthetic measurement data based on the first sensor-specific features and the sensor-agnostic features by means of a first sensor model and generating second synthetic measurement data based on the second sensor-specific features and the sensor-agnostic features by means of a second sensor model.

[0006] Synthetic measurement data refers to generated, realistic measurement data from a physical or virtual measurement modality.

[0007] According to the disclosure, at least two measurement modalities, which are sensors of a different measurement principle, are used. Advantageously, more than two, in particular between three and ten, or more measurement modalities, or the measurement data of the measurement modalities, are used. More than two measurement modalities can be sensors of the same and different measurement principles as the first and second measurement modalities.

[0008] The first and second measurement modalities capture the observation space from different positions. Accordingly, the first and second measurement modalities capture a respective view of a scene that can be captured in the observation space at a given time. According to the method, the system is trained on a scene that can be captured in the observation space at a given time, which is captured as different views of the scene in the form of the acquired measurement data at the respective known positions of the sensors.

[0009] According to the disclosure, it is provided that the measurement data of a respective measurement modality are used to predict respective sensor-specific features with a sensor-specific neural network and to predict sensor-agnostic features with a generic neural network.

[0010] The synthetic measurement data is then generated using a respective sensor model based on the respective sensor-specific features and the sensor-agnostic features. Sensor-agnostic features can be fed into a respective sensor model as additional input. This enables the exchange of information between the different sensors and allows for the learning of a more comprehensive representation of the observation space. A respective sensor model, for example, uses a volumetric rendering equation.

[0011] According to one embodiment, the method comprises: comparing the first synthetic measurement data with the first measurement data and / or comparing the second synthetic measurement data with the second measurement data and assessing the comparison with a loss function, optimizing parameters of at least the first and / or the second sensor model in order to optimize a result of the assessment by means of the loss function

[0012] According to one embodiment, the method comprises: discretizing the first measurement data and the second measurement data, wherein the measurement data each comprise acquired data of the observation space in the form of rays of a respective measurement modality, into data that describe a number of spatial observation points along the rays, wherein a respective observation point is described by its respective spatial position and a respective viewing direction of the respective measurement modality. The discretization is carried out, for example, using sensor-specific sampler modules. The method is based on the method of neural radiation fields for reconstructing a three-dimensional representation of a scene from two-dimensional images.

[0013] According to one embodiment, the discretization is performed depending on respective sensor-specific parameters of the respective first and / or second measurement modality. The respective sensor-specific parameters include, for example, extrinsic parameters of the respective measurement modality, such as the position of the measurement modality relative to the observation space, and intrinsic parameters of the respective measurement modality, such as a focal length (camera) or a number of beams (lidar) depending on the measurement principle.

[0014] According to one embodiment, the data describing a respective spatial observation point are encoded and provided as encoded data describing a respective spatial observation point as input for a respective sensor-specific neural network and as input for the generic neural network. The data describing a respective observation point by its respective spatial position and a respective viewing direction of the respective measurement modality are encoded, for example, by means of a respective coder, in particular an encoder. One possible method of sample encoding uses hashing techniques to map Euclidean coordinates to learned latent values, see T. Müller, A. Evans, C. Schied, and A. Keller, "Instant neural graphics primitives with a multiresolution hash encoding," ACM Trans. Graph., vol. 41, pp. 102:1-102:15, July 2022.

[0015] According to one embodiment, it is provided that at least one sensor-specific feature and / or at least one sensor-agnostic feature is predicted for a respective spatial observation point.

[0016] According to one embodiment, it is provided that a sensor-agnostic feature is at least one of the following features: radiation density, geometry, and / or wherein a sensor-specific feature is at least one of the following features: color value, intensity, reflection property, material property.

[0017] According to one embodiment, the method comprises: detecting the observation space by means of at least a first measuring modality and a second measuring modality, wherein the first and the second measuring modality detect the observation space from positions different from one another.

[0018] According to one embodiment, it is provided that a respective measuring modality is a sensor according to one of the following measuring principles: camera, lidar, radar, ultrasonic sensor.

[0019] Further embodiments relate to a system for generating synthetic measurement data, wherein the system is or is trained according to a method according to the embodiments described above.

[0020] The trained system can ultimately be used to generate synthetic measurement data, where the synthetic measurement data represents measurement data from a sensor at a further, new position relative to the observation space. Thus, synthetic measurement data is generated in the sense of further, new views of the scene that can be captured in the observation space.

[0021] Further embodiments relate to methods for generating synthetic measurement data of a measurement modality using a system for generating synthetic measurement data according to the embodiment described above.

[0022] Further embodiments relate to the use of a system according to the embodiment described above and / or a method for generating synthetic measurement data of a measurement modality according to the embodiment described above and / or the synthetic measurement data for training a technical system that is or is installed in a vehicle for assistance-based and / or autonomous driving or in a robot, or which is part of a traffic monitoring system, wherein the synthetic measurement data each relate to an environmental detection of a vehicle environment and / or a robot environment and / or a traffic situation by a system with corresponding sensors. Thus, synthetic measurement data is generated as measurement data of a sensor at a further, new position in relation to a vehicle environment and / or a robot environment and / or a traffic situation.These synthetic measurement data are used, for example, to train the technical system that is or will be installed in a vehicle for assistance-based and / or autonomous driving or in a robot, or which is part of a traffic monitoring system.

[0023] The trained technical system can be used in a vehicle for assistance-based and / or autonomous driving or in a robot or as part of a traffic monitoring system to control the vehicle for assistance-based and / or autonomous driving or the robot or the traffic monitoring system.

[0024] Further embodiments relate to a computer program implementing one of the aforementioned methods, a machine-readable data carrier and / or a download product comprising such a computer program, and one or more computers comprising the aforementioned computer program.

[0025] Further advantages will become apparent from the description and the accompanying drawings. Exemplary embodiments of the invention are illustrated in the drawings and explained in more detail in the following description. Identical reference numerals in different figures designate identical or at least functionally comparable elements. When describing individual figures, reference may also be made to elements from other figures. These show, in schematic form: Fig. 1 a schematic representation of steps of a method for training a system for generating synthetic measurement data; Fig. 2 a schematic representation of a data flow for training a system for generating synthetic measurement data, and Fig. 3 a schematic representation of a recording of an observation space using two measurement modalities.

[0026] For the sake of simplicity, the embodiments shown in the figures refer to a system with two measurement modalities S1 and S2. The system can be expanded to any number i of sensors Si.

[0027] Fig. 1 shows a method 100 for training a system 200 for generating synthetic measurement data.

[0028] The system 200 comprises, for example, a first sensor-specific neural network MLP_s1 and a second sensor-specific neural network MLP_s1, as well as a first sensor model SM_s1 and a second sensor model SM_s2 cf. Fig. 2. Other components of the system include sampler modules and encoder modules.

[0029] The method 100 includes, for example, the following steps: a step 110 for providing first measurement data m_s1 of at least one first measurement modality S1 and second measurement data m_s2 of a second measurement modality S2, wherein the first and second measurement data m_s1, m_s2 comprise data of an observation space acquired by the first and second measurement modalities S1, S2, and wherein the first and second measurement modalities acquire the observation space from mutually different positions, and wherein the measurement modalities are sensors of a different measurement principle; a step 120_1 for predicting first sensor-specific features f_s1 based on the first measurement data m_s1 by means of a first sensor-specific neural network MLP_s1; a step 120_2 for predicting second sensor-specific features f_s2 based on the second measurement data m_s2 by means of a second sensor-specific neural network MLP_s2; a step 120_gen for predicting sensor-agnostic features f_gen based on the first and second measurement data m_s1, m_s2 by means of a generic neural network MLP_gen; and a step 130 for generating first synthetic measurement data m'_s1 based on the first sensor-specific features f_s1 and the sensor-agnostic features f_gen by means of a first sensor model SM_s1 and generating second synthetic measurement data m'_s2 based on the second sensor-specific features f_s2 and the sensor-agnostic features f_gen by means of a second sensor model SM_s1.

[0030] The procedure is described below with reference to Fig. 2 and Fig. 3 described.

[0031] First, a data set comprising sensor-specific parameters p_si; p_s1, p_s2 of the respective first and second measurement modalities and measurement data m_si; m_s1, m_s2 of the respective first and second measurement modalities is provided for a respective sensor S1, S2.

[0032] The respective sensor-specific parameters p_si; p_s1, p_s2 include, for example, extrinsic parameters of the respective measurement modality S1, S2, for example the position of the measurement modality S1, S2 with respect to the observation space B, and intrinsic parameters of the respective measurement modality S1, S2, for example, depending on the measurement principle, a focal length (camera), a number of beams (lidar).

[0033] The respective measurement data m_si; m_s1, m_s2 each comprise acquired data of the observation space B in the form of rays S_s1, S_s2 of a respective measurement modality S1, S2, which describe a view of a scene S of the observation space B acquired with a respective measurement modality Si, S1, S2 from a respective known position of the respective measurement modality Si, S1, S2.

[0034] Depending on the respective sensor-specific parameters p_si; p_s1, p_s2, sensor-specific sampler modules 210_s1, 210_s2 discretize the measurement data m_si; m_s1, m_s2 into data that describe a number of spatial observation points P_s1, P_s2 along the rays S_s1, S_2, wherein a respective observation point P_s1, P_s2 is described by its respective spatial position x_si, x_s1, x_s2 and a respective viewing direction θ_si, θ_si1, θ_s2 onto the observation space B of the respective measurement modality Si, S1, S2. The discretization step corresponds, for example, to step 150_1, 150_2 in Fig. 1

[0035] This is shown schematically in Fig. 3 shown.

[0036] The data (x_si, θ_si), (x_s1, θ_s1, (x_s2, θ_s2), which describe a number of spatial observation points P_s1, P_s2 along the rays S_s1, S_s2, are encoded by corresponding encoders 220_s1, 220_s2, in particular encoders, and are provided as encoded data ρ_si, p_s1, p_s2, which describe a respective spatial observation point, as input for a respective sensor-specific neural network MLP_si, MLP_s1, MLP_s2 and as input for the generic neural network MLP_gen. The encoding step corresponds, for example, to step 160_1, 160_2 in Fig. 1

[0037] Sensor-specific features f_si, f_s1, f_s2 are predicted using a respective sensor-specific neural network MLP_si, MLP_s1, MLP_s2.

[0038] The generic neural network MLP_gen is used to predict sensor-agnostic features f_gen.

[0039] With a respective sensor model SM_si, SM_s1, SM_s2, respective synthetic measurement data m'_si, m'_s1, m'_s2 are generated based on respective sensor-specific features f_si, f_s1, f_s2 and the sensor-agnostic features f_gen.

[0040] The respective synthetic measurement data m'_si, m'_s1, m'_s2 are compared with the respective measurement data m_si, m_s1, m_s2, i.e., for example, the difference is determined. The comparison is evaluated using a respective loss function L_s1, L_s2. The respective loss function L_s1, L_s2 can be optimized during the training process 100. Furthermore, within the framework of the training process 100, parameters of the respective sensor model SM_si, SM_s1, SM_s2 can be optimized in order to optimize a result of the evaluation using the loss function L_s1, L_s2. The loss function L_s1, L_s2 uses, for example, a volumetric rendering equation. These steps are described in Fig.1 summarized in step 140_1, 140_2.

[0041] The trained system 200 can ultimately be used to generate synthetic measurement data m'_si, m'_s1, m'_s2, wherein the synthetic measurement data represent measurement data of a sensor at a further, new position with respect to the observation space B. Accordingly, synthetic measurement data are generated in the sense of further, new views of the scene detectable in the observation space.

[0042] Further embodiments relate to methods for generating synthetic measurement data of a measurement modality with a trained system 200 for generating synthetic measurement data according to the embodiment described above.

[0043] Further embodiments relate to the use of a system according to the embodiment described above and / or a method for generating synthetic measurement data of a measurement modality according to the embodiment described above and / or the synthetic measurement data for training a technical system that is or is installed in a vehicle for assistance-based and / or autonomous driving or in a robot, or which is part of a traffic monitoring system, wherein the synthetic measurement data each relate to an environmental detection of a vehicle environment and / or a robot environment and / or a traffic situation by a system with corresponding sensors. Thus, synthetic measurement data is generated as measurement data of a sensor at a further, new position in relation to a vehicle environment and / or a robot environment and / or a traffic situation.This synthetic measurement data is used to train the technical system that is or will be installed in a vehicle for assistance-based and / or autonomous driving or in a robot, or which is part of a traffic monitoring system.

[0044] The trained technical system can be used in a vehicle for assistance-based and / or autonomous driving or in a robot or as part of a traffic monitoring system to control the vehicle for assistance-based and / or autonomous driving or the robot or the traffic monitoring system. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph., vol. 41, pp. 102:1-102:15, July 2022

[0014]

Claims

[1] Method (100) for training a system (200) for generating synthetic measurement data (m'_si, m'_s1, m'_s2), the method comprising: Providing (110) first measurement data (m_si, m_s1) of at least one first measurement modality (Si, S1) and second measurement data (m_si, m_s2) of a second measurement modality (Si, S2), wherein the first and second measurement data (m_si, m_s1, m_s2) comprise data of an observation space (B) acquired by the first and second measurement modalities (Si, S1, S2), and wherein the first and second measurement modalities (Si, S1, S2) acquire the observation space (B) from mutually different positions, and wherein the measurement modalities (Si, S1, S2) are sensors of a different measurement principle; Predictions (120_i, 120_1) of first sensor-specific features (f_si, f_s1) based on the first measurement data (m_si, m_s1) by means of a first sensor-specific neural network (MLP_s1); Predictions (120_2) of second sensor-specific features (f_s2) based on the second measurement data (m_si, m_s2) by means of a second sensor-specific neural network (MLP_s2); Predictions (120_gen) of sensor-agnostic features (f_gen) based on the first and second measurement data using a generic neural network (MLP_gen); Generating (130_1) first synthetic measurement data (m'_s1) based on the first sensor-specific features (f_s1) and the sensor-agnostic features (f_gen) by means of a first sensor model (SM_s1) and generating (130_2) second synthetic measurement data (m'_s2) based on the second sensor-specific features (f_s2) and the sensor-agnostic features (f_gen) by means of a second sensor model (SM_s1). [2] The method (100) of claim 1, wherein the method (100) comprises: Comparing (140_1) the first synthetic measurement data (m'_s1) with the first measurement data (m_s1) and / or comparing (140_2) the second synthetic measurement data (m'_s2) with the second measurement data (m_s2) and assessing the comparison with a loss function (L), optimizing parameters of at least the first and / or the second sensor model (SM_s1, SM_s2) in order to optimize a result of the assessment by means of the loss function (L). [3] Method (100) according to one of the preceding claims, wherein the method (100) comprises: discretizing (150_i, 150_1) the first measurement data (m_s1) and discretizing (150_i, 150_2) the second measurement data (m_s2), wherein the measurement data each comprise acquired data of the observation space (B) in the form of rays (S_si, S_s1, S_s2) of a respective measurement modality (Si, S1, S2), into data which describe a number of spatial observation points (P_si, P_s1, P_s2) along the rays (S_si, S_s1, S_s2), wherein a respective point (P_si, P_s1, P_s2) is defined by its respective spatial position (x_si; x_s1; x_s2) and a respective viewing direction (θ_si; θ_s1; θ_s2) of the respective measurement modality (Si, S1, S2). [4] Method (100) according to claim 3, wherein the discretization (150_i, 150_1, 150_2) is carried out as a function of respective sensor-specific parameters (p_si; p_s1, p_s2) of the respective first and / or second measuring modality. [5] Method (100) according to one of claims 3 or 4, wherein the data ((x_si, θ_si); (x_s1, θ_s1); (x_s2, θ_s2)) describing a respective spatial observation point (P_si, P_s1, P_s2) are coded (160_i, 160_1, 160_2) and provided as coded data describing a respective spatial observation point (P_si, P_s1, P_s2) as input to a respective sensor-specific neural network (MLP_si; MLP_s1; MLP_s2) and as input to the generic neural network (MLP_gen). [6] Method (100) according to one of the preceding claims, wherein at least one sensor-specific feature (f_s1; f_s2) and / or at least one sensor-agnostic feature (f_gen) is predicted for a respective spatial observation point (P_si, P_s1, P_s2). [7] Method (100) according to one of the preceding claims, wherein a sensor-agnostic feature (f_gen) is at least one of the following features: radiation density, geometry, and / or wherein a sensor-specific feature (f_s1; f_s2) is at least one of the following features: color value, intensity, reflection property, material property. [8] Method (100) according to one of the preceding claims, wherein the method (100) comprises: detecting the observation space (B) by means of at least a first measuring modality (Si, S1) and a second measuring modality (Si, S2), wherein the first and the second measuring modality (Si, S1, S2) detect the observation space (B) from mutually different positions. [9] Method (100) according to one of the preceding claims, wherein a respective measuring modality (Si, S1, S2) is a sensor according to one of the following measuring principles: camera, lidar, radar, ultrasonic sensor. [10] System (200) for generating synthetic measurement data (m'_si, m'_s1, m'_s2), wherein the system (200) is or is trained according to a method (100) according to claims 1 to 9. [11] Method for generating synthetic measurement data (m'_si, m'_s1, m'_s2) of a measurement modality (Si, S1, S2) with a system for generating synthetic measurement data (m'_si, m'_s1, m'_s2) according to claim 10. [12] Using a system according to claim 10 and / or a method for generating synthetic measurement data (m'_si, m'_s1, m'_s2) of a measurement modality (Si, S1, S2) according to claim 11 and / or the synthetic measurement data (m'_si, m'_s1, m'_s2) for training a technical system which is or is installed in a vehicle for assistance-based and / or autonomous driving or in a robot, or which is part of a system for traffic monitoring, wherein the synthetic measurement data each relate to an environmental detection of a vehicle environment and / or a robot environment and / or a traffic situation by a system with corresponding sensors.

Images