Method for creating reference map representation

Generate reference map representations and reconstruct point clouds through artificial neural networks, solving the problems of motor vehicle positioning and point cloud simulation, achieving efficient and accurate environmental understanding and simulation, and is suitable for a variety of surrounding environments.

CN119007183APending Publication Date: 2024-11-22ROBERT BOSCH GMBH
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410616139.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-17
Filing Date
2024-05-17
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

It is difficult for the prior art to efficiently create reference map representations for motor vehicle positioning and point cloud simulation, and traditional methods change strongly when the environmental sensor state changes, lacking correlation map features.

Method used

Through artificial neural network, the reference map representation is generated using encoder and decoder training. The encoder creates an environmental map based on point cloud data. The decoder reconstructs the point cloud without using point cloud data, and combines the environmental sensor state data to realize the machine High-speed train positioning and point cloud simulation.

Benefits of technology

It realizes efficient creation of reference map representations and point cloud reconstruction, improves the accuracy of motor vehicle positioning and point cloud simulation, and is suitable for applications in different surrounding environments, reducing dependence on real data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007183A_ABST
    Figure CN119007183A_ABST
Patent Text Reader

Abstract

A method for creating a reference map representation using an artificial neural network comprising an encoder and a decoder, comprising: receiving point cloud data representing a point cloud created by a motor vehicle-based environment sensor detection environment of an environment of a motor vehicle; receiving state data representative of a state of the environment sensor during detection of the environment by the environment sensor; training encoders and decoders based on these data, including: the encoder creating a reference map representation of the environment based on these data; a decoder reconstructs the point cloud based on the reference map representation created by the encoder and based on the state data and without using the point cloud data in order to create a reconstructed point cloud; the encoder and decoder are trained until a predetermined end-of-training decision criterion is satisfied in order to obtain a reference map representation created by the trained encoder, decoder, and trained encoder. The invention also relates to a method for locating a motor vehicle, to a method for simulating a point cloud, to a device, to a computer program, and to a machine-readable storage medium.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for creating a reference map representation, a method for localizing a motor vehicle, a method for simulating a point cloud, a device, a computer program and a machine-readable storage medium. Background Art

[0002] The published patent application DE 10 2016 109 298 A1 discloses a fusion method for transverse traffic applications using a radar device and a camera.

[0003] Publication DE 10 2019 127 283 A1 discloses a method for detecting a vehicle carrying a load A system and method for detecting an object in a three-dimensional surrounding environment.

[0004] The publication EP 3 929 807 A1 discloses a system and a method for a sensor fusion system. Summary of the invention

[0005] The task on which the invention is based can be seen as providing a method for creating a reference map representation.

[0006] The object on which the present invention is based can also be seen as providing a method for locating a motor vehicle.

[0007] The object on which the present invention is based can also be seen as providing a method for simulating point clouds.

[0008] The object underlying the present invention can also be seen as providing a device.

[0009] The task on which the invention is based can also be seen as providing a computer program.

[0010] The object underlying the present invention can also be seen as providing a machine-readable storage medium.

[0011] These objects are achieved by means of the present invention. Advantageous embodiments of the present invention are the subject matter of the individual preferred embodiments.

[0012] According to a first aspect, a method for creating a reference map representation using an artificial neural network is provided, wherein the neural network comprises an encoder and a decoder, the method comprising the following steps:

[0013] receiving point cloud data representing a point cloud of the surroundings of a motor vehicle created based on detection of the surroundings of the motor vehicle by an environmental sensor of the motor vehicle,

[0014] receiving state data representing a state of the environmental sensor during said detection of the surrounding environment by the environmental sensor,

[0015] The encoder and the decoder are trained based on the point cloud data and the state data, wherein the training comprises the following steps:

[0016] creating a reference map representation of the surroundings of the motor vehicle by means of the encoder based on the point cloud data and the status data,

[0017] reconstructing the point cloud by the decoder based on the reference map representation created by the encoder and based on the state data and without using the point cloud data, so as to create a reconstructed point cloud,

[0018] The encoder and the decoder are trained until a predetermined end-of-training criterion is met in order to obtain a trained encoder and a trained decoder and a reference map representation created by the trained encoder.

[0019] According to a second aspect, a method for localizing a motor vehicle using a decoder trained according to the method according to the first aspect and a reference map representation representing the surroundings of the motor vehicle created according to the method according to the first aspect is provided, the method comprising the following steps:

[0020] receiving point cloud data representing a point cloud of the surroundings of the motor vehicle created based on detection of the surroundings of the motor vehicle by an environmental sensor of the motor vehicle,

[0021] If the state of the surroundings sensor during the detection is a query state, querying the decoder with the query state of the surroundings sensor using a reference map representation in order to create a further point cloud by the decoder, the further point cloud representing a point cloud of the surroundings of the motor vehicle that was created based on the detection of the surroundings of the motor vehicle by the surroundings sensor of the motor vehicle,

[0022] The motor vehicle is localized based on the point cloud and the further point cloud.

[0023] According to a third aspect, there is provided a method for simulating a point cloud using a decoder trained according to the method according to the first aspect and a reference map representation created according to the method according to the first aspect, the method comprising the following steps:

[0024] determining a simulated trajectory of the motor vehicle by means of the surroundings represented by the reference map representation,

[0025] determining a state course of a simulated state of an environmental sensor of the motor vehicle during a simulated passage of the vehicle over the trajectory,

[0026] A point cloud is created by the decoder based on the state data in order to simulate a point cloud that is created based on detection of the surroundings of the motor vehicle by the surroundings sensor during a simulated travel of the motor vehicle along the trajectory.

[0027] According to a fourth aspect, a device is provided, which is configured to implement all steps of the method according to the first aspect and / or according to the second aspect and / or according to the third aspect.

[0028] According to a fifth aspect, a computer program is provided, which comprises instructions which, when executed by a computer, for example by a device according to the fourth aspect, cause the computer to implement a method according to the first aspect and / or according to the second aspect and / or according to the third aspect.

[0029] According to a sixth aspect, there is provided a machine-readable storage medium on which is stored the computer program according to the fifth aspect.

[0030] The device comprises, for example, one or more processor devices, which may, for example, each have one or more processors, wherein the one or more processor devices each execute one or more, in particular different, steps of the computer program.

[0031] The invention is based on and includes the following recognition: The corresponding above-mentioned tasks can be solved by creating a reference map representation using an artificial neural network, wherein the neural network includes an encoder and a decoder. The encoder and the decoder are trained based on training data until a predetermined end-of-training criterion is met. The training data include point cloud data and state data. Within the scope of this training, the point cloud is reconstructed by the decoder. This is based solely on the reference map representation created by the encoder and on the state data, and therefore no point cloud data are used. This means that the decoder does not use point cloud data to reconstruct the point cloud.

[0032] Thus, as a result of the training a trained decoder, a trained encoder and a reference map representation created by the trained encoder are obtained.

[0033] The correspondingly trained decoder and the correspondingly created reference map representation can then be used in an efficient manner for further applications. Two exemplary applications are defined according to the method according to the second aspect and according to the third aspect: localization of a motor vehicle and simulation of a point cloud.

[0034] Thus, by using an artificial neural network, a reference map representation can be efficiently created, which can then be used in an advantageous manner for further useful applications.

[0035] The trained encoder can be used in an advantageous manner to create other reference map representations for other surroundings. Thus, the application described here can be expanded or applied, by way of example, to other surroundings.

[0036] This means, for example, that according to the approach described here, a reference map representation can be created for a first city. The trained encoder can then create a further reference map representation for a second city different from the first city. Thus, corresponding reference map representations can be created and subsequently used in an efficient manner for different surroundings, i.e., for example, for different cities, by a single training of the encoder and decoder.

[0037] The reference map representation includes, or is, for example, a reference point cloud.

[0038] In one implementation of the method according to the first aspect, it is provided that the state of the environmental sensor includes attitude and speed.

[0039] This achieves the technical advantage, for example, that the state indicates particularly meaningful information. The attitude and velocity of the surroundings sensor can be used in an efficient manner to enrich or expand the points of the point cloud with meaningful information, as described below.

[0040] In one embodiment of the method according to the second aspect, a Kalman filter is used. For example, a previous state estimate based on the Kalman filter predicts what the current state of the environmental sensor is. Thus, for example, the query state is the state predicted when using the Kalman filter. For example, an algorithm for registering a point cloud is used in order to correct the predicted state based on a further point cloud, wherein the motor vehicle is localized based on the corrected predicted state.

[0041] In one embodiment of the method according to the second aspect, a brute force localization algorithm can be used, in which the encoder is queried in a certain space in a plurality of possible states, in particular all possible states, and in which a state is selected in which the further point cloud best corresponds to the point cloud of the surroundings of the motor vehicle, which was created based on the detection of the surroundings of the motor vehicle by the environmental sensors of the motor vehicle, and the motor vehicle is localized based on the selected state.

[0042] In one embodiment of the method according to the first aspect, it is provided that points of a point cloud are extended based on state data with relative positions, orientations and speeds relative to an environmental sensor, respectively, so as to obtain a state-extended point cloud, wherein the points of the state-extended point cloud respectively contain corresponding relative positions, corresponding orientations and corresponding speeds relative to the environmental sensor, wherein the encoder and the decoder are trained based on the state-extended point cloud, thereby creating a reference map representation through the encoder based on the state-extended point cloud.

[0043] This achieves the following technical advantage, for example: a reference map representation can be created efficiently. This means that, according to this embodiment, it is provided in particular that the points of the point cloud are enriched, i.e. extended, with further information. In English, the term "augment" is used for such an enrichment or extension. The encoder can then efficiently create a reference map representation based on such an enriched point cloud. A state-extended point cloud can also be referred to as an enriched point cloud.

[0044] It is very advantageous to supplement the reference map representation with information about the state of the environment sensors. This is due to the fact that the point cloud measured by the environment sensors may change strongly if the state of the environment sensors changes, for example if the viewing angle of the environment sensors relative to the surroundings is different, or if the speed is different, or if the distance is different. It is therefore advantageous to provide information about the state of the environment sensors in order to create a map feature that detects the dependency on the viewing angle.

[0045] In one embodiment of the method according to the first aspect, it is provided that, before the training, points assigned to mobile objects are filtered from the following point clouds: the encoder should create a reference map representation based on the point cloud in order to obtain a filtered point cloud, the points of which are state-expanded and are respectively only still assigned to static objects, wherein the encoder creates a reference map representation based on the filtered point cloud.

[0046] This achieves the technical advantage that, for example, the reference map representation can be efficiently created. According to this embodiment, moving objects are therefore filtered from the point cloud on which the encoder is to create the reference map representation. Since moving objects may change from one trip of the motor vehicle to another, it is advantageous not to use such moving objects for creating the reference point cloud. For this reason, static objects have a much higher relevance.

[0047] The expression "those point clouds" refers to the point cloud, for example. For example, it may also refer to the state-expanded point cloud.

[0048] In one embodiment of the method according to the first aspect, it is provided that the encoder creates a reference map representation in such a way that a point of the reference map representation contains one or more features, from which only one or more determined features are selected based on state data, so that a reference map representation is obtained from the selected features, the point of the reference map representation only also containing the selected features, wherein the decoder reconstructs a point cloud based on the reference map representation obtained from the selected features.

[0049] This achieves, for example, the following technical advantage: a reference map representation can be created efficiently. This achieves, for example, the following technical advantage: a point cloud can be reconstructed efficiently.

[0050] In addition to efficiency, this selection can also be helpful during the training phase, since it acts as a regularizer and forces the optimizer to only update relevant model parameters for the respective input data, which in turn can contribute to improving the accuracy and generalizability of the trained model.

[0051] In one embodiment of the method according to the first aspect, it is provided that a subregion of the reference map representation is selected based on the state data and / or based on the field of view of the environment sensor, wherein the decoder reconstructs the point cloud based on the selected subregion. Such a subregion is also called a "Region of Interest" in English and a "Region von Interesse" or "Bereich von Interesse" in German.

[0052] This achieves, for example, the technical advantage that the point cloud can be efficiently reconstructed.

[0053] In addition to efficiency, the selection can also be helpful in the training phase, as it acts as a regularizer and forces the optimizer to update only the relevant model parameters for each input data, which in turn can contribute to improving the accuracy and generalizability of the learned model.

[0054] In one embodiment of the method according to the first aspect, it is provided that the predetermined training end criterion states that, in order to satisfy the requirement, a chamfer distance between the reconstructed point clouds must be smaller than or equal to a predetermined threshold value.

[0055] This achieves, for example, the following technical advantage: it is possible to efficiently determine when a training end decision criterion is met, and thus it is possible to efficiently determine when the training should be ended.

[0056] In one embodiment of the method according to the first aspect, it is provided that the encoder is a DGCNN and the decoder is a Transformer Network (Transformer-Netz).

[0057] This achieves, for example, the technical advantage of using a particularly suitable encoder or a particularly suitable decoder.

[0058] The abbreviation "DGCNN" stands for "Dynamic Graph Convolutional Neural Network".

[0059] In one embodiment of the method according to the second aspect, it is provided that a point of a reference map representation contains one or more features, from which only one or more determined features are selected based on status data, so that a reference map representation is obtained from the selected features, the point of which reference map representation also only contains the selected features, wherein the decoder creates a further point cloud based on the reference map representation obtained from the selected features.

[0060] This achieves, for example, the technical advantage that further point clouds can be efficiently created.

[0061] In one embodiment of the method according to the second aspect, it is provided that a subregion of the reference map representation is selected, wherein the decoder creates a further point cloud based on the selected subregion. Such a subregion is also called "Region of Interest" in English and "Region von Interesse" or "Bereich von Interesse" in German.

[0062] In an embodiment of the method according to the first aspect, the method is a computer-implemented method.

[0063] In an embodiment of the method according to the second aspect, the method is a computer-implemented method.

[0064] In an embodiment of the method according to the third aspect, the method is a computer-implemented method.

[0065] The device according to the fourth aspect is, for example, configured in terms of programming to execute the computer program according to the fifth aspect.

[0066] The implementations made in conjunction with the method according to the first aspect apply analogously to the method according to the second aspect and / or the method according to the third aspect, and vice versa.

[0067] The technical features and technical functionality of the method according to the first aspect result analogously from the corresponding technical functionality and technical features of the method according to the second aspect and / or the method according to the third aspect, and vice versa.

[0068] The embodiments and examples described here can be combined with one another in any desired manner, even if these combinations are not explicitly described.

[0069] The step of detecting the surroundings of the motor vehicle by means of environmental sensors of the motor vehicle is, for example, explicitly included in the method according to the first aspect and / or in the method according to the second aspect.

[0070] In the sense of this description, an environmental sensor is, for example, one of the following environmental sensors: a radar sensor, a lidar sensor and an ultrasonic sensor.

[0071] The point cloud is thus, for example, a radar point cloud or a lidar point cloud or an ultrasonic point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The present invention will be further described below based on preferred embodiments. It is shown here:

[0073] Figure 1 : A flow chart of the method according to the first aspect;

[0074] Figure 2 : A flow chart of the method according to the second aspect;

[0075] Figure 3 : A flow chart of the method according to the third aspect;

[0076] Figure 4 :equipment;

[0077] Figure 5 : a machine-readable storage medium; and

[0078] Figure 6 : Block diagram. DETAILED DESCRIPTION

[0079] Figure 1 A flow chart showing a method for creating a reference map representation using an artificial neural network, wherein the neural network comprises an encoder and a decoder, the method comprising the following steps:

[0080] receiving 101 point cloud data representing a point cloud of the surroundings of a motor vehicle created based on detection of the surroundings of the motor vehicle by an environmental sensor of the motor vehicle,

[0081] receiving 103 state data, the state data representing the state of the environmental sensor during detection of the surrounding environment by the environmental sensor,

[0082] An encoder and a decoder are trained 105 based on the point cloud data and the state data, wherein the training comprises the following steps:

[0083] creating 107 a reference map representation of the surroundings of the motor vehicle by means of the encoder based on the point cloud data and the status data,

[0084] reconstructing 109 the point cloud by a decoder based on the reference map representation created by the encoder and based on the state data and without using the point cloud data, so as to create a reconstructed point cloud,

[0085] The encoder and the decoder are trained 105 until a predetermined end-of-training criterion is met in order to obtain a trained encoder and a trained decoder and a reference map representation created by the trained encoder.

[0086] This means that after the training step, it is checked in step 111 whether a predetermined end-of-training decision criterion is met. If this is not the case, the training is continued. If the predetermined end-of-training decision criterion is met, the trained encoder, the trained decoder and the reference map representation created by the trained encoder are output in step 113.

[0087] For example, a plurality of predetermined training end criteria are provided which must be met in order for the training to be considered completed and thus terminate the training.

[0088] In the sense of this specification, a state includes, for example, a pose and a velocity or a velocity vector. The pose describes the position and orientation.

[0089] In the sense of this description, state data include, for example, the state of the motor vehicle during the detection of its surroundings by environmental sensors. The state of the motor vehicle includes, for example, the attitude and speed or speed vector. The attitude of the motor vehicle specifies, for example, the position and orientation.

[0090] If the term “speed” is used, the term “speed vector” should always be read in conjunction. This means in particular that in the sense of this description, a state includes, for example, a posture and a speed vector.

[0091] Figure 2 A flow chart showing a method for localizing a motor vehicle using a decoder trained according to the first aspect and a reference map representation representing the surroundings of the motor vehicle created according to the first aspect, the method comprising the following steps:

[0092] receiving 201 point cloud data representing a point cloud of the surroundings of a motor vehicle created based on detection of the surroundings of the motor vehicle by environmental sensors of the motor vehicle,

[0093] If the state of the surroundings sensor during the detection is a query state, querying 203 the decoder with this query state of the surroundings sensor using the reference map representation in order to create 205 a further point cloud by the decoder, the further point cloud representing a point cloud of the surroundings of the motor vehicle created based on the detection of the surroundings of the motor vehicle by the surroundings sensor of the motor vehicle,

[0094] The motor vehicle is localized 207 based on the point cloud and the further point cloud.

[0095] Figure 3 A flow chart showing a method for simulating a point cloud using a decoder trained according to the first aspect and a reference map representation created according to the first aspect, the method comprising the following steps:

[0096] determining 301 a simulated trajectory of the motor vehicle by reference to a surrounding environment represented by the map representation,

[0097] determining 303 a state progression of a simulated state of an environmental sensor of the motor vehicle during a simulated passage of the motor vehicle over the trajectory,

[0098] A point cloud is created 305 by the decoder based on the state data in order to simulate a point cloud that is created based on the detection of the surroundings of the motor vehicle by the surroundings sensor during the simulated travel of the motor vehicle along the trajectory.

[0099] Figure 4 A device 401 is shown which is configured to carry out all steps of the method according to the first aspect and / or according to the second aspect and / or according to the third aspect.

[0100] Figure 5 A machine-readable storage medium 501 is shown on which a computer program 503 is stored. The computer program 503 comprises commands which, when executed by a computer, cause the computer to carry out the method according to the first aspect and / or according to the second aspect and / or according to the third aspect.

[0101] Figure 6 A block diagram 601 is shown, which is intended to explain the approach described here by way of example.

[0102] The block diagram 601 is divided into an upper part 603 and a lower part 605, which are separated from one another by a line with reference numeral 607. The upper part 603 relates to the encoder, while the lower part 605 relates to the decoder.

[0103] After training of the encoder and the decoder, only the lower part 605 is needed in the application. This is further explained below.

[0104] A plurality of point clouds 609 are used, which have been created, for example, by a plurality of motor vehicles during a plurality of driving periods by detecting corresponding surroundings by means of environmental sensors of the motor vehicles. The point clouds are, for example, radar point clouds. This means that, for example, a plurality of motor vehicles detect a common surrounding during corresponding driving periods of the motor vehicles using corresponding environmental sensors, thereby creating a plurality of radar point clouds, generally point clouds, and these are used for training accordingly according to the approach described here. For example, it can be provided that the motor vehicles drive through a certain area once or several times, so that the same surroundings are detected once or several times by the environmental sensors of the motor vehicles. Thus, a plurality of point clouds can also be created, which can be used for training encoders and decoders according to the approach described here.

[0105] It should be noted here that the approach described here can also be extended to a plurality of point clouds and a plurality of states and a plurality of vehicles.

[0106] Furthermore, it is provided that the corresponding state of the corresponding environment sensor during the detection of the surroundings is used. Accordingly, a plurality of states 611 are available for training the encoder and the decoder. In addition to the corresponding state 611 of the corresponding environment sensor, the state of the motor vehicle can optionally also be used for training.

[0107] The corresponding state 611 of the motor vehicle is measured, for example, by suitable sensors, which are known per se to a person skilled in the art.

[0108] Therefore, for the training of the encoder and the decoder, a plurality of point clouds 609 and a plurality of states 611 of the surrounding sensors during the detection are available, in particular optionally also the states of the motor vehicle during the detection of the corresponding surroundings.

[0109] According to function block 613, point cloud 609 is extended with corresponding state 611. Thus, a plurality of state-extended point clouds 615 are obtained. According to function block 617, the plurality of state-extended point clouds 615 are dynamically filtered (English: "dynamics filtering"), which means that moving objects are filtered from the state-extended point clouds 615.

[0110] Subsequently, i.e. after filtering, a plurality of filtered state-extended point clouds 619 are available, which are used by the encoder 621 to create a reference map representation 623. In particular, each point of the reference map representation 623 contains a feature vector, which represents a plurality of features. This means that one or more features can be assigned to a point of the created reference map representation 623.

[0111] According to the lower part 605 of the block diagram 601, the created reference map representation 623 is used for training a decoder 629. In detail, the state 611 that has been used for the upper part 603 is used in order to select one or more determined features from the features of the reference map representation 623 based thereon. This is done according to functional block 625. Subsequently, a reference map representation 627 is obtained from the selected features in the sense of the present description. The decoder 629 uses the reference map representation 627 obtained from the selected features for reconstructing the point cloud 611 belonging to the corresponding state 611. For this purpose, the decoder 629 uses the state 611 and the reference map representation 627 obtained from the selected features. For the reconstruction, the decoder 629 does not use the original point cloud 609.

[0112] The decoder 629 thus creates a reconstructed point cloud 631. This reconstructed point cloud is compared with the point cloud 609 within the scope of training in order to determine a reconstruction loss 633. The reconstruction loss 633 can be represented, for example, by a chamfer distance. If the reconstruction loss 633 is small enough, the training is terminated, for example. Otherwise, the encoder 621 and the decoder 629 are further trained in this way until the reconstruction loss 633 is acceptable, i.e., for example, the chamfer distance is less than or less than or equal to a predetermined threshold.

[0113] A trained encoder 621 and a trained decoder 629 as well as a reference map representation 623 are then therefore available, which can be further used according to the approach described herein.

[0114] For example, the input point cloud information (radar point cloud) is enriched with information about the apparent state, ie the relative position, orientation and speed between each measured point and the radar sensor, in order to obtain a state-extended point cloud.

[0115] For example, a filtering of radar points corresponding to dynamic objects is performed, for example based on a self-compensated radial velocity (“ego” stands for the motor vehicle), wherein the radial velocity is ascertained by the radar sensor using the Doppler effect.

[0116] For certain states, for example the attitude and speed of an environmental sensor or optionally of the motor vehicle, a point cloud is reconstructed using a decoder.

[0117] In other words, a further point cloud is created by the decoder solely on the basis of the reference map representation and on the basis of the states, which further point cloud should ideally correspond to the original point cloud, wherein in reality certain losses usually exist.

[0118] During training, a set of measured point clouds together with the corresponding vehicle states and environmental sensor states are fed into a first stage (upper part 603 of block diagram 601 ) in order to create a reference map representation which in particular detects correlations between visual states.

[0119] This reference map representation is then fed into the second stage (lower portion 605 of block diagram 601) together with the vehicle state and environmental sensor state used to create the reference map representation, in order to reconstruct the point cloud or to generate radar measurements for the prediction of these states. The reconstruction loss (e.g., "chamfer distance") can then be used to optimize the trainable parameters of the encoder and decoder.

[0120] As long as the trainable components are optimized for a specific data set, a reference map representation of any surroundings can be created when there is a set of measurements from the surroundings with environmental sensors and, in particular, optionally known states of the motor vehicle. And as long as a reference map representation is created, it can be used to generate or create a predicted measurement for each requested state within the surroundings represented by the reference map representation.

[0121] Thus, for example, improved localization can be achieved as follows: After creating a reference map representation that detects information about the visual state, the reference map representation can be queried for the expected radar measurements in each arbitrary ambient sensor state, and the localization can thus be formulated as a point cloud registration problem between the expected and actual radar measurements.

[0122] The solution to the point cloud registration problem is significantly more robust and more accurate than localization based on traditional radar maps, which contain agnostically accumulated data from all viewing states, because the algorithm is able to focus on the relevant data from the reference map representation and not be distracted by other irrelevant data or try to assign things that look very different.

[0123] The reference map representation can be used in an advantageous manner to determine in which lane the motor vehicle is located, which can be regarded as a rough localization. This means, for example, that a classifier based on a comparison of actual measurements with expected measurements for each lane can be defined.

[0124] Furthermore, radar data can be simulated: With the latest progress on the road to autonomous driving, the need for huge amounts of data for training artificial intelligence algorithms and for validating implemented algorithms and functions has increased. Simulation is a relatively cost-effective method that can be used for this purpose, since it reduces the amount of real data required (and the costs associated therewith) and the simulation can generally be set up in such a way that rare events or desired application situations can be simulated.

[0125] The proposed reference map representation is particularly suitable for this purpose, in contrast to other approaches, which are based on complex radar sensor simulations (physics-based or machine learning) and detailed 3D surrounding environment models with known material characteristics, which require huge modeling overhead and have only limited faithfulness and true proximity.

[0126] The decoder can generate synthetic data (point clouds) from a number of simulated drives using a reference map representation, in which, for example, the trajectory of the motor vehicle (including speed) and the state of the surroundings can be defined and the reference map representation can be queried for radar measurements along the trajectory. In this way, huge data volumes are generated that enable virtual training / testing / validation for a number of application situations (e.g. lane changes, acceleration and braking, etc.).

[0127] The simulation can be used for measurements in a static world. The simulation can be extended by modeling the occlusion effects of dynamic objects in the scene by not returning radar points of beams that are occluded by such objects.

Claims

1. A method for creating a reference map representation (623) using an artificial neural network, wherein: The neural network comprises an encoder (621) and a decoder (629), and the method comprises the following steps: receiving (101) point cloud data representing a point cloud (609) of the surroundings of a motor vehicle created based on detection of the surroundings of the motor vehicle by an environmental sensor of the motor vehicle, receiving (103) state data representing a state of the environmental sensor during the detection of the surrounding environment by the environmental sensor (611), The encoder and the decoder (629) are trained (105) based on the point cloud data and the state data, wherein the training comprises the following steps: creating (107) a reference map representation (623) of the surroundings of the motor vehicle by means of the encoder (621) based on the point cloud data and the status data, reconstructing (109) a point cloud (609) by means of the decoder (629) based on a reference map representation (623) created by means of the encoder (621) and based on the state data and without using the point cloud data, so as to create a reconstructed point cloud (631), The encoder (621) and the decoder (629) are trained (105) in this way until a predetermined training end judgment criterion is met, so as to obtain a trained encoder (621) and a trained decoder (629) and a reference map representation (623) created by the trained encoder (621).

2. The method according to claim 1, wherein: The state (611) of the environmental sensor includes attitude and speed.

3. The method according to claim 1 or 2, wherein: Based on the state data, the points of the point cloud (609) are respectively extended with relative positions, orientations and speeds relative to the environmental sensor to obtain a state-extended point cloud (615), wherein the points of the state-extended point cloud respectively contain corresponding relative positions, corresponding orientations and corresponding speeds relative to the environmental sensor, wherein the encoder (621) and the decoder (629) are trained based on the state-extended point cloud (615), thereby creating the reference map representation (623) based on the state-extended point cloud (615) through the encoder (621).

4. A method according to any one of the preceding claims, wherein: Prior to the training, points assigned to mobile objects are filtered from those point clouds (609) from which the encoder (621) is to create the reference map representation (623) based on the point clouds in order to obtain a filtered point cloud (619), the points of which are only still assigned to static objects, wherein the encoder (621) creates the reference map representation (623) based on the filtered point cloud.

5. A method according to any one of the preceding claims, wherein: The encoder (621) creates the reference map representation (623) in such a way that a point of the reference map representation contains one or more features, from which only one or more determined features are selected based on the status data, so that a reference map representation (623) is obtained from the selected features, and the point of the reference map representation only further contains the selected features, wherein the decoder (629) reconstructs the point cloud (609) based on the reference map representation (623) obtained from the selected features.

6. A method according to any one of the preceding claims, wherein: The predetermined training end judgment criterion states that in order to satisfy the requirement, the chamfer distance between the reconstructed point cloud and the point cloud (609) must be less than or equal to a predetermined threshold.

7. A method according to any one of the preceding claims, wherein: The encoder (621) is a DGCNN and / or the decoder (629) is a Transformer Network.

8. A method for localizing a motor vehicle using a decoder (629) trained according to any one of the preceding claims and a reference map representation (623) representing the surroundings of the motor vehicle created according to any one of the preceding claims, the method comprising the following steps: receiving (201) point cloud data representing a point cloud (609) of the surroundings of the motor vehicle created based on detection of the surroundings of the motor vehicle by an environmental sensor of the motor vehicle, If the state of the environmental sensor during the detection is a query state, the decoder (629) is queried (203) with the query state of the environmental sensor using the reference map representation (623) in order to create (205) a further point cloud (609) by the decoder (629), the further point cloud representing a point cloud of the surroundings of the motor vehicle created based on the detection of the surroundings of the motor vehicle by the environmental sensor of the motor vehicle, The motor vehicle is localized (207) based on the point cloud (609) and the further point cloud (609).

9. The method according to claim 8, wherein: The point of the reference map representation (623) contains one or more features, from which only one or more determined features are selected based on the status data, so as to obtain a reference map representation (623) from the selected features, and the point of the reference map representation also only contains the selected features, wherein the decoder (629) creates a further point cloud (609) based on the reference map representation (623) obtained from the selected features.

10. A method for simulating a point cloud (609) using a decoder (629) trained according to any one of claims 1 to 7 and a reference map representation (623) created according to any one of claims 1 to 7, the method comprising the following steps: determining (301) a simulated trajectory of the motor vehicle by means of the surroundings represented by the reference map representation (623), determining (303) a state progression of a simulated state of an environmental sensor of the motor vehicle during a simulated travel of the trajectory by the motor vehicle, A point cloud (609) is created (305) by the decoder (629) based on the state data in order to simulate a point cloud (609) created based on detection of the surroundings of the motor vehicle by the environmental sensor during simulated driving of the motor vehicle along the trajectory.

11. A device (401) configured to carry out all the steps of the method according to any one of the preceding claims.

12. A computer program (503), comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 10.

13. A machine-readable storage medium (501) on which a computer program (503) according to claim 12 is stored.

Citation Information

Cited By

  • Simulation point cloud generation method and system for earthwork variation calculation and evaluation

    CN122574295A

  • A simulation point cloud generation method and system for earthwork change quantity calculation evaluation

    CN122574295B