Method and device for evaluating sensor data

Abstract

In a computer-implemented method for evaluating sensor data, a point cloud generated on the basis of sensor data is provided. The point cloud is broken down into a plurality of partial point clouds using a specified spatial division. Each partial point cloud is transformed into a corresponding vector representation by an input component of a machine learning model, and a corresponding intermediate output is produced by processing the vector representation by a pre-trained basic model of the machine learning model. An evaluation output is produced and output by processing the intermediate outputs by an output component of the machine learning model.

Description

[0001] Description

[0002] title

[0003] Method and device for evaluating sensor data

[0004] The invention relates to a method and a device for evaluating sensor data. Specifically, the invention relates to a method and a device for evaluating radar data, LiDAR data, ultrasound data, or the like, particularly for use in a motor vehicle.

[0005] State of the art

[0006] Driver assistance systems and autonomous driving require a precise representation of the vehicle's surroundings. In this context, radar sensors are often used alongside camera sensors and LiDAR (Light Detection and Ranging) sensors, as radar is more reliable under various weather conditions and allows for direct speed measurement.

[0007] After processing the raw data, the radar data can be displayed as a point cloud. A radar point is represented by three-dimensional spatial coordinates and additional properties, such as the radar cross-section or the measured Doppler velocity. These radar points can serve as input for machine learning-based artificial neural networks, e.g.,

[0008] B. Object detection networks that output bounding boxes to describe the position and shape of objects and a classification.

[0009] A grid-based network for object detection is described in Ulrich et al., “Improved orientation estimation and detection with hybrid object detection networks for automotive radar,” in 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp. 111-117, 2022. The radar point cloud is first projected into a grid in a bird's eye view and then processed in a neural network similar to an image.

[0010] Disclosure of the invention

[0011] The invention provides a method and a device for evaluating sensor data with the features of the independent patent claims.

[0012] Preferred embodiments are the subject of the respective subclaims.

[0013] According to a first aspect, the invention relates to a computer-implemented method for evaluating sensor data. A point cloud generated from sensor data is provided. The point cloud is decomposed into a plurality of sub-point clouds using a predefined spatial division. Each sub-point cloud is transformed into a corresponding vector representation by an input component of a machine learning model, and a corresponding intermediate output is generated by processing the vector representation by a pre-trained base model of the machine learning model. An evaluation output is generated and output by processing the intermediate outputs by an output component of the machine learning model.

[0014] According to a second aspect, the invention relates to a device for evaluating sensor data, comprising an interface that receives a point cloud generated from sensor data. A computing device divides the point cloud into a plurality of partial point clouds using a predefined spatial division. Each partial point cloud is transformed into a corresponding vector representation by an input component of a machine learning model. A pre-trained basic model of the machine learning model generates a corresponding intermediate output by processing the vector representation. An output component of the machine learning model generates an evaluation output by processing the intermediate outputs and outputs it.

[0015] Advantages of the invention

[0016] The invention enables the use of pre-trained base models for point cloud data for various tasks. By using a simple predefined spatial division, the method is computationally efficient and easily parallelizable, while also being able to extract features effectively.

[0017] Especially compared to approaches that consider relationships between the points in the point cloud, such as Farthest Point Sampling (FPS) or k-Nearest Neighbors (KNN), the proposed method is very computationally inexpensive and easily parallelizable. Furthermore, training and inference time are reduced.

[0018] A foundation model (FM) is a model that is trained on broad data, typically based on large amounts of data, and that can be adapted to many downstream tasks. Foundation models are pre-trained models for a specific downstream task such as classification, segmentation, or object detection, leveraging the advantages of models trained on large and diverse datasets.

[0019] A vector representation (token) is understood to be a (particularly high-dimensional) vectorized representation (or embedding) such that the provided data (i.e., the point cloud) is converted into a form that can be processed by the basic model. The point cloud in vector representation can then be provided to the basic model as an input variable. The pre-trained (and possibly frozen) basic model can thus be used even if the basic model was not trained on the specific input modality (i.e., the original point cloud). According to one embodiment of the computer-implemented method for evaluating sensor data, empty subregions, i.e., subregions of the spatial division in which no points are located, can be ignored for further processing and thus do not need to be evaluated by the machine learning model. This allows the method to be carried out more quickly.

[0020] According to one embodiment of the computer-implemented method for evaluating sensor data, the machine learning model is trained. The input component and / or the output component are trained, while the base model is not trained. The base model has already been previously trained using large amounts of data, so further adaptation is not necessary. Training the input component and / or the output component then enables fine-tuning of the machine learning model.

[0021] According to one embodiment of the computer-implemented method for evaluating sensor data, the sensor data is received from at least one radar sensor, LiDAR sensor, or ultrasonic sensor. In principle, all sensor data that provide point data, i.e., data with (at least among other things) spatial coordinates, can be processed.

[0022] According to one embodiment of the computer-implemented method for evaluating sensor data, each point in the point cloud is assigned two-dimensional or three-dimensional coordinates and at least one sensor feature. The sensor feature depends on the sensor type and can include, for example, a radar scattering cross section or a Doppler velocity.

[0023] According to one embodiment of the computer-implemented method for evaluating sensor data, the input component further processes points of the partial point cloud using point-based transformations and transforms the processed points into the vector representation. By using point-based transformations, the vector representation can subsequently be processed with improved results.

[0024] According to one embodiment of the computer-implemented method for evaluating sensor data, the input component transforms points of the partial point cloud into the vector representation using aggregation. For example, an average can be calculated across the features.

[0025] According to one embodiment of the computer-implemented method for evaluating sensor data, the spatial division comprises subdividing a detected space into equally sized sub-areas. The sub-areas can be disjoint or partially overlap.

[0026] According to one embodiment of the computer-implemented method for evaluating sensor data, the spatial division is independent of the sensor data. The spatial division can be performed, in particular, based on a fixed grid.

[0027] According to one embodiment of the computer-implemented method for evaluating sensor data, the spatial division may comprise several volumes of different sizes in order to generate multi-scale extractions or multi-scale tokens.

[0028] According to one embodiment of the computer-implemented method for evaluating sensor data, the point clouds are transformed using parallel data processing, and the corresponding intermediate outputs are generated. Parallel data processing enables shorter computing times.

[0029] According to one embodiment of the computer-implemented method for evaluating sensor data, generating and outputting the evaluation output comprises generating bounding boxes to describe the position and shape of objects and / or classifying objects. The invention can be used in various fields, such as radar sensors, LiDAR sensors, or ultrasonic sensors in motor vehicles; for radar sensors that use object information, such as stationary radar sensors for traffic monitoring; or for radar sensors for bicycles or other vehicles. Furthermore, the invention can be applied, for example, in the field of robotics, e.g., for obstacle detection in autonomous lawnmowers.

[0030] Further advantages, features and details of the invention will become apparent from the following description, in which various embodiments are described in detail with reference to the drawings.

[0031] Short description of the drawings

[0032] They show:

[0033] Figure 1 is a schematic block diagram of a device for evaluating sensor data according to an embodiment of the invention;

[0034] Figure 2 is a schematic representation of a machine learning model according to an embodiment of the invention;

[0035] Figure 3 is a schematic representation of an input component of the machine learning model shown in Figure 2; and

[0036] Figure 4 is a flowchart of a method for evaluating sensor data according to an embodiment of the invention.

[0037] In all figures, identical or functionally equivalent elements and devices are provided with the same reference numerals. The numbering of process steps serves the purpose of clarity and is generally not intended to imply a specific chronological order. In particular, several process steps can be performed simultaneously. Description of the Embodiments

[0038] Figure 1 shows a schematic block diagram of a device 1 for evaluating sensor data. The device 1 comprises an interface 2, which is wirelessly or wired coupled to at least one sensor device 6. The sensor device 6 can be at least one of a radar sensor, a LiDAR sensor, an ultrasonic sensor, or the like, and generates sensor data. The device 1 and the sensor device can also be configured as a common unit.

[0039] A point cloud is generated from the sensor data. Each point is assigned two-dimensional or three-dimensional spatial coordinates, as well as at least one sensor variable (i.e., a feature). In the case of a radar sensor, the at least one sensor variable can be, for example, a radar cross-section, a Doppler velocity, an azimuth angle, an elevation angle, or the like.

[0040] The sensor device 6 generates a point cloud based on the sensor data and transmits it to the interface 2 of the device 1. The point cloud can be preprocessed by the sensor device 6 or by the device 1, for example by projecting the point cloud into a grid from a bird's eye view.

[0041] The device 1 further comprises a computing device 3 with a memory device 4 in which the received data can be stored. The memory device 4 also stores program instructions, which are executed on a processor 5 of the computing device 3.

[0042] The processor 5 may comprise an application-specific integrated circuit (ASIC), a microprocessor, or the like. The program instructions, in particular, enable the execution of a machine learning model on the processor 5, which includes an input component 8, a pre-trained basic model 9, and an output component 10.

[0043] The machine learning model can first be trained. The input component 8 and / or the output component 10 are trained, while the base model 9 is not trained. For this purpose, the machine learning model can comprise at least one artificial neural network, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), or the like.

[0044] The computing device 3 first divides the point cloud into a plurality of partial point clouds using a predefined spatial division. For this purpose, the space that can be detected by the sensor device is divided or subdivided into several sub-areas. The spatial division is preferably independent of the sensor data of the at least one sensor device 6. The sub-areas can be selected such that they do not overlap. For example, the space can be divided using a regular grid, i.e., each sub-area corresponds to a two-dimensional grid element (e.g., a square area) or a three-dimensional grid element (e.g., a cuboid area).

[0045] According to other embodiments, the subregions may also partially overlap. It is also possible for some subregions to appear multiple times in the spatial decomposition. For example, the room can be divided into smaller subregions (subvolumes) (e.g., with edge lengths of one meter) and simultaneously into larger areas (e.g., with edge lengths of 10 meters), and the entire room can also be viewed. This allows the context of the environment to be additionally taken into account.

[0046] For each sub-area of ​​the space, a corresponding point cloud subset is generated. This means that the point cloud points within the sub-area are assigned to a corresponding point cloud subset. Each point cloud subset is transformed into a corresponding vector representation (i.e., a token) by the input component of the machine learning model. The input component is thus a tokenizer.

[0047] If multiple sensor devices 6, e.g., multiple radar devices, are used, one input component can be learned per sensor device 6. This allows, for example, information from different sensor devices 6 to be fused or different sensor types to be combined. If different sensor modalities are used, the use of different input components per modality enables the fusion of sensor information at the token level.

[0048] The input component is not limited to extracting only the vector representation, but can also compute a position-aware embedding, e.g., based on the position of the extracted volume.

[0049] The pre-trained base model of the machine learning model generates a corresponding intermediate output by processing the vector representation. The pre-trained base model can, for example, be a Vision Transformer (ViT). The use of the pre-trained base model is possible because the input component transforms the point cloud into the same embedding on which the base model was trained.

[0050] The output component generates and outputs an evaluation output by processing the intermediate outputs. The evaluation output can include, for example, object recognition.

[0051] Possible implementations of the machine learning model are explained in more detail below with reference to Figures 2 and 3.

[0052] Figure 2 shows a schematic representation of a machine learning model.

[0053] For this purpose, the captured space B is divided, for example, into four sub-areas B1 to B4, whereby the invention is not limited to a specific number of sub-areas. The point cloud 7 is divided accordingly, i.e., based on the coordinates of each point in the point cloud 7, the point is assigned to a sub-point cloud corresponding to the sub-area B1 to B4. An input component 8 then processes the sub-point clouds and generates four vector representations (or tokens) 21-24, i.e., one vector representation 21-24 for each sub-point cloud.

[0054] A pre-trained base model 9 receives a vector representation 21-24 and generates a corresponding intermediate output 31-34. All intermediate outputs 31-34 are then provided together to an output component 10, which creates an evaluation output.

[0055] Figure 3 shows a schematic representation of an input component 8 of the machine learning model shown in Figure 2.

[0056] For this purpose, the input component 8 comprises a point transformation device 81, which transforms points of a partial point cloud 71. The input component 8 uses point-based transformations, such as a kernel-point convolution (KPConv) network or a graph neural network (GNN), which generally aggregate information from neighboring points in a nonlinear manner (in one or more layers). The points are transformed and enriched in the process. This step can consist of (multiple) fully connected dense layers to transform each point feature individually. The result of this processing step can be a point cloud with the same coordinates for each point, but with enriched features.

[0057] The transformed points 91 are provided to an aggregation device 82 of the input component 8, which transforms the transformed points 91 of the partial point cloud 71 into a vector representation 21 by aggregation. All points and their processed and enhanced features can, for example, be aggregated into a single vector, the so-called token. The aggregation is performed, for example, by a pooling operation, e.g., by averaging, such as calculating the average across all features.

[0058] Figure 4 shows a flowchart of a computer-implemented method for evaluating sensor data.

[0059] In step S1, a point cloud 7 generated from sensor data is provided, and corresponding input data for a machine learning model is generated. The sensor data can be received from at least one radar sensor, LiDAR sensor, or ultrasonic sensor.

[0060] The input data can be a point cloud, or the point cloud can be processed first. The machine learning model comprises an input component (8), a pre-trained base model (9), and an output component (10).

[0061] The machine learning model can be trained. The input component 8 and / or the output component 10 are trained (e.g., using supervised learning or self-supervised learning), but the base model 9 is not trained. Since the base model has already been trained on a large and diverse dataset, the base model is not further adjusted during the training of the machine learning model.

[0062] In a step S2, the point cloud 7 is divided into a plurality of partial point clouds 71 ​​using a predetermined spatial division. For this purpose, the space B detected by the at least one sensor device is divided or subdivided into equally sized partial areas B1-B4. The spatial division is preferably independent of the sensor data of the at least one sensor device 6. In a step S3, each partial point cloud 71 is transformed by the input component 8 into a corresponding vector representation 21-24. For this purpose, the input component 8 can first transform the points of the partial point cloud 71 using point-based transformations and then transforms the processed points into the vector representation 21-24 using aggregation.

[0063] A corresponding intermediate output 31-34 is generated by processing the vector representation 21-24 by the pre-trained basic model 9. The transformation of the partial point clouds 7 and the subsequent generation of the vector representations 21-24 can be calculated in parallel, for example, on different processor cores or different processors.

[0064] In a step S4, the output component 10 generates an evaluation output by processing the intermediate outputs 31-34 and outputs it.

Claims

Claims 1 . Computer-implemented method for evaluating sensor data, comprising the steps: Providing (S1) a point cloud (7) generated from sensor data; Dividing (S2) the point cloud (7) into a plurality of partial point clouds (71) using a predetermined spatial division; Transforming (S3) each partial point cloud (71) into a corresponding vector representation (21-24) by an input component (8) of a machine learning model, and generating a corresponding intermediate output (31-34) by processing the vector representation (21-24) by a pre-trained basic model (9) of the machine learning model; and Generating and outputting (S4) an evaluation output by processing the intermediate outputs (31-34) by an output component (10) of the machine learning model.

2. The method according to claim 1, wherein the machine learning model is trained, wherein the input component (8) and / or the output component (10) are trained and the basic model (9) is not trained.

3. The method according to claim 1 or 2, wherein the sensor data is received from at least one radar sensor, LiDAR sensor or ultrasonic sensor.

4. Method according to one of the preceding claims, wherein each point of the point cloud (7) is assigned two-dimensional or three-dimensional coordinates and at least one sensor feature.

5. Method according to one of the preceding claims, wherein the input component (8) further processes points of the partial point cloud (71) by means of point-based transformations and transforms the processed points into the vector representation (21-24).

6. Method according to one of the preceding claims, wherein the input component (8) transforms points of the partial point cloud (71) into the vector representation (21-24) by means of aggregation.

7. Method according to one of the preceding claims, wherein the spatial division comprises a subdivision of a detected space (B) into equally sized sub-areas (B1 -B4).

8. Method according to one of the preceding claims, wherein the spatial division is independent of the sensor data.

9. Method according to one of the preceding claims, wherein the point clouds (7) are transformed by means of parallel data processing and the corresponding intermediate outputs (31-34) are generated.

10. A device (1) for evaluating sensor data, comprising: an interface (2) for receiving a point cloud (7) generated from sensor data; and a computing device (3) designed to: divide the point cloud (7) into a plurality of partial point clouds (71) using a predetermined spatial division, to transform each partial point cloud (71) into a corresponding vector representation (21 - 24) by an input component (8) of a machine learning model, and by a pre-trained basic model (9) of the machine learning model by processing the vector representation (21 - 24) to generate a corresponding intermediate output (31-34), and to generate and output an evaluation output by an output component (10) of the machine learning model by processing the intermediate outputs (31-34).

Images