Method for processing sensor data, control device, program product and storage medium

Abstract

The invention relates to a method for processing sensor data, the sensor data being present as a point cloud (100) consisting of individual points (102), the points being projected into at least two grids (106) having different resolutions, points (102) from a partial region (108) of the point cloud (100) are projected into a first grid (106a) having a higher resolution and further points (102) of the point cloud (100) are projected into a second grid (106b) having a lower resolution, where before compressing attributes from the second grid (106b), the points (102) of the partial region (108) of the point cloud (100) are compressed into the first grid (106a) and the further points (102) of the point cloud (100) are compressed into the second grid (106b). Attributes from the first grid (106a) are compressed and arranged in the second grid (106b). The invention also relates to a control device, a computer program product and a machine-readable storage medium.

Description

Technical Field

[0001] The invention relates to a method for processing sensor data, a corresponding control device, and a corresponding computer program product. Background Art

[0002] The point cloud from the sensor can be rasterized, i.e., projected into the cells of a grid. For object recognition, the grid can be compressed by a neural network in multiple stages until it becomes a bottleneck for the information. Beyond the bottleneck, objects can be searched, found, classified, and labeled in the compressed information.

[0003] Here, depending on the local point density of the point cloud, many cells of the grid remain empty when projecting, while multiple points are projected into other cells. Summary of the Invention

[0004] Against this background, a method for processing sensor data, a corresponding control device, a corresponding computer program product, and a corresponding storage medium are proposed using the solution proposed here. Advantageous expansions and improvements of the solution proposed here are derived from the description and are described below.

[0005] Advantages of the invention

[0006] In the solution proposed here, grids with different extension scales and different cell densities are used to receive the points of the point cloud during projection. In areas with high point density, small-scale grids are used locally and in a limited manner. A grid with small cells. Areas with low point density are projected onto at least one large-scale grid with large cells. The small-scale grid is then compressed in the compression phase of the neural network and then inserted into the large-scale grid with larger cells at the appropriate location. The larger grid with the inserted compressed smaller grid is then compressed in the next compression phase of the neural network and again inserted into the larger grid with larger cells. This process continues until a predefined compression is achieved. The object is then searched for in the compressed information in a conventional manner.

[0007] The proposed method saves storage space and computing power because only regions of the point cloud with a high point density (i.e., high specific information content) are subjected to the multiple compression stages of the neural network. The storage requirement is reduced with each compression because the entire point cloud is not stored in each compression stage. Since only the corresponding subregion of the point cloud has a high point density, the corresponding grid to be compressed is smaller than the entire point cloud. This requires limited computing power. Only in one of the final compression stages are the last remaining points compressed, and thus the entire point cloud.

[0008] Additionally, information may be lost in each compression stage. This is less relevant in areas of the point cloud with many points and therefore a lot of information than in areas with few points and therefore little information. In areas with little information, this small amount of information can be obtained in an improved manner by reducing the number of compression stages.

[0009] A method for processing sensor data is proposed, wherein the sensor data is present as a point cloud composed of individual points, wherein the points are projected into at least two (two-dimensional) grids with different resolutions, wherein points from a partial area of ​​the point cloud are projected into a first grid with a higher resolution and further points of the point cloud are projected into a second grid with a lower resolution, wherein attributes from the first grid are compressed and arranged in the second grid before the attributes from the second grid are compressed.

[0010] The conception of the embodiments of the present invention can be considered to be based in particular on the ideas and findings described below.

[0011] A point cloud can be provided by a radar sensor or a lidar sensor, for example. A point cloud consists of individual points with coordinates and attributes. Each point can, for example, represent a reflection received by the corresponding sensor. Possible attributes of a point include its relative velocity with respect to the sensor.

[0012] The sensor can be assumed to be point-like. However, given the sensor's maximum physical resolution, the theoretically possible point density decreases as the cube of the distance from the sensor. Consequently, the resulting point cloud has a sharply decreasing point density as the distance increases. In a composite point cloud of several different sensors, areas with high point density can be located at various locations.

[0013] When projecting a point cloud into a grid with cells of constant size, many points with a small spacing from the sensor are projected into a single cell, and / or many cells remain empty between the cells into which points with a large spacing from the sensor are projected.

[0014] Here, cells of different sizes are used. In addition, grids of different sizes are used, each covering an area of ​​the point cloud of different sizes.

[0015] In theory, only small regions of the point cloud at a small distance from the sensor have a high point density. In practice, locally close to the sensor, there may be no points or only a small number of points if, for example, there are no reflecting objects. This small region is projected onto a grid that is just large enough to cover essentially only this region. To achieve this, the grid has a high resolution, meaning small cells, so that as few points as possible are projected into the same cell.

[0016] This first grid is compressed, particularly in a neural network, or in other words, the information contained in the grid (i.e., the coordinates and attributes of the points) is compressed. This reduces the amount of data. For example, adjacent cells of the grid are aggregated using rules. The compressed information can then have a reduced resolution.

[0017] The compressed information is then inserted into the next grid at the appropriate location. The next grid has a lower resolution than the first grid. The cells of the next grid are larger than those of the first grid. The next grid has a larger extension scale than the first grid. The area of ​​the point cloud covered by the first grid is contained within the coverage of the next grid. The area of ​​the point cloud covered by the first grid can, for example, be arranged at or near the edge of the next grid.

[0018] The points of the point cloud covered by the next grid have been projected into the cells of the next grid. Therefore, the next grid once again contains the positions and attributes of the points of the point cloud. Points in the area of ​​the first grid can be reprojected into the next grid. The compressed attributes of the first grid are interpolated into the next grid in the area of ​​the first grid.

[0019] The next grid is then also compressed, in particular in a neural network, or the information contained in the next grid (i.e., the coordinates and attributes of the points) is compressed. The amount of data is reduced again, for example, by aggregating adjacent cells of the grid using rules. The compressed information can then have a reduced resolution again.

[0020] This process is repeated until the entire point cloud, or at least a predefined portion of the point cloud, has been projected into the last grid with the lowest resolution, the previous compressed grids are interpolated and the last grid is also compressed.

[0021] Object recognition may then be performed on the compressed attributes.

[0022] The compressed attributes from the previous grid can be aligned with the next grid before being arranged in the next grid. The cells of the next grid can be aligned differently from the cells of the previous grid. For example, the cells can be rotated or shifted. A coordinate transformation can then be performed to achieve the alignment. This alignment preserves the original spatial relationships between the points projected into the different grids.

[0023] The compressed attributes from the previous grid can be scaled to the resolution of the next grid before being arranged in the next grid. The compressed attributes may still have a higher or lower resolution than the cells of the next grid. The compressed attributes can then be stretched or compressed, for example, until they match the cells of the next grid.

[0024] Furthermore, points from a partial area can be projected onto the next grid. When placed into the next grid, the compressed attributes from the first grid can be merged with the uncompressed attributes from the points from the partial area that already exist in the next grid. For example, the attributes can be fused. This prevents loss of uncompressed information. Each cell can be described by a vector of a predefined length. The length of the vector can be preserved during merging.

[0025] During compression, attributes from the previous raster can be compressed down to the resolution of the next raster. The compression stage of the neural network can be configured to output a matching resolution. With matching resolution, attributes from the previous raster can be easily integrated into the next raster.

[0026] The attributes of multiple points can be merged during projection if they are projected into a common cell of the corresponding grid. Attributes can be merged into a vector of a defined length. Empty cells can be filled with a predefined blank vector to simplify compression.

[0027] Points arranged in the near region may be projected into the first grid. Points arranged in the far region may be projected into the larger grid. Points arranged in the middle region may be projected into the middle grid.

[0028] Further points of the point cloud may be projected into another grid having a lower resolution.Before compressing the attributes from the other grid, the attributes from the second grid may be compressed into second compressed attributes and arranged in the other grid.

[0029] The method is preferably computer-implemented and can be implemented, for example, in software or hardware or in a hybrid form of software and hardware, for example in a driver assistance system.

[0030] The solution proposed here also provides a control device in the form of a driver assistance system for a vehicle, wherein the driver assistance system is designed to execute, control or implement the steps of the variant of the method proposed here in a corresponding device.

[0031] A control device or driver assistance system can be an electrical device comprising at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, and at least one interface and / or communication interface for reading or outputting data embedded in a communication protocol. The computing unit can be, for example, a signal processor, a so-called system ASIC, or a microcontroller, which processes sensor signals and outputs data signals based on the sensor signals. The memory unit can be, for example, a flash memory, an EPROM, or a magnetic storage unit. The interface can be configured as a sensor interface for reading sensor signals from a sensor and / or as an actuator interface for outputting data signals and / or control signals to an actuator. The communication interface can be configured for wireless and / or wired data reading or output. The interface can also be a software module, which resides on a microcontroller, for example, along with other software modules.

[0032] Also advantageous is a computer program product or computer program having a program code which can be stored on a machine-readable carrier or storage medium (e.g. a semiconductor memory, hard disk storage or optical storage) and which is used to execute, implement and / or control the steps of a method according to one of the above-described embodiments, in particular when the program product or program is implemented on a computer or device.

[0033] It should be noted that some of the possible features and advantages of the present invention are described herein with reference to different embodiments. Those skilled in the art will recognize that the features of the control device and method can be combined, adapted or exchanged in an appropriate manner to obtain other embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings, wherein neither the drawings nor the description should be construed as limiting the present invention.

[0035] Figure 1 A block diagram illustrating a method according to one embodiment is shown.

[0036] The figures are schematic and not to scale. Identical reference numerals indicate identical or equivalent features. DETAILED DESCRIPTION

[0037] Figure 1A block diagram of a method according to one embodiment is shown. A point cloud 100 is provided by a sensor (e.g., a radar sensor or a lidar sensor). Point cloud 100 is composed of individual points 102, each having coordinates and at least one attribute. Here, points 102 are presented in two dimensions (x and y), but spatial representation is also possible.

[0038] The point cloud 100 is projected into different grids 106 in a grid projection 104 . Here, for simplicity of illustration, the point cloud 100 is projected into only two different grids 106 a and 106 b . However, the point cloud 100 may be projected into a plurality of different grids 106 .

[0039] In this example, a first partial region 108a of point cloud 100 is projected into first grid 106a. First partial region 108a includes a portion of point cloud 100 with a high point density. First grid 106a has a high resolution with a limited extent. This extent corresponds substantially to the size of first partial region 108a. Cells 110 of first grid 106a are small and arranged closely together. Due to the small, densely arranged cells 110, a small number of points 102 are projected into each cell 110. The attributes of points 102 projected together into one cell 110 are aggregated.

[0040] Second sub-region 108b of point cloud 100 is projected into second grid 106b. Second sub-region 108b is larger than first sub-region 108a and completely encompasses first sub-region 108a. In other words, first sub-region 108a is located within second sub-region 108b. For simplicity, second sub-region 108b is assumed to encompass the entire point cloud 100. However, second sub-region 108b may be only slightly larger than first sub-region 108a. Second grid 106b has a lower resolution than first grid 106a. That is, cells 110 of second grid 106b are larger and spaced farther apart than cells 110 of first grid 106a. Consequently, all points 102 of first sub-region 108a are re-projected into cells 110 of second grid 106b. At least in first sub-region 108a, more points 102 are projected into common cells 110 than in the first projection. In regions of point cloud 100 with a lower point density, only a small number of points 102 are again jointly projected into cells 110 of second grid 106 b .

[0041] In one embodiment, the first partial region 108 a covers a near region in front of the sensor, while the second partial region 108 b covers the near region and a far region.

[0042] After grid projection 104, the grid 106 populated with attributes is processed, for example, in a convolutional neural network 112. In a first processing stage 114a, the attributes from the first grid 106a are compressed into first compressed attributes 116a. The first compressed attributes 116a are then inserted into the second grid 106b. The first compressed attributes 116a are inserted into the cells 110 of the second grid 106b corresponding to the first sub-region 108a. The uncompressed attributes already present in these cells 110 are merged with the compressed attributes 116a.

[0043] In the second processing stage 114b, the attributes from the second grid 106b are compressed into second compressed attributes 116b. The first compressed attributes 116a and the fused attributes are recompressed. This process is repeated until at least a predefined portion of the point cloud 100 is compressed to a predefined degree of compression.

[0044] The compressed attributes 116 are then arranged, for example, in a feature pyramid 118 . Then, in the feature pyramid 118 , objects 122 are identified, for example, by a detection head 120 and are marked, for example, by a frame 124 .

[0045] In one embodiment, the compressed attributes 116 are adapted to the next coarser grid 106 of the next processing stage 114 after the respective processing stage 114. To this end, the compressed attributes 116 are, for example, aligned and / or scaled.

[0046] Below, possible configurations of the invention are summarized again or presented in slightly different wording.

[0047] A method is proposed for range adaptation in a multi-scale neural network for object recognition using point clouds.

[0048] Advanced driver assistance systems (ADAS) and autonomous driving (AD) require an accurate representation of the vehicle's surroundings. Many perception sensors, such as LiDAR and radar, represent distant objects in the surrounding environment as a collection of points called a point cloud. Typically, each point represents the 3D position (x, y, z) of a reflection, along with other characteristics such as intensity and measurement quality.

[0049] When identifying objects from point clouds, deep neural networks (DNNs) recognize point patterns and represent possible objects in the surrounding environment using lists of oriented boxes (Bounding Boxes) with presence probabilities and object categories. DNNs require sufficient labeled data collected from different scenes for training. Grid-based DNNs are one of the most commonly used models for object recognition. Inspired by image processing, these methods extract features from each pixel and learn the connections between pixels using, for example, convolutional neural networks (CNNs). To obtain an image-like feature map, the point cloud is first projected onto a predefined grid. The position in the grid depends on the orientation of the point. Starting from a bird's-eye view grid image (BEV) in the xy plane, the area and grid resolution of the BEV are first defined. Points are then assigned to corresponding grid cells based on their orientation in the xy plane. The properties of each grid cell are determined by all the points within it and are then fed into the other layers of the DNN, such as the convolutional backbone network. The convolutional backbone network typically consists of multiple stages that process the data at different scales. Processing input. The outputs of the stages are combined into a feature pyramid, which is then processed in a feature pyramid network and a detection head for further pattern recognition or to extract bounding boxes.

[0050] Grid-based object recognition methods using LiDAR point clouds have been widely studied. Grid-based deep neural networks (such as PointNet, PointNet++, and PointPillars) have achieved promising recognition performance in various benchmarks. Based on the successful examples used for LiDAR recognition, these architectures have been adapted for radar point clouds in multiple studies. Due to the low density of radar points, the attributes of the points may be weakened by multiple stages in the convolutional backbone network, which leads to degraded recognition performance. In addition, grid-based methods often suffer from high computational overhead, especially in long-range detection where a grid map with large range and high resolution is defined.

[0051] In order to improve the recognition performance, it is proposed to project the point cloud into multiple grid maps with different grid cell sizes. The point cloud is projected onto a grid map with a predefined working distance but different grid resolution. The generated feature map is used in the backbone network stage with the corresponding scale (Skalen). In this way, the different stages in the backbone network benefit from directly extracting information from the point cloud, which is not lost in the pyramid structure. Therefore, the architecture with multiple grid maps as input is superior to the architecture with a single grid map as input in terms of recognition accuracy. However, this solution does not solve the problem of computational complexity. On the contrary, more computing time and storage space are required to store and process multi-scale grid projections.

[0052] In the context of grid-based deep neural network object recognition using point clouds, the proposed scheme aims to address the computational complexity of multi-scale grid projections. The scheme adapts the scale of each grid projection based on point density. Only nearby points are projected onto a high-resolution grid and processed, while more distant points are projected onto a larger, lower-resolution grid. This reduces computational complexity because only nearby points are processed at high resolution in the early stages, while more distant points are processed at low resolution in the later stages.

[0053] The input of the object recognition network for object recognition using deep neural networks is a set of unordered points in n dimensions. Each point is characterized by its orientation and other characteristics (such as reflection intensity in LiDAR or radial velocity in radar). To process the point cloud, grid maps with different ranges and resolutions are first defined, and the point cloud is projected into these grid maps, thereby generating multi-range feature maps. These feature maps are fed simultaneously into the convolutional backbone network. In the convolutional backbone network, a method for grid alignment is applied to align the ranges from different stages. After the modules "Feature Pyramid Network" and "Detection Head", the output of the network is a list of oriented bounding boxes, each of which encloses a possible object and contains information about the object type, position and size (length, width, height, orientation).

[0054] Experiments show that while maintaining similar recognition accuracy compared to known methods, this scheme reduces computational complexity in terms of GPU runtime and memory usage. Specifically, this scheme can reduce computer memory requirements. Through range adaptation, only grids within a smaller range are used in the initial phase. Compared to known methods that use a constant full range in each phase, this scheme avoids unnecessarily recognizing large ranges in the early backbone phase, thus saving on memory requirements. Furthermore, the proposed scheme can reduce processing runtime. Since only a small-scale grid graph is processed in the early phase, the computational time of convolution operations on distant grids in the early backbone phase is saved. The proposed scheme also enables high-resolution object detection. Adaptive distance enables high-resolution recognition at a smaller range grid while keeping the computational load low. Recognition accuracy is also maintained. Thanks to multi-scale grid projection, this method maintains high recognition accuracy even when only a small-scale grid graph is used initially.

[0055] The proposed solution offers great advantages and can be used for a large number of products.

[0056] exist Figure 1 In

[15] , an exemplary block diagram of the proposed scheme is shown using a two-dimensional grid projection as an example, which adapts the range of the grid projection and aligns it in the backbone network. The input of the module for range adaptation is the same as the input of the network, that is, a set of n-dimensional unordered points. The module outputs multiple n-dimensional feature maps with different ranges, which are composed of [xmini, xmaxi, ymini, ymaxi, ---]. Here, xmin, xmax, ymin, ymax represent the range, and i represents the i-th feature map, which is then fused in the convolutional backbone network in the corresponding processing stage.

[0057] To generate these feature maps, the module for range adaptation first defines the required range [xmini, xmaxi, ymini, ymaxi, ---] and grid size [csxi, csyi] based on experimental prior knowledge. In this way, for example, the entire perception range can be divided into ranges with small and large ranges based on the point position. Next, the point cloud is projected onto a grid map with a predefined range and grid cell size. If a grid cell contains a point, the attributes of the point are incorporated into the grid cell. If multiple points fall into a grid cell, the attributes of these points are aggregated, for example, by means of a pooling function or PointNet, in which a columnar feature network is used to project the point-by-point aggregated features into a bird's-eye view grid map. To this end, all points located in a grid cell are aggregated into pillars (English: Pillars). The features of each point are individually embedded by a fully connected neural network. In the case where multiple points fall into the same column, all points in the column are mean pooled to obtain a feature vector with a fixed length.

[0058] Next, the multi-stage grid maps, along with the features generated by the columnar feature network, are fed into the corresponding processing stages of the convolutional backbone network, which uses convolutional neural networks to process data at multiple spatial scales and ranges. Each processing stage produces a feature map with the same range as the input. To adjust the feature maps in different stages, a module for grid adjustment is placed between each processing stage. For example, feature map A in the small range has a range of [xmin1, xmax1, ymin1, ymax1], and its grid cell size is [csx1, csy1]. Feature map B in the large range has a range of [xmin2, xmax2, ymin2, ymax2], and its grid cell size is [csx2, csy2]. The size of the grid cell depends on the processing stage of the backbone network. Since the first stage in the convolutional backbone network accepts the grid map in the small range as input, it produces feature map C with a range of [xmin1, xmax1, ymin1, ymax1], but a grid cell size of [csx2, csy2]. The module for grid alignment between the first and second processing stages compares the differences between the ranges and performs, for example, padding of the backbone feature map C to adapt its range to that of the feature map B. More precisely, the number of grids to be padded for each dimension is determined as follows:

[0059]

[0060] The values ​​of the cells added by padding can be set to constant values ​​(e.g., zero) or set in other ways. It should be noted that the padding values ​​are integers so that the grid can be distorted. After the grid is aligned, the feature map B in the large area can be fused with the aligned feature map D for further processing stages. Finally, all aligned feature maps are forwarded to the feature pyramid network and detection head for further processing.

[0061] The architecture shown and described above is a specific implementation that can be modified in many ways. The general basic architecture can be described as follows. The input to the network is an unordered set of points representing a point cloud from LiDAR, radar, or other sensors that produce points as output. Distance adaptation can be extended to applications in n-dimensional space, such as distance adaptation for 3D mesh-based object recognition. The concept of distance adaptation can be extended with other point features, such as distance adaptation based on point intensity and point filtering. The architecture can use any type of processing before distance adaptation. The module for mesh projection can be any type of module that takes an unordered list of points as input and outputs a mesh covering an area with a certain resolution (2D, 3D, or other dimensions). Given a predefined range, the architecture can use any type of processing after range adaptation. A variety of neural networks (NNs) can be used as the backbone and head, including convolutional neural networks (CNNs), Transformer architectures, and others. Output formats other than oriented bounding boxes are also possible, such as semantic segmentation, occupancy grids, or visibility maps.

[0062] The proposed solution can also be used in other fields. For example, it can be used in automated assembly systems, for example to identify components and their orientation to determine gripping points. The method can also be used in automated lawn mowers, for example to identify objects (obstacles) or occupancy. In addition, the method can be used for automatic access control, for example for person recognition and identification to automatically open doors. The method can be used for monitoring squares or buildings, for example for detecting, inspecting, and classifying dangerous goods. The method can be used for traffic monitoring using fixed radar sensors. The method can be used to identify and classify traffic participants in assistance systems for bicycles or other two-wheeled vehicles (motorcycles, mopeds, etc.).

[0063] Finally, it should be pointed out that terms such as "having", "comprising" etc. do not exclude other elements or steps, and terms such as "a" or "an" etc. do not exclude a plurality. Reference signs in the claims should not be considered as limitations.

Claims

1. A method for processing sensor data, wherein: The sensor data is present as a point cloud (100) consisting of individual points (102), wherein the points (102) are projected into at least two grids (106) with different resolutions, wherein points (102) from a partial area (108) of the point cloud (100) are projected into a first grid (106a) with a higher resolution and further points (102) of the point cloud (100) are projected into a second grid (106b) with a lower resolution, wherein the attributes from the first grid (106a) are compressed and arranged in the second grid (106b) before the attributes from the second grid (106b) are compressed.

2. The method according to claim 1, wherein Compressed attributes (116a) from the first grid (106a) are aligned with the second grid (106b) before being arranged in the second grid (106b).

3. A method according to any one of the preceding claims, wherein Compressed attributes (116a) from the first grid (106a) are scaled to a resolution of the second grid (106b) before being arranged in the second grid (106b).

4. A method according to any one of the preceding claims, wherein Furthermore, the points (102) of the partial area (108) are projected into the second grid (106b), and the compressed attributes (116a) from the first grid (106a) are merged with the uncompressed attributes of the points (102) from the partial area (108) that are already present in the second grid (106b) when arranged in the second grid (106b).

5. A method according to any one of the preceding claims, wherein During compression, attributes from the first grid (106a) are compressed to the resolution of the second grid (106b).

6. A method according to any one of the preceding claims, wherein Attributes of a plurality of points (102) are merged during projection if the plurality of points (102) are projected into a common cell (110) of a corresponding grid (106).

7. A method according to any one of the preceding claims, wherein Points (102) arranged in a near region are projected into the first grid (106a), and points (102) arranged in a far region are projected into the second grid (106b).

8. A method according to any one of the preceding claims, wherein Further points (102) of the point cloud (100) are projected into another grid (106n) having a lower resolution, wherein, before the attributes from the other grid (106n) are compressed, the attributes from the second grid (106b) are compressed into second compressed attributes (116b) and arranged in the other grid (106n).

9. A control device, wherein: The control device is configured to carry out, implement and / or control the method according to one of the preceding claims in a corresponding device. 10 . A computer program product, which is configured to guide a processor to carry out, realize and / or control the method according to claim 1 when the computer program product is executed.

11. A machine-readable storage medium having stored thereon the computer program product according to claim 10.

Images