Improved object detection and / or classification for point cloud input data
By combining recurrent neural networks with kernel convolution, the difficulty of detecting unordered point cloud input data is solved, enabling efficient processing and object classification of different point cloud data, and improving the object detection performance of advanced driver assistance systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CONTINENTAL AUTONOMOUS MOBILITY GERMANY GMBH
- Filing Date
- 2026-01-28
- Publication Date
- 2026-07-31
AI Technical Summary
Existing unordered point cloud input data has low performance in object detection, especially in non-uniform scenes depicting multiple object categories. The label of a single point depends on the local neighborhood rather than the global representation, and the identical arrangement of unordered point clouds makes detection difficult.
A recurrent neural network is used to process point cloud data by combining kernel convolution and transform gate. Kernel convolution is used to match the dimension of the input data with the hidden state, and input data of different sizes and dimensions are processed by row-by-row concatenation. Long short-term memory network (LSTM) or gated recurrent unit (GRU) is used to maintain long-term dependencies to process unordered point cloud data obtained by radar sensors.
It improves the performance of object detection, effectively processes point cloud data with different point densities and sequence lengths, enhances the detection and classification capabilities of object categories, and is suitable for object detection in advanced driver assistance systems.
Smart Images
Figure CN122493420A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a computer-implemented method for locating and / or classifying objects based on input data in the form of a point cloud—the input data being related to a scene depicting at least one object—including a radar sensor, a computer program, and a computer-readable (storage) medium, which are implemented as electronic units performing the method. Both the method and the radar sensor are preferably used in advanced driver assistance systems. Background Technology
[0002] Object detection is widely used in various computer vision tasks and refers to the localization and / or classification of objects based on semantic categories, i.e., determining the location and type of an object. Object localization typically involves defining the object's location using bounding boxes. Classification tasks refer to assigning an object to a specific object category based on its unique features.
[0003] There are various techniques available for object detection, including classical techniques as well as techniques based on machine learning models (especially those using neural networks).
[0004] Regarding deep neural networks, on the one hand, several approaches involving single-shot object detection have been proposed. On the other hand, in order to incorporate potential information from the time domain, object detection methods based on recurrent neural networks have also been developed.
[0005] Recurrent neural networks (RNNs) are used to process sequential input data across multiple time steps and include recurrent units with hidden states that represent the form of memory and are updated at each time step based on the current input data and the previous hidden states. This establishes a feedback loop that allows the network to learn from past inputs and incorporate the knowledge gained from past inputs into the processing of the current input. Examples of object detection methods utilizing RNNs are disclosed in EP4365624A1, EP4379580A2, or US11093819B1.
[0006] A prominent application of object detection involves Advanced Driver Assistance Systems (ADAS), which utilize object detection techniques to analyze input data from various environmental sensors, such as radar sensors, lidar sensors, ultrasonic sensors, or cameras. Typical ADAS functions support the driver while they maintain control of the vehicle. On the other hand, depending on the level of automation, semi-autonomous or fully autonomous driving can also be achieved. Exemplary functions of ADAS include methods for identifying various objects in the vehicle environment, methods for determining road lanes or for lane keeping, braking assistance functions, or various parking assistance functions. While the following description will utilize examples of object detection for use in Advanced Driver Assistance Systems, the invention is by no means limited to this specific application. Rather, it is applicable to any neural network-based object detection methods and applications.
[0007] Often, and especially in the case of radar sensors that can be part of driver assistance systems, the recorded environmental data is provided in the form of point clouds. A point cloud consists of multiple points spatially located by their coordinates. Similar points in the point cloud can be identified by observing the local neighborhood of a particular point. Point clouds recorded using sensors—especially radar sensors—are typically provided as unordered point clouds, meaning they consist of a sequence of 3D coordinates, each representing a single point. In contrast, an ordered point cloud would resemble a 2D matrix of input data partitioned according to the spatial relationships between points. A problem associated with unordered point clouds is that all permutations of an unordered point cloud are identical to each other.
[0008] Therefore, when applying meaningful object detection methods, special attention needs to be paid to input data in the form of point clouds, especially unordered point clouds.
[0009] The paper "PointNet: Deep learning on point sets for 3D classification and segmentation," published by CR Qi et al. in the 2017 IEEE / CVF International Conference on Computer Vision and Pattern Recognition (pp. 652-660), proposed a possibility for performing classification and segmentation on unordered point cloud input data. A drawback of the proposed PointNet is that its performance for segmentation tasks may be relatively low because in non-uniform scenes depicting multiple objects of various object categories, the label of a single point depends on its neighborhood, rather than on the global representation of all points included in the point cloud.
[0010] An improvement to PointNet, PointNet++, was proposed by CR Qi et al. in their 2019 conference proceedings, Volume 30, entitled "PointNet++: Deep hierarchical feature learning on point sets in a metric space". PointNet++ addresses the dependency relationship between points and their local neighborhoods.
[0011] In their presentation "Semantic segmentation on radar point clouds" at the 21st International Conference on Fusion of Information (FUSION) in 2018, O. Schumann et al. proposed an adaptation of the PointNet-based architecture to recurrent neural networks, presented on pages 2179-2186. Considering the available latent temporal information is particularly important when processing data acquired from radar sensors, as it significantly enhances detection and classification performance for different object categories.
[0012] Another possibility proposed by H. Thomas et al. utilizes kernel-point convolution, as described in their paper "KPConv: Flexible and Deformable Convolution of Point Clouds" published in the IEEE / CVF International Conference on Computer Vision, pages 6411-6420, 2019. Similar to image convolution, kernel-point convolution can be used as a fundamental building block for implementing neural network architectures that process point cloud input data; specifically, it can be used for classification and / or segmentation tasks based on point cloud input data. A kernel point is a certain number of points contained within a radius that defines a local neighborhood. In the case of a rigid kernel, the points are located at rigid positions within the radius, while in the case of a deformable kernel, the points may be offset. Each kernel point is equipped with a weight matrix for feature mappings about features and structure. Therefore, the set of kernel points is used to define the convolution region, rather than discrete image kernels with fixed positions. Kernel-point convolution works best with relatively high-density point clouds (e.g., point clouds obtained from LiDAR sensors).
[0013] In 2021, F. Nobis et al. proposed an adaptation of this technique for recurrent neural networks, particularly long short-term memory (LSTM) networks, in their article “Kernal pointconvolution LSTM networks for radar point cloud segmentation” published in Applied Science 11(6) on page 2599. Summary of the Invention
[0014] Based on the existing technology, the purpose of this invention is to propose a method for object detection with improved performance, which is capable of processing input data in the form of unordered point clouds.
[0015] This objective is achieved by the computer-implemented method according to claim 1, the radar sensor according to claim 8, the computer program according to claim 9, and the computer-readable (storage) medium according to claim 10.
[0016] The proposed method for detecting and / or classifying objects based on input data in the form of point clouds—which is related to a scene depicting at least one object—includes the following steps:
[0017] - Obtain the current input data sample; and
[0018] - The current input data sample is fed as input to the recurrent neural network, which is implemented to determine the object and / or the object category associated with the object based on the current input data sample.
[0019] According to the present invention, the recurrent neural network includes a transformation gate that is implemented to perform a kernel convolution on the current input data sample such that the current input data sample has the same dimension as the hidden state, the hidden state relating to a recording time earlier than the recording time of the current input data sample.
[0020] This invention relates to processing time series of point clouds obtained at different recording times. Therefore, the input data includes temporal dependencies considered within a recurrent neural network. Thus, a specific input data sample refers to a point cloud recorded at a specific point in time.
[0021] By implementing a transform gate and applying kernel-point convolution to the input data samples, it becomes possible to process input data of different sizes and / or dimensions. The proposed method allows processing of point clouds with different point counts at different recording times, and / or input data with different sequence sizes. Therefore, the process better reflects real-world situations where the available input data provided over time does not have a constant dimension or sequence length.
[0022] In a preferred embodiment of the invention, the number of neighbors used within the kernel convolution is variable. Therefore, the radius selected in the kernel convolution can vary. Through this variation, point clouds with different point densities can be processed.
[0023] According to a preferred embodiment, the input data refers to data obtained from at least one radar sensor. Radar sensors are frequently used to sense the environment and objects in its vicinity. Radar sensors advantageously operate independently of weather conditions or light source availability. When inferring information about objects in the vicinity of the radar sensor, the time dependence or time component of a measurement series over time provides valuable insights, such as information about the radial velocity of points contained in a point cloud. Additionally, the assessment of so-called microDoppler differences is used to distinguish different object categories from one another.
[0024] Another embodiment of the method proposed by this invention includes implementing the concatenation step of the current input data sample and the hidden state as a row-by-row concatenation. This measure further ensures processing of input data samples of different sizes within the recurrent neural network, wherein the current input data sample and the hidden state are concatenated before being processed by other layers of the recurrent neural network. This contrasts with the typical operation of recurrent neural networks performing column-by-column concatenation.
[0025] The splicing step can be performed in a separate module or as part of a conversion gate.
[0026] In another preferred embodiment, the recurrent neural network is a long short-term memory network or a gated recurrent unit.
[0027] The LSTM network was proposed by S. Hochreiter et al. in their 1997 paper "Long Short-Term Memory" published in *Neural Computation 9(8)*, pp. 1735-1780. LSTMs typically consist of cells and gates—namely, input gates, output gates, forget gates, and update gates. Cells serve as memories that hold values for a given time interval, while gates regulate the flow of information into and out of the cell. Forget gates determine which information to discard from previous states; input gates organize which new information should be stored in the current cell state; and output gates control which information in the current cell state should be provided as output. This selective output probability allows LSTMs to maintain useful long-term dependencies.
[0028] Subsequently, K. Cho et al. proposed the Gated Recurrent Unit (GRU) in 2014 in "Learning Phrase Representations using RNNEncoder-Decoder for Statistical Machine Translation," available on arXiv, doi: arXiv:1406.1078. In principle, the GRU is similar to LSTM, but with fewer gates.
[0029] Finally, another embodiment of the proposed method involves a recurrent neural network that is a long short-term memory network, wherein linear transformations are replaced by kernel-point convolutions in at least one gate and / or unit of the long short-term memory network. Therefore, kernel-point convolutions are performed not only in the transformation gates, but also within at least one gate, and preferably within all other gates—i.e., the input gate, forget gate, output gate, and update gate within each unit. It should be noted that subsequent computational steps then need to be adapted at the modified settings of the neural network.
[0030] The method according to the invention is preferably used for object detectors, particularly object detectors implemented in advanced driver assistance systems, which preferably include at least one radar sensor. The object detector can be implemented to determine the location of an object and / or to classify the determined object into a certain object category based on point cloud input data.
[0031] The objective problem under this invention is further solved by a radar sensor implemented to perform the proposed method. The radar sensor may include an electronic unit implemented to perform the method.
[0032] The objective problems under this invention are also solved by: a computer program including instructions that, when executed by a computer, cause the computer to perform the method proposed by this invention; and a computer-readable (storage) medium including instructions executable by at least one processor to perform the method, or storing the computer program thereon. Attached Figure Description
[0033] The invention and its preferred embodiments will be described in more detail with reference to the following figures:
[0034] Figure 1 shows schematic diagrams of (a) a classic recurrent neural network and (b) a recurrent neural network according to a first embodiment of the present invention.
[0035] Figure 2 The recurrent neural network involved in this invention is a second embodiment of the LSTM network.
[0036] In the accompanying drawings, the same elements are always provided with the same reference symbols. Detailed Implementation
[0037] Figure 1a The diagram illustrates a recurrent neural network. RNN1 receives a time series x(t) as input, which is typically combined with the hidden state h(t-1) from the previous time step t-1 via horizontal concatenation 2. Furthermore, for each time step t, the hidden state h(t) and optional output are typically computed by applying a linear transformation and an activation function σ.
[0038] Figure 1b A first embodiment of the present invention is shown. Based on Figure 1a The diagram illustrates a transformation gate 3, in which a kernel-point convolution (KPC) is performed on the current input data sample PC(t) in point cloud form. The KPC is configured such that the current input data sample PC(t) has the same dimension as the hidden state h(t-1), which relates to a recording time t-1 earlier than the current input data sample t, referring to the previous time step t-1. Subsequent stitching steps 3 are adapted accordingly, i.e., they are performed row-by-row or vertically.
[0039] In the embodiment shown, transform gate 3 does not include splicing step 2. However, in other embodiments, transform gate 3 may include both kernel-point convolution (KPC) and splicing step 2.
[0040] Such typical recurrent neural networks often cannot include information about the past beyond one time step. Another problem associated with this setup is the potential for gradient vanishing or gradient exploding.
[0041] Therefore, according to the second embodiment of the present invention, as Figure 2 As shown, RNN1 is provided in the form of an LSTM network, consisting of three gates 4a to 4c (input gate 4b, output gate 4c, and forget gate 4a) and cell 4d. Again, PC(t) refers to the point cloud input data sample at time step t, and h refers to the hidden state. Additionally, C(t) and C(t-1) refer to the cell states at time steps t and t-1, respectively.
[0042] and Figure 1a Similarly, a transformation gate 3 is provided to enable the processing of input data samples PC(t) that are in the form of point clouds and have different dimensions and / or sequence lengths.
[0043] Alternatively, the linear transformations of one or more of gates 4a to 4c and / or unit 4d may be further replaced by kernel-point convolutions, as indicated by the dashed lines.
[0044] If Hadamard multiplication is applied within an LSTM network, this operation needs to be modified by replacing the linear transformation with kernel-point convolution (KPC) in gates 4a to 4c and / or units.
[0045] If the recurrent neural network is replaced by a GRU, the network architecture will need to be modified according to the case of LSTM. Therefore, no separate example of an RNN in the form of a GRU is shown here.
[0046] In summary, this invention effectively solves the problem of different input sizes for input data samples in the form of sequences and point clouds. The input data sample point clouds do not need to have the same size, and complex data preprocessing is no longer required. Furthermore, it allows data sequences with variable sequence lengths to be used as input data samples for RNNs, enabling the use of entire measurement datasets from environmental sensors such as radar sensors for object detection tasks. The proposed network architecture can be trained using existing training methods for available recurrent neural networks.
[0047] Furthermore, multiple kernels can be used within a single kernel convolution, which in turn improves the ability to analyze complex patterns and object shapes contained in the input data samples. Finally, the network size and required computational power can be tuned by selecting the features used: for example, meaningful object detection capabilities can be achieved for radar sensors by analyzing the absolute ground velocity of radar cross-sections and points.
Claims
1. A computer-implemented method for detecting and / or classifying objects based on input data (PC) in the form of a point cloud, said input data (PC) being related to a scene depicting at least one object, and said method comprising the steps of: - Obtain the current input data sample (PC(t)); as well as - The current input data sample (PC(t)) is provided as input to a recurrent neural network (RNN), which is implemented to determine an object and / or the object category associated with the object based on the current input data sample (PC(t)); A further feature of the method is that: The recurrent neural network (RNN) includes a transformation gate (3) which is implemented by performing a kernel point convolution (KPC) on the current input data sample (PC(t)) such that the current input data sample (PC(t)) has the same dimension as the hidden state (h(t-1)), the hidden state relating to a recording time (t-1) earlier than the recording time (t) of the current input data sample (PC(t)).
2. The computer-implemented method according to claim 1, The number of neighbors used within the kernel point convolution (KPC) is variable.
3. The computer-implemented method according to claim 1 or 2, The input data (CP) mentioned above is data obtained from at least one radar sensor.
4. The computer-implemented method according to any one of the preceding claims, The concatenation step (2) of the current input data sample (PC(t)) and the hidden state (h(t-1)) is implemented by performing the concatenation of the current input data sample (PC(t)) and the hidden state (h(t-1)) line by line.
5. The computer-implemented method according to any one of the preceding claims, The recurrent neural network (RNN) mentioned therein is a long short-term memory (LSTM) network or a gated recurrent unit (GRU).
6. The computer-implemented method according to any one of the preceding claims, The recurrent neural network (RNN) therein is a long short-term memory (LSTM) network, and in at least one gate (4a to 4c) and / or unit (4d) of the long short-term memory (LSTM) network, the linear transformation is replaced by kernel point convolution (KPC).
7. Use of the method according to any one of the preceding claims for an object detector, particularly an object detector implemented in an advanced driver assistance system, wherein the driver assistance system preferably includes at least one radar sensor.
8. A radar sensor implemented to perform the method according to any one of claims 1 to 6.
9. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.
10. A computer-readable (storage) medium comprising instructions executable by at least one processor to perform the method according to any one of claims 1 to 6, or having a computer program according to claim 9 stored thereon.