Method for classifying tracked objects

Abstract

The present disclosure provides a method for classifying a tracked object, and in particular, a method for classifying a tracked object in a vehicle environment. The vehicle includes a plurality of radar sensors and a processing device configured to establish a neural network. According to the method, local radar detection results are captured from objects in the vehicle environment by the radar sensor. Based on the local radar detection results, point features and tracker features are determined. The point features are encoded by a point encoding layer of the neural network, and the tracker features are encoded by a trajectory encoding layer of the neural network. Temporal fusion is performed on the encoded point features and the encoded tracker features by a temporal fusion layer of the neural network. The tracked object is classified based on the fused encoded point features and the encoded tracker features by the classification layer of the neural network.

Description

Technical Field

[0001] The present disclosure relates to a method for classifying a tracked object in an environment of a vehicle including a plurality of radar sensors and a processing device configured to establish a neural network. Background Art

[0002] Modern vehicles are usually provided with one or more radar sensors to be able to detect and track objects in the vehicle's environment. Conventional tracker systems that rely on data provided by the vehicle's radar sensors can usually only provide spatial information of objects, such as information about the position and velocity of detected objects.

[0003] In advanced driver assistance systems (ADAS), complex decisions need to be made to correctly navigate a vehicle. For these navigation decisions, the spatial information provided by conventional tracker systems is not sufficient. This is especially the case in the field of autonomous driving.

[0004] For reliable ADAS-based navigation, additional semantic information about the detected objects is important, i.e. information about the type of the detected object. For example, a pedestrian may have a higher priority for automatic braking than some small debris on the road. In this case, detecting and tracking the object is not enough to make an appropriate decision about the vehicle navigation.

[0005] In existing ADAS, simple classification methods can be applied to object perception. For example, a Bayesian classifier can be used, however, it has been shown that the Bayesian classifier does not have reliable performance in some relevant driving situations.

[0006] Furthermore, neural networks have been proposed for providing semantic segmentation based on radar point clouds. Although good performance has been achieved on test data, deep neural networks for providing semantic segmentation on radar point clouds require complex calculations and a high computational workload that exceeds the limitations of the computer systems available in the vehicle. Therefore, passenger car suppliers have mostly avoided implementing such neural networks applied to radar detection results for semantic segmentation so far.

[0007] Therefore, there is a need for a method and system that can provide semantic information of tracked objects in a vehicle environment. Summary of the invention

[0008] In one aspect, the present disclosure relates to a computer-implemented method for classifying a tracked object in an environment of a vehicle, the vehicle comprising a plurality of radar sensors and a processing device configured to establish a neural network. According to the method, local radar detection results are captured from objects in the environment of the vehicle by the radar sensor. Point features and tracker features are determined based on the local radar detection results. The point features are encoded by a point encoding layer of the neural network, and the tracker features are encoded by a trajectory encoding layer of the neural network. The encoded point features and the encoded tracker features are temporally fused by a temporal fusion layer of the neural network, such as by a recurrent neural network. The tracked object is classified by a classification layer of the neural network based on the fused encoded point features and encoded tracker features.

[0009] Since many modern vehicles are already equipped with radar sensors and means for processing radar detection results and for tracking detected objects, means and processing units for determining point features and tracker features may also already be available in such vehicles. Therefore, the method according to the present disclosure can be easily implemented in existing systems and merged with existing tracking processes. Therefore, the method can be performed using hardware components that are already available on the vehicle and the method according to the present disclosure can therefore be implemented at low cost.

[0010] Furthermore, the neural network is run individually on each track of the detected object. Therefore, the method only considers the local radar detection results and does not require global information of the environment. This again reduces the difficulty and cost of implementing the method according to the present disclosure.

[0011] The neural network may include fully connected layers, wherein the point encoding layer, trajectory encoding layer, temporal fusion layer, and classification layer may each form a separate layer group of the neural network. For each group of these layers, a predetermined minimum number of layers and a predetermined maximum number of layers may be set, such as two to four layers. If there are too many layers, the neural network will not be able to run in real time. On the other hand, if the number of layers is too small, the network may not be able to perform proper classification.

[0012] Since the maximum number of layers of the entire neural network is limited, the requirements regarding memory and computing performance are much lower than, for example, for deep neural networks. Therefore, the method according to the invention is suitable for use on standard computing systems in passenger cars.

[0013] These point features may include the radar cross section of each radar detection result, the distance of each radar detection result relative to the center of the track (i.e., relative to the center of mass of the track), and the compensated Doppler value, i.e., the relative speed of the corresponding radar detection result relative to the vehicle, where the speed may be initially determined relative to the corresponding radar sensor and thereafter compensated for the motion of the vehicle. For a time series of a certain number of subsequent radar detection results, for example, for ten subsequent detection results, the point features may be determined. Tracker characteristics may include the change in speed for each track, the length of each track, the speed of each track in the vehicle coordinate system, the tangential acceleration of each track, the change in heading direction of each track (i.e., the change in direction of the velocity vector), the absolute value of the curvature of each track, and the absolute difference in pointing direction between two adjacent scans of the track in the world coordinate system.

[0014] In order to classify the tracked objects, classes of objects may be predefined, wherein the classes may be labeled, for example, as "car" (i.e., small vehicle), "truck / bus" (i.e., large vehicle), "pedestrian", "motorcycle", "bicycle", and "background", the background referring to unknown objects that may not be of interest. However, the classes are not limited to these examples, and the predefinition of the classes may be performed flexibly to adapt the method to the requirements within the corresponding vehicle, i.e., within the corresponding advanced driver assistance system (ADAS).

[0015] Due to the classification of the tracked objects, the method is able to provide semantic information about the detected objects in addition to the spatial information provided by standard tracking methods. Due to the additional semantic information that can be provided to the vehicle's advanced driver assistance system, the navigation of the vehicle in general can be improved, especially with regard to the navigation decisions to be taken.

[0016] The method may include one or more of the following features:

[0017] The point encoding layer of the neural network can encode each point feature separately, and then the feature pooling layer of the neural network can perform feature pooling. Feature pooling can include maximum pooling or average pooling.

[0018] The temporal fusion layers of the neural network may form a recurrent neural network. The recurrent neural network may be configured as a gated recurrent unit or a long-short term memory. Each layer of the neural network may include a predetermined maximum number of neurons.

[0019] Before determining the point features based on the local radar detection results, a predetermined number of local radar detection results may be sampled.Performing temporal fusion may include performing a hidden state reset.

[0020] According to one embodiment, the point encoding layer of the neural network can encode each point feature separately, and thereafter, the feature pooling layer of the neural network can perform feature pooling. Feature pooling can include maximum pooling or alternatively average pooling. Due to feature pooling, features from different points related to the radar detection results or features from different point features can be combined, which can support the classification task of the neural network.

[0021] The time fusion layer of the neural network can form a recurrent neural network (RNN), which can be configured as a gated recurrent unit (GRU) or alternatively as a long short-term memory (LSTM). Since the recurrent neural network (RNN) has the connection between paths and the connection over time (i.e., feedback) as an inherent characteristic, when using a recurrent neural network, the time fusion or "over time" fusion of the encoded point features and tracker features can be automatically performed. The internal feedback of the recurrent neural network can further support the classification task of the method. Compared with the long short-term memory (LSTM), the gated recurrent unit (GRU) generally requires fewer parameters, and therefore requires less memory and computational workload. On the other hand, the long short-term memory (LSTM) can increase the accuracy and reliability of the neural network and the method.

[0022] Each layer of the neural network may include a predetermined maximum number of neurons. For example, the maximum number of neurons in each layer may be limited to 256. Due to this limitation, the real-time capability of the neural network and the entire method may be ensured.

[0023] Before determining the point features based on the local radar detection results, a predetermined number of local radar detection results may be sampled. The predetermined number may be, for example, ten detection results. Due to the minimum number required for determining the point features, the reliability of the determined point features may be improved.

[0024] Performing temporal fusion may include performing a hidden state reset within the neural network. The reset may be performed periodically, for example every 20 to 80 radar scans or every 1 to 4 seconds. Due to this reset, the noise remaining in the hidden state of the temporal fusion layer may be reduced, which in turn may improve the quality of the classification task. However, when performing temporal fusion of encoded features, alternatively, no reset may be performed.

[0025] In another aspect, the present invention relates to a system for classifying a tracked object in an environment of a vehicle. The system includes: a plurality of radar sensors configured to capture local radar detection results from objects in the environment of the vehicle; a point feature module configured to determine point features based on the local radar detection results; a tracker feature module configured to determine tracker features based on the local radar detection results; and a processing device configured to establish a neural network. The processing device includes: a point encoding module, which includes a point encoding layer of the neural network that encodes the point features; a trajectory encoding module, which includes a trajectory encoding layer of the neural network that encodes the tracker features; a time fusion module, which includes the time fusion layer of the neural network that temporally fuses the encoded point features and the encoded tracker features; and a classification module, which includes a classification layer of the neural network, the classification layer classifying the tracked object based on the fused encoded point features and the encoded tracker features.

[0026] As used herein, the term module may refer to, be part of, or include an application specific integrated circuit (ASIC), an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared, dedicated, or memory group) that executes code; other suitable components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip. The term module may include a memory (shared, dedicated, or memory group) that stores code executed by a processor.

[0027] In summary, the system according to the present disclosure comprises a radar sensor, a point feature module, a tracker feature module and a processing device, the processing device comprising four additional modules installed in a vehicle and configured to perform the steps for the corresponding method as described above. Therefore, the benefits, advantages and disclosures described above for the method are also applicable to the system according to the present disclosure.

[0028] According to an embodiment, the point encoding module may include an additional layer of the neural network that performs feature pooling. Feature pooling may facilitate classification of tracked objects. The point feature module may include a flexible list of point features, and the tracker feature module may include a flexible list of tracker features. Due to the flexible feature lists, the system may be adaptable to existing and future tracking devices.

[0029] In another aspect, the present disclosure relates to a computer system configured to perform several or all of the steps of the computer-implemented methods described herein.

[0030] The computer system may include a processing unit, at least one memory unit and at least one non-transitory data storage. The non-transitory data storage and / or the memory unit may include a computer program for instructing the computer to perform several or all steps or aspects of the computer-implemented method described herein.

[0031] In another aspect, the present disclosure relates to a non-transitory computer-readable medium comprising instructions for performing some or all steps or aspects of the computer-implemented methods described herein. The computer-readable medium can be configured as: an optical medium, such as a compact disc (CD) or a digital versatile disc (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid-state drive (SSD); a read-only memory (ROM); a flash memory; etc. In addition, the computer-readable medium can be configured as a data storage accessible via a data connection such as an Internet connection. The computer-readable medium can be, for example, an online data repository or cloud storage.

[0032] The present disclosure is also directed to a computer program for instructing a computer to perform several or all steps or aspects of the computer-implemented method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Exemplary embodiments and functions of the present disclosure are described herein in conjunction with the following schematically illustrated drawings:

[0034] Figure 1 An overview of a system according to the present disclosure included in a vehicle is depicted, and

[0035] Figure 2 Details of systems and methods according to the present disclosure are described. DETAILED DESCRIPTION

[0036] Figure 1 Schematically shown is a vehicle 11 comprising a plurality of radar sensors 13, point signature modules 15, tracker signature modules 17 and processing means 19. The radar sensors 13, point signature modules 15 and tracker signature modules 17 may have been incorporated in a known modern vehicle 11 which is equipped with a system configured to track objects in the environment of the vehicle 11.

[0037] However, the processing device 19 is configured to perform the method according to the present invention as described below. Therefore, the processing device 19 includes a point encoding module 21, a trajectory encoding module 23, a time fusion module 25 and a classification module 27. Through these modules, the processing device 19 is not only able to track objects in the environment of the vehicle 11 spatially, but also to provide semantic information about the type of tracked objects.

[0038] The radar sensor 13 includes four short-range radars. Two of these radars are mounted on the left and right sides of the front of the vehicle, respectively, and the other two radars are mounted on the left and right sides of the rear of the vehicle, respectively. However, the use of the system and method according to the present disclosure is not limited to a radar system comprising four short-range radars. As an alternative, a system comprising only two radars mounted on the left and right sides of the front of the vehicle, or a system comprising six short-range radars (two additional radars on the left and right sides of the middle of the vehicle, respectively) can be used. In addition, there is no restriction on the type of sensor used in the radar system. The radar sensor 13 is typically part of a centralized tracking system that can be found in many modern vehicles.

[0039] Such a centralized tracking system generally includes a tracker feature providing module 17, which provides tracker features 18 (see Figure 2 ) as output. For the system of this embodiment, Ft tracker features are provided by the tracker feature module 17, where for this example, Ft is 7. The tracker features 18 include the change in velocity of the track (i.e., the velocity of the object tracked by the radar sensor 13), the length of the track, the velocity of the track in the vehicle coordinate system (VCS), the tangential acceleration of the track, the change in heading direction of the track, the absolute value of the curvature of the track, and the absolute difference in pointing direction between two adjacent scans of the track in the world coordinate system (WCS). The heading direction refers to the direction of the velocity vector of the detected or tracked object. Similarly, the pointing direction refers to the orientation of the body frame of the track, and the absolute difference in pointing direction refers to the absolute difference in pointing directions from two adjacent radar scans.

[0040] In addition, the centralized tracking system of the vehicle 11 includes a point feature module 15, which outputs point features 16 (see Figure 2 ). There are Fp point features 16, where for this embodiment, Fp is 3, and the point feature 16 includes the radar cross section (RCS) of the current radar detection result of the radar sensor 13, the distance of the detection result relative to the center of the track (i.e., relative to the center of mass of the track), and the compensated Doppler value, i.e., the relative speed of each detection result relative to the vehicle 11. In practice, the Doppler value is first determined relative to each radar sensor, and then compensated for the self-motion of the vehicle 11.

[0041] For this example, for a sequence of ten subsequent detection results, each of the three point features 16 is determined to provide a set of point features 16 as input to the point feature encoding module 21, i.e., as output of the point feature module 15. In other words, the radar detection results of the radar sensor 13 are "upsampled" until the number of measured detection results reaches 10.

[0042] Figure 2The modules of the processing device 19 and the method steps performed by these modules are described in more detail. The point encoding module 21 and the trajectory encoding module 23 receive point features 16 and tracker features 18 as input from the point feature module 15 and the tracker feature module 17, respectively. Specifically, Fp×np point features 16 (for this embodiment, Fp×np is 3×10) and Ft tracker features 18 (for this embodiment, Ft is 7) are provided.

[0043] The point encoding module 21 comprises two fully connected layers 29 of a neural network, which are used for point encoding, i.e. applied to each point feature 16. For the present embodiment, the first of these two layers 29 comprises pC1 neurons (pC1=8 in the present example), and the second layer comprises pC2 neurons (pC2=32 in the present example) for encoding each point feature. In the point encoding module 21, each point feature is encoded separately, so the point encoding module 21 comprises nP groups of corresponding fully connected layers 29 for point encoding, where nP is the number of detection results to be considered and is equal to 10 for the present embodiment. In general, as is known in the field of neural networks, the neurons of the fully connected layer comprise a plurality of weights W and biases b. That is, the output of the neurons of the layer is obtained by applying a nonlinear function to Wx+b, where x is the corresponding input vector of the layer, which is multiplied by the weight matrix W.

[0044] Therefore, for point encoding, all fully connected layers 29 output a total of pC2×nP encoded point features, i.e., a total of 320 for this embodiment. These encoded point features are used as input to another fully connected layer 31 for feature pooling. Specifically, these layers 31 perform maximum pooling to merge features from different points or point features 16. The mechanism of maximum pooling is known in the art. As an alternative, average pooling can be used. Finally, the point encoding module 21 outputs pC2 encoded point features to which maximum pooling has been applied (for this example, pC2=32).

[0045] Similarly, Ft tracker features 18 are encoded by two fully connected layers 33 for tracker encoding (for this example, Ft=7). For this embodiment, the first layer includes tC1=16 neurons and the second layer includes tC2=32 neurons for tracker encoding. Therefore, the trajectory encoding module 23 outputs tC2 encoded tracker features, and for this embodiment, tC2 is 32.

[0046] The outputs of the point encoding module 21 and the trajectory encoding module 23, i.e., the encoded point and tracker features, are provided as inputs to the temporal fusion module 25. The temporal fusion module 25 first performs a concatenating step 35, in which the encoded point features and the encoded tracker features are concatenated to form a common tensor, which is passed to the gated recurrent unit 37 of the temporal fusion module 25.

[0047] The gated recurrent unit 37 is a special type of recurrent neural network (RNN) in which the input information is not simply passed layer by layer, but in which the information provided as input runs in a loop and also takes into account information from previous inputs. That is, in a recurrent neural network, information from the recent past is added to information from the present, which is actually received layer by layer. In addition, similar to the long-short term memory (LSTM), the gated recurrent unit (GRU) 37 is an extension of the RNN. In addition to the internal feedback of the recurrent neural network, the LSTM also includes three gates, an input gate, a forget gate, and an output gate. The gated recurrent unit 37 used in the present embodiment is similar to the LSTM, but lacks an output gate. Therefore, in terms of the demand for memory, the gated recurrent unit (GRU) 37 is lower than the LSTM.

[0048] The gated recurrent unit (GRU) 37 comprises gC so-called hidden states (for this embodiment, gC=32), which comprises a temporal feedback or loop as described above. Thus, the cascaded input data of the GRU 37 are fused over time, i.e., after nP=10 detection results have passed through the radar sensor 13 (see Figure 1 ) are fused over the sampled time period.

[0049] In addition, the time fusion module 25 performs a reset of the hidden states of the GRU 37 every 60 tracker scans (e.g., every three seconds). These resets are performed to avoid noise accumulation in the memory of the GRU 37 by setting all hidden states within the GRU 37 to zero. Typically, such a reset may be performed every 20 to 80 scans, e.g., every 1 to 4 seconds. The output of the time fusion module 25, i.e., the output of the GRU 37, is provided as an input to the classification module 27.

[0050] The classification module 27 includes two fully connected classification layers 39. For the classification module or classification head 27, nC=6 categories are predefined in order to classify the tracked object. For the vehicle 11, these six categories are labeled as "car" (i.e., small vehicle), "truck / bus" (i.e., large vehicle), "pedestrian", "motorcycle", "bicycle" and "background", the background referring to unknown objects that may not be of interest. However, the definition of categories is not limited to these examples. For this example, the classification module 27 receives gC=32 input data values, and there are oC=32 neurons in the first layer and nC=6 neurons in the second layer of the classification head. That is, the classification module 27 provides a classification output 41, which associates the trajectory of the detected object with one of the six categories listed above. Therefore, in addition to the spatial information provided by the tracking system, semantic information, i.e., about the type of tracked object, is also provided by the method according to the present disclosure.

[0051] The method and system modules as described above are applied to each track (i.e., to each detected object) separately. However, the method and system can be applied to multiple tracks and objects in the environment of the vehicle 11 at the same time. In addition, the system and method according to the present disclosure require a small number of layers for the entire neural network, and a small number of neurons for each layer. Therefore, the system of the present disclosure can be considered "lightweight" and is therefore suitable for running on the ECU of an ordinary passenger car, such as on the Aurix platform.

[0052] Reference numerals list

[0053] 10.11 Vehicles

[0054] 13 Radar Sensor

[0055] 15-point feature module

[0056] 16-point features

[0057] 17Tracker Feature Module

[0058] 18Tracker Features

[0059] 19 Processing device

[0060] 21-point coding module

[0061] 23 track encoding module

[0062] 25 Time Fusion Module

[0063] 27 Classification Modules

[0064] 29 fully connected layers for point encoding

[0065] 31 fully connected layers for feature pooling

[0066] 33 fully connected layers for tracker encoding

[0067] 35 Cascade Steps

[0068] 37Gated Recurrent Unit (GRU)

[0069] 39 classification layers

[0070] 41 Classification Output

Claims

1. A computer-implemented method for classifying a tracked object in an environment of a vehicle (11), the vehicle (11) comprising a plurality of radar sensors (13) and a processing device (19) configured to establish a neural network, the method comprising the steps of: - capturing local radar detection results from objects in the environment of the vehicle (11) by means of the radar sensor (13), - determining point features (16) based on the local radar detection results, - determining a tracker characteristic (18) based on said local radar detection result, - encoding the point features (16) by a point encoding layer (29) of the neural network, - encoding the tracker features (18) by a trajectory encoding layer (33) of the neural network, - concatenate the encoded point features and the encoded tracker features to form a common tensor, - passing the common tensor to a recurrent neural network, the recurrent neural network providing a temporal fusion layer in the form of a neural network to temporally fuse the encoded point features and the encoded tracker features through the temporal fusion layer, and - Classifying the tracked object based on the fused encoded point features and the encoded tracker features by a classification layer (39) of the neural network.

2. The method according to claim 1, wherein: The point encoding layer (29) of the neural network encodes each point feature (16) separately, and then the feature pooling layer (31) of the neural network performs feature pooling.

3. The method according to claim 2, wherein: The feature pooling includes maximum pooling.

4. The method according to claim 2, wherein: The feature pooling includes average pooling.

5. The method according to any one of claims 1 to 4, wherein: The recurrent neural network is configured as a gated recurrent unit (37).

6. The method according to any one of claims 1 to 4, wherein: The recurrent neural network is configured as a long short-term memory.

7. The method according to any one of claims 1 to 4, wherein: Each layer of the neural network includes a predetermined maximum number of neurons.

8. The method according to any one of claims 1 to 4, wherein: Before determining the point features (16) based on the local radar detection results, a predetermined number of local radar detection results are sampled.

9. The method according to any one of claims 1 to 4, wherein: The step of performing temporal fusion includes performing hidden state reset.

10. A system for classifying a tracked object in an environment of a vehicle (11), the system comprising: - a plurality of radar sensors (13) configured to capture local radar detections from objects in the environment of the vehicle (11), - a point feature module (15) configured to determine point features (16) based on the local radar detection results, - a tracker signature module (17) configured to determine a tracker signature (18) based on the local radar detection results, and - processing means (19) configured to establish a neural network, said processing means (19) comprising: a point encoding module (21), comprising a point encoding layer (29) of the neural network for encoding the point features (16), a trajectory encoding module (23) comprising a trajectory encoding layer (33) of the neural network encoding the tracker features (18), a temporal fusion module (25) configured to concatenate the encoded point features and the encoded tracker features to form a common tensor, and pass the common tensor to a recurrent neural network, the recurrent neural network providing a temporal fusion layer of the neural network, the temporal fusion layer temporally fusing the encoded point features and the encoded tracker features; and A classification module (27) includes a classification layer (39) of a neural network, which classifies the tracked object based on the fused encoded point features and the encoded tracker features.

11. The system according to claim 10, wherein: The point encoding module (21) includes an additional layer of the neural network for feature pooling.

12. The system according to claim 10 or 11, wherein: The point feature module (15) includes a flexible list of the point features (16), and the tracker feature module (17) includes a flexible list of the tracker features (18).

13. A computer system comprising a processing unit, at least one memory unit and at least one non-transitory data storage, wherein the at least one non-transitory data storage and / or the memory unit comprises instructions for executing the computer-implemented method of any one of claims 1 to 9.

14. A non-transitory computer-readable medium comprising instructions for executing the computer-implemented method of any one of claims 1 to 9.

Images