Real-time target detection, tracking and motion prediction method for automatic driving

Abstract

The invention discloses a real-time target detection, tracking and motion prediction method for automatic driving, and belongs to the technical field of automatic driving. The method mainly comprises the following steps: 1, receiving three-dimensional point cloud data and corresponding two-dimensional image data; 2, extracting feature representation of the point cloud and the image; 3, performing multi-modal information fusion on the image features and the point cloud features by adopting a step-by-step fusion mode to obtain self-adaptive fused voxel features; 4, according to the self-adaptive fused voxel features, a target detection output head, a behavior tracking output head and a behavior prediction output head are adopted for analysis, and a detection result, a tracking result and a prediction result of the moving objects around the vehicle are obtained. According to the method, a single convolutional neural network is utilized, and three-dimensional point cloud data and corresponding two-dimensional image information are combined, so that efficient and accurate three-dimensional object detection, tracking and motion prediction are realized. Through a unified network architecture, traditional multi-stage processes are integrated into one model, and the efficiency and performance of the system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a real-time target detection, tracking and motion prediction method for autonomous driving. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, autonomous driving technology is gradually becoming a reality. Autonomous driving systems need to accurately perceive and understand various objects in the surrounding environment in order to make accurate decisions and plans. Among them, target detection, tracking and prediction technologies are crucial components of autonomous driving systems. Object detection refers to accurately locating and identifying objects of different categories in images or videos. In autonomous driving, target detection technology is used to detect and identify various traffic participants on the road, such as vehicles, pedestrians, and bicycles. Traditional target detection methods are mainly based on manually designed features and classifiers, but these methods have challenges in accuracy and robustness in complex scenarios. Object tracking refers to continuously tracking the position and motion of target objects in video sequences. In autonomous driving, target tracking technology is used to track the motion trajectories of other traffic participants such as vehicles, pedestrians, or bicycles. Target tracking algorithms need to address challenges such as target appearance changes, occlusion, and camera motion. Target prediction refers to predicting the future position and behavior of a target object based on its historical trajectory and current state. In autonomous driving, target prediction technology is used to predict the trajectory and intention of other traffic participants such as vehicles, pedestrians, or bicycles. Accurate target prediction is essential for autonomous driving systems to make safe and efficient decisions.

[0003] Autonomous driving target detection, tracking, and prediction technologies are key to achieving safe and efficient autonomous driving systems. With the continuous advancement of deep learning and computer vision technologies, modern autonomous driving systems can perceive and understand the surrounding environment more accurately, and achieve real-time detection, tracking, and prediction of traffic participants. The development of these technologies provides important support for achieving safer and more efficient autonomous driving traffic. However, autonomous driving target detection, tracking, and prediction still face some challenges. First, there are a large number of factors such as occlusion, posture changes, and illumination changes in complex traffic scenes, which puts forward requirements for the accuracy of target detection and tracking. Second, the diversity and uncertainty of targets make it more difficult to accurately predict the future behavior of targets. In addition, real-time performance is an important consideration in autonomous driving systems, requiring target detection, tracking, and prediction algorithms to run in real time at high frame rates.

[0004] To overcome these challenges, researchers continue to propose innovative methods and technologies. The rapid development of deep learning technology has brought breakthroughs in target detection, tracking, and prediction. Through end-to-end training, deep neural networks can automatically learn feature representations and patterns, improving the accuracy and robustness of target detection and tracking. In general, the continuous advancement of autonomous driving target detection, tracking, and prediction technology provides key support for achieving safer and more efficient autonomous driving transportation. With the continuous development of technologies such as deep learning, multimodal information fusion, and hardware acceleration, we can expect autonomous driving systems to make greater breakthroughs in future developments and achieve safer and more efficient road travel. Summary of the invention

[0005] To overcome the above problems, the present invention provides a method and system for real-time end-to-end three-dimensional detection, tracking and motion prediction.

[0006] The technical solution adopted by the present invention is as follows:

[0007] In one aspect, the present invention provides a real-time target detection, tracking and motion prediction method for autonomous driving, comprising the following steps:

[0008] Step 1): Collect camera image data and lidar point cloud data around the vehicle through cameras and lidar;

[0009] Step 2): extracting image features and point cloud features based on camera image data and lidar point cloud data respectively;

[0010] Step 3): adopt a step-by-step fusion method to perform multi-modal information fusion on image features and point cloud features to obtain adaptively fused voxel features;

[0011] Step 4): Based on the adaptively fused voxel features, the target detection output head, behavior tracking output head and behavior prediction output head are used to analyze and obtain the detection results, tracking results and prediction results of the moving objects around the vehicle.

[0012] Furthermore, the step 2) includes: taking the camera image data as input, generating image features through an image feature extraction network; normalizing the lidar point cloud data to voxel space, and then generating point cloud features through a point cloud feature extraction network.

[0013] Further, in step 3), the multimodal information fusion includes:

[0014] The corresponding adaptive fusion weights are generated from the image features and the point cloud features, and the generated adaptive fusion weights are numerically normalized; the normalized adaptive fusion weights are used to weight the sum of the image features and the point cloud features to obtain the adaptively fused voxel features.

[0015] Furthermore, the step 4) includes: based on the adaptively fused voxel features, using a target detection output head to detect the position of the three-dimensional object in real time and output it, wherein the detection result is a series of detection frames, and the position of each detection frame is the position information of the three-dimensional object in the detection frame; using a behavior tracking output head to correlate and match the position information of all three-dimensional objects in the detection results of the current moment and the past moment, to obtain the position information of the same three-dimensional object at different moments, to form the motion trajectory of the object and output it; based on the motion trajectory obtained by the behavior tracking output head, using a behavior prediction output head to predict the position information of the object at the next moment and output it.

[0016] On the other hand, the present invention also provides a real-time target detection, tracking and motion prediction system for autonomous driving, which is used to implement the above method.

[0017] Beneficial effects of the present invention:

[0018] (1) Real-time performance: This method can perform target detection, tracking, and motion prediction at real-time speed. This is crucial for autonomous driving systems because objects in the surrounding environment need to be accurately and timely perceived and predicted during driving. By integrating these tasks in a single network, the processing speed can be increased and the demand for computing resources can be reduced, thus achieving real-time performance.

[0019] (2) End-to-end training: This method uses an end-to-end training approach to unify the target detection, tracking, and motion prediction tasks into one network for training. This can fully utilize the spatiotemporal continuity and semantic information of the data to improve the accuracy and robustness of 3D object perception and prediction. At the same time, end-to-end training reduces the information transfer and error accumulation between multiple independent steps and processes, simplifying the system design and optimization process.

[0020] (3) Comprehensive performance: This method can comprehensively complete target detection, tracking and motion prediction tasks, and output the object's position, posture, motion state and future motion prediction results. This comprehensive performance method can provide more comprehensive scene understanding and prediction capabilities, enabling the autonomous driving system to more accurately perceive and predict moving objects in the surrounding environment, thereby improving driving safety and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a framework diagram of the real-time target detection, tracking and motion prediction method for autonomous driving proposed in the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described below with reference to the accompanying drawings. The accompanying drawings are only schematic illustrations of the present invention, and some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities, and these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0023] In the present invention, variable subscripts "img" and "pc" are used to represent camera image data and lidar point cloud data, respectively.

[0024] like Figure 1 As shown, the specific implementation steps of the real-time target detection, tracking and motion prediction method for autonomous driving of the present invention include:

[0025] Step (1). Inputting multimodal data collected by the camera and the laser radar about the vehicle's surrounding environment includes the following steps:

[0026] Step (1.1). Input the image data I collected by the camera img ;

[0027] Step (1.2). Input the point cloud data collected by the LiDAR pc .

[0028] Step (2). After operations such as convolution and pooling, feature representations of point clouds and images are extracted, including the following steps:

[0029] Step (2.1). Through the convolution-based image feature extraction network Φ img Generate image features F img :

[0030] F img =Φ img (I img )

[0031] Step (2.2). For point cloud data, because its disorder and irregularity are not conducive to the processing of neural networks, it is necessary to first normalize the irregular point cloud to voxel space Then through the point cloud feature extraction network Φ based on sparse convolution pc Generate point cloud features:

[0032] F pc =Φ pc (V(I pc ))

[0033] Among them, the image features generated in step (2.1) H×W corresponds to the original image resolution after trimming; the point cloud features generated in step (2.2) X×Y×Z corresponds to the horizontal and vertical directions of the three-dimensional space, which can be mapped to a real space range of 102.4m×102.4m×8m. The feature dimension C of both is set to 256; Represents normalizing the point cloud to voxel space.

[0034] In order to avoid using 3D convolution on point cloud data and the computational complexity it introduces, the point cloud feature F is also extracted in the feature extraction stage. pc The vertical dimension Z is compressed to 1, so that each frame of the point cloud feature is represented as a three-dimensional tensor, so that a unified network can be used for multimodal information fusion.

[0035] Step (3) adopts a step-by-step fusion method to gradually fuse multimodal information together through a multi-layer convolutional network. Specifically, the corresponding adaptive fusion weights are generated from the features of each modality. The features of each modality are multiplied by the corresponding adaptive fusion weights and then added to obtain the adaptive fusion feature F fus , including the following steps:

[0036] Step (3.1). Based on the image feature F img Generate the corresponding image feature adaptive fusion weight α img :

[0037] α img =Θ img (F img )

[0038] Step (3.2). From the point cloud feature F pc Generate the corresponding point cloud feature adaptive fusion weight α pc :

[0039] α pc =Θ pc (F pc )

[0040] Step (3.3). The fusion weight α generated from the above steps (3.1) to (3.2) img and α pc Normalize the values:

[0041]

[0042] Among them, Θ img (.) and Θ pc (.) is a multi-layer convolutional network of images and point clouds, [.] represents a matrix, σ is a normalization function, which can be implemented using the Softmax function;

[0043] Step (3.4). The characteristics F of the two modes img and F pc And the corresponding normalized adaptive fusion weight and Multiply and add to obtain the adaptive fused voxel feature F fus :

[0044]

[0045] The fused voxel features and F img 、F pc With the same feature dimension C and resolution W×H, it can flexibly adapt to the increase and decrease of the number of sensors.

[0046] Step (4). Based on the image feature F obtained in step (3) img and point cloud features F pc The multimodal features composed of the target detection output head, behavior tracking output head and behavior prediction output head are used to infer the corresponding results, specifically:

[0047] Step (4.1). Fusion feature F fus Input the target detection output head and get the detection result O det :

[0048]

[0049] Step (4.2). Fusion feature F fus Input the behavior tracking output header and get the tracking result O tra :

[0050]

[0051] Step (4.3). Fusion feature F fus Input the behavior prediction output head and get the prediction result O pre :

[0052]

[0053] Among them, the detection head is responsible for detecting the existence and position of the three-dimensional object in the input point cloud data and outputting the bounding box of the object. The tracking head uses the detection results and historical information to continuously track the motion trajectory of the object through association and matching. Specifically, according to the feature vectors of the detection frames at the current moment and the previous moment in the corresponding adaptively fused voxel features, the feature matching matrix is ​​calculated, and the detection results at the current moment are matched with the motion trajectory at the past moment based on the feature matching matrix, and the motion trajectory of the three-dimensional object is updated according to the matching results. The prediction head uses the tracking information and historical trajectory to predict the future motion state and trajectory of the object. In this embodiment, each output head is implemented using a linear layer neural network.

[0054] The present invention realizes efficient and accurate three-dimensional object detection, tracking and motion prediction by inputting three-dimensional point cloud data and two-dimensional image data into a single convolutional neural network. The network uses deep learning technology to extract feature representations through operations such as convolution and pooling, and outputs the object's position, posture, motion state and future motion prediction results through regression and classification modules. Compared with the traditional multi-stage process, the method and system of the present invention have higher efficiency and accuracy. Through an end-to-end approach, information transmission and error accumulation between multiple independent steps are eliminated, while reducing computing and storage overhead. The method is suitable for real-time application fields such as autonomous driving, robot navigation and intelligent transportation systems.

[0055] In this embodiment, a real-time target detection, tracking and motion prediction system for autonomous driving is also provided, which is used to implement the above-mentioned embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible.

[0056] Data acquisition module, used to collect image data and lidar point cloud data around the vehicle;

[0057] Feature extraction module, used to extract image features and point cloud features;

[0058] The multimodal fusion module is used to fuse the multimodal information of image features and point cloud features and output adaptive fused voxel features;

[0059] The detection, tracking and prediction module is used to obtain the detection results, tracking results and prediction results of the moving objects around the vehicle based on the adaptively fused voxel features using the target detection output head, behavior tracking output head and behavior prediction output head.

[0060] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0061] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0062] The above description is only a specific implementation scheme of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the concept of the present application. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. A real-time target detection, tracking and motion prediction method for autonomous driving, comprising the following steps: Step 1): Collect camera image data and lidar point cloud data around the vehicle through cameras and lidar; Step 2): extracting image features and point cloud features based on camera image data and lidar point cloud data respectively; Step 3): adopt a step-by-step fusion method to perform multi-modal information fusion on image features and point cloud features to obtain adaptively fused voxel features; Step 4): Based on the adaptively fused voxel features, the target detection output head, behavior tracking output head and behavior prediction output head are used to analyze and obtain the detection results, tracking results and prediction results of the moving objects around the vehicle.

2. The real-time target detection, tracking and motion prediction method for autonomous driving according to claim 1, characterized in that: The step 2) comprises: Taking the camera image data as input, the image features are generated through the image feature extraction network; The lidar point cloud data is normalized to voxel space, and then point cloud features are generated through the point cloud feature extraction network.

3. The real-time target detection, tracking and motion prediction method for autonomous driving according to claim 2, characterized in that: The image feature extraction network and the point cloud feature extraction network are both convolutional neural networks, including multiple layers of convolution and pooling layers.

4. The real-time target detection, tracking and motion prediction method for autonomous driving according to claim 1, characterized in that: The step 2) also includes compressing the vertical dimension of the point cloud feature to 1.

5. The real-time target detection, tracking and motion prediction method for autonomous driving according to claim 1, characterized in that: In step 3), the multimodal information fusion includes: 3.1) Generate corresponding adaptive fusion weights from image features and point cloud features, and perform numerical normalization on the generated adaptive fusion weights: Among them, α img , α pc is the adaptive fusion weight corresponding to the image features and point cloud features, is the normalized adaptive fusion weight, [.] represents the matrix, and σ is the normalization function; 3.2) Use the normalized adaptive fusion weights to weight the image features and point cloud features to obtain the adaptive fused voxel features: Among them, F fus is the voxel feature of adaptive fusion, F img and F pc are image features and point cloud features.

6. The real-time target detection, tracking and motion prediction method for autonomous driving according to claim 1, characterized in that: The step 4) comprises: Based on the voxel features of adaptive fusion, the target detection output head is used to detect the position of the three-dimensional object in real time and output it. The detection result is a series of detection frames, and the position of each detection frame is the position information of the three-dimensional object in the detection frame; Using the behavior tracking output head, the position information of all three-dimensional objects in the detection results of the current moment and the past moment are correlated and matched, the position information of the same three-dimensional object at different moments is obtained, and the motion trajectory of the object is formed and output; Based on the motion trajectory obtained by the behavior tracking output head, the behavior prediction output head is used to predict the position information of the object at the next moment and output it.

7. The real-time target detection, tracking and motion prediction method for autonomous driving according to claim 6, characterized in that: The associative matching of the position information of all three-dimensional objects in the detection results of the current moment and the past moment is specifically: feature matching is performed according to the feature vectors of the detection frames of the current moment and the previous moment in the corresponding adaptively fused voxel features, and the motion trajectory of the three-dimensional object is updated according to the matching results.

8. The real-time target detection, tracking and motion prediction method for autonomous driving according to claim 6, characterized in that: The target detection output head, behavior tracking output head and behavior prediction output head are all implemented using a linear layer neural network.

9. A real-time target detection, tracking and motion prediction system for autonomous driving, used to implement the method of claim 1, characterized in that: The system comprises: Data acquisition module, used to collect image data and lidar point cloud data around the vehicle; Feature extraction module, used to extract image features and point cloud features; The multimodal fusion module is used to fuse the multimodal information of image features and point cloud features and output adaptively fused voxel features; The detection, tracking and prediction module is used to obtain the detection results, tracking results and prediction results of the moving objects around the vehicle based on the adaptively fused voxel features using the target detection output head, behavior tracking output head and behavior prediction output head.

Images