Automatic driving end-to-end trajectory prediction method based on look-around fisheye camera

Abstract

The invention discloses an automatic driving end-to-end trajectory prediction method based on a look-around fisheye camera, and the method comprises the steps: carrying out the distortion processing of a plane camera image, and generating an original image of the look-around fisheye camera; a non-time-sequence decoder and a time-sequence decoder are adopted, object query, historical object query and image features are subjected to feature interaction through an attention mechanism by initializing object query and extracting hidden features of an original image, and targets around a vehicle at the current moment are detected; the object query of the detected target is input into a trajectory prediction module to perform interaction between the current object query and the historically stored object query to obtain the historical prediction trajectory and historical information of the model, then interaction between the targets is performed, and finally the object query is interacted with the image hidden features to enable the object query to obtain global environment information; and inputting the object query after feature interaction into the multilayer perceptron, and outputting and obtaining a trajectory prediction result. According to the method, the trajectory prediction performance and speed are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and end-to-end trajectory prediction, and in particular relates to an end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera. Background Art

[0002] With the continuous development of autonomous driving technology, the ability of vehicles to perceive and predict the environment has become one of the key factors affecting the safety and reliability of autonomous driving systems. Traditional environmental perception systems usually rely on the fusion of multiple sensors such as lidar, millimeter-wave radar and ordinary cameras. However, these sensors have their own advantages and disadvantages. For example, lidar is expensive and its performance will be affected in bad weather conditions; although millimeter-wave radar can work in bad weather, its resolution is low; although ordinary cameras are low-cost and can provide rich image information, their field of view is limited.

[0003] In recent years, surround-view fisheye cameras, as an emerging visual sensor, have gradually been used in autonomous driving systems. Surround-view fisheye cameras have the advantages of wide viewing angle, large coverage, and relatively low cost. They can effectively solve the problem of limited field of view of traditional cameras, thereby enhancing the vehicle's environmental perception ability. However, surround-view fisheye cameras also face some challenges in their application. Due to its imaging principle, the images of fisheye cameras have obvious distortion, and traditional target detection and trajectory prediction algorithms have great difficulty in processing such images. In addition, although the multi-sensor fusion method can improve the accuracy of environmental perception, its complex system architecture increases the computing and hardware costs, which is not conducive to the real-time and economic performance of the system.

[0004] Most of the existing target detection and trajectory prediction methods based on surround-view fisheye cameras are performed in steps, that is, target detection is performed first, and then trajectory prediction is performed based on the detection results. This method has problems of information loss and error accumulation, and it is difficult to meet the requirements of high precision and real-time performance. Therefore, there is an urgent need for a new method that can fully utilize the advantages of surround-view fisheye cameras, realize the integrated processing of target detection and trajectory prediction while processing distorted images, and thus improve the environmental perception ability and trajectory prediction accuracy of the autonomous driving system. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes an end-to-end trajectory prediction method based on a surround-view fisheye camera, aiming to solve the problems in the prior art and provide an efficient and accurate environment perception and trajectory prediction solution.

[0006] The present invention provides an end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera, comprising:

[0007] Based on the existing public autonomous driving data set, the planar camera images in the data set are distorted to generate the original images of the surround fisheye camera;

[0008] A non-temporal decoder and a temporal decoder are used to initialize an object query and extract hidden features of the original image, and the object query is interacted with the historical object query and the image features through an attention mechanism to detect the objects around the vehicle at the current moment;

[0009] The object query of the detected target is input into the trajectory prediction module. First, the current object query interacts with the historically stored object query to obtain the predicted trajectory and historical information of the model history. Then, the objects interact with each other. Finally, the object query interacts with the image hidden features to enable the object query to obtain global environmental information.

[0010] The object query after feature interaction is input into the multi-layer perceptron, and the trajectory prediction result is obtained as output.

[0011] Preferably, the process of performing distortion processing on the planar camera images in the data set includes:

[0012] The radial distortion and tangential distortion of the planar camera images in the dataset are modeled. During the modeling process, the coordinates of the planar camera images are first normalized, and the coordinate (0,0) is used to represent the center of the image, and the coordinates (1,1), (1,-1), (-1,-1), and (-1,1) represent the coordinates of the four corners of the image. Radial distortion and tangential distortion are introduced, and the image pixels are moved to obtain the processed fisheye image.

[0013] Preferably, the corresponding position relationship between the original pixel coordinates (x, y) of the planar camera image and the pixel coordinates (x', y') after fisheye distortion is:

[0014]

[0015] The corresponding relationship between the coordinates (x', y') after distortion scaling and the initial distortion coordinates (x', y') is:

[0016]

[0017] Among them, r is the distance between the current pixel and the initial distortion coordinate (x', y'), and n is the distortion coefficient.

[0018] Preferably, before querying and extracting the hidden features of the original image by initializing the object, the method further includes:

[0019] Based on the fisheye camera model in OpenCV, an odd-order polynomial is used to model the incident angle θ. The fisheye camera model is unified in the form of an isometric projection model. The original image is calibrated to obtain the intrinsic and extrinsic parameters of the camera image. The formula expression is:

[0020] r d =fθ d

[0021] Where f is the focal length of the camera, θ d is the incident angle after distortion, and θ d Make an approximation;

[0022] Its fisheye camera model is:

[0023] θ d =k 0 θ+k 1 θ 3 +k 2 θ 5 +k 3 θ 7 +k 4 θ 9 +···

[0024] Where θ is the original incident angle.

[0025] Preferably, a non-temporal decoder and a temporal decoder are used to initialize an object query and extract hidden features of the original image, and the object query is interacted with the historical object query and the image features through an attention mechanism. The process of detecting the target around the vehicle at the current moment includes:

[0026] Use convolutional neural network to encode the original image and obtain the hidden features of the image;

[0027] Randomly initialize object queries, input the initialized object queries into the non-temporal decoder to interact with the image hidden features, and select high-confidence object queries to pass into the temporal decoder;

[0028] The newly initialized object query with high confidence and the time series object query stored at the previous moment are merged and input into the time series decoder to detect all co-occurring targets to obtain the output of the detection module; the newly initialized object query is used to detect newly appearing targets, the time series object query is used to track the detected targets, and the high confidence object query is saved as the time series information of the next moment.

[0029] Preferably, the process of encoding the original image using a convolutional neural network to obtain hidden features of the image includes:

[0030] After the original image is preprocessed by cropping and rotating, ResNet50 or ResNet101 is used as the model backbone convolutional neural network to encode the panoramic fisheye image, extract image information through multi-layer convolution operations, compress high-dimensional image data into low-dimensional feature vectors, and obtain the hidden features of the panoramic fisheye image.

[0031] Preferably, the object query is randomly initialized, the initialized object query is input into the non-temporal decoder to interact with the image hidden features, and the process of selecting the object query with high confidence and inputting it into the temporal decoder includes:

[0032] Randomly initialize object query and query the initialized object Q init The attention mechanism is used to interact the object query with the hidden features of the fisheye image when input into the non-temporal decoder, identify the potential target in the current frame, filter out the target information with high confidence, and take the three hundred object queries with the highest confidence to pass into the temporal decoder. The calculation formula is:

[0033]

[0034] Among them, M' i is the bias coefficient, i is the sequence number of the query vector, q i is the i-th randomly initialized query vector, is the i-th calculated query vector, v i is the i-th value vector, k i is the i-th key vector, d k is the dimension of the vector, W i are the parameter weights of the three-layer feed-forward network.

[0035] Preferably, the time series decoder uses a time series model to combine the information of the current frame and the historical frame to continuously track and detect the target. The formula expression is:

[0036]

[0037]

[0038] Q = CrossAttention(Q,F,F)

[0039] Among them, Q represents the query vector obtained after calculation, represents the 600 historical query vectors stored at the last moment, and F represents the hidden features of the surround fisheye image.

[0040] Preferably, the process of inputting the object query after feature interaction into a multi-layer perceptron and outputting the trajectory prediction result includes:

[0041] The final object query output trajectory prediction results are used by a multi-layer perceptron. The trajectory prediction results of each detected target include multiple possible future trajectories. Each trajectory contains multiple unknown future coordinates of targets in a certain period of time in the future. The output coordinates include the coordinates of the X-axis and Y-axis in the map coordinate system (X map ,Y map )Saves the object query as timing information for subsequent moments.

[0042] Compared with the prior art, the present invention has the following advantages and technical effects:

[0043] The present invention achieves accurate prediction of the future trajectories of targets around the vehicle by jointly optimizing the model end-to-end, avoiding the complex post-processing process and accumulated errors required for traditional multiple models to achieve trajectory prediction. The effectiveness of the model proposed in the present invention is verified by comparative experiments with existing deep learning methods.

[0044] The present invention is based on a surround-view fisheye camera. By detecting the target and predicting its future trajectory, the predicted trajectory of the target can be directly obtained from the input image, which greatly simplifies the intermediate processing flow of the model, avoids the cumulative error caused by using multiple models, and improves the performance and speed of trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 A schematic diagram of the fisheye distortion modeling principle of an embodiment of the present invention;

[0047] Figure 2 It is a diagram of the overall structure of the model of an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the structure of a non-sequential decoder according to an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of the structure of a timing decoder according to an embodiment of the present invention;

[0050] Figure 5 Schematic diagram of the detection module structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] like Figure 1-5 As shown, this embodiment provides an end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera, including:

[0054] Based on the existing public autonomous driving data set, the planar camera images in the data set are distorted to generate the original images of the surround fisheye camera;

[0055] A non-temporal decoder and a temporal decoder are used to initialize object queries and extract hidden features of surround fisheye images. The object queries are interacted with historical object queries and image features through an attention mechanism to detect targets around the vehicle at the current moment.

[0056] The object query of the detected target is input into the trajectory prediction module. First, the current object query interacts with the historically stored object query to obtain the predicted trajectory and historical information of the model history. Then, the targets interact with each other to make them pay attention to each other's motion status. Finally, the object query interacts with the image hidden features to enable the object query to obtain global environmental information.

[0057] The object query after feature interaction is input into the multi-layer perceptron, and the trajectory prediction is output. The prediction content is multiple possible future trajectories of the target. Each motion trajectory contains the coordinate points of the predicted target at a fixed time interval in the future, and the object query is saved as the time series information of the subsequent moment. By predicting multiple future motion trajectories, the target's future multiple intentions and routes can be inferred, thereby achieving accurate prediction of the target's future trajectory.

[0058] In a further optimization scheme, the method of the present invention comprises the following steps:

[0059] Step 1: Based on the existing public autonomous driving data set, the planar camera image in the data set is distorted to generate the original image of the surround fisheye camera;

[0060] Step 2: Calibrate the original image based on the fisheye camera model in OpenCV to obtain the intrinsic and extrinsic parameters of the image;

[0061] Step 3: Use convolutional neural network to encode the original image and obtain the hidden features of the image;

[0062] Step 4: Randomly initialize the object query, input the initialized object query into the non-temporal decoder to interact with the image hidden features, and select the object query with high confidence to pass it into the temporal decoder;

[0063] Step 5: merge the newly initialized object query with high confidence and the time series object query stored at the previous moment, input them into the time series decoder to detect all co-occurring targets, and obtain the output of the detection module, where the newly initialized object query is used to detect the newly appeared target, and the time series object query is used to track the detected target, and the object query with high confidence is saved as the time series information of the next moment;

[0064] Step 6: Input the object query of the detected target into the trajectory prediction module, interact with the historical object query stored in the module, obtain the time series information and output the object query that obtains the historical information;

[0065] Step 7: The output object queries are interacted with each other so that each object query obtains the interactive information between the targets;

[0066] Step 8: Interact the object query with the hidden features encoded by the convolutional neural network so that the object query can obtain global environment information;

[0067] Step 9: Use a multi-layer perceptron to predict the trajectory of the final object query output. The prediction content is multiple possible future trajectories of the target. Each trajectory includes the coordinate points of the target at a fixed time interval in the future, and the object query is saved as the timing information of the subsequent moment.

[0068] Furthermore, in the step 1, the planar camera image in the data set is distorted, and radial distortion and tangential distortion modeling are performed on the planar camera image in the data set, and the image pixels are moved to obtain a processed fisheye image.

[0069] Specifically, the open source NuScenes dataset is used to obtain original images from existing public datasets. The images contain complex scenes of different weather conditions, different cities, and different weather conditions. Each scene is 20 seconds long, and two keyframes are annotated per second. The surround camera images obtained from the dataset are modeled for radial distortion and tangential distortion. In the modeling process, the original image coordinates are first normalized, and the (0,0) coordinate represents the center of the image, (1,1), (1,-1), (-1,-1), and (-1,1) represent the coordinates of the four corners of the image, and radial distortion and tangential distortion are introduced. Specifically, the fisheye camera is regarded as a spherical model, and the degree of distortion of its pixels is closely related to the distance between the pixel and the center of the sphere. The closer the distance to the center of the sphere, the smaller the degree of distortion, and vice versa. The corresponding position relationship between the original pixel coordinates (x,y) and the pixel coordinates after fisheye distortion (x',y') is as follows:

[0070]

[0071] In order to control the degree of distortion, the distortion coefficient is introduced on the basis of distortion modeling. The corresponding relationship between the coordinates (x', y') after distortion scaling and the initial distortion coordinates (x', y') is as follows:

[0072]

[0073] Among them, r is the distance between the current pixel and the initial distortion coordinate (x', y'), and n is the distortion coefficient.

[0074] Furthermore, in step 2, the fisheye camera model in OpenCV is used to model the incident angle θ using an odd-order polynomial, and the form of the fisheye camera model is unified in the form of an equidistant projection model. After calibration, the camera internal and external parameters are obtained, that is,

[0075] r d =fθ d

[0076] Where f is the focal length of the camera, θ d is the incident angle after distortion, and θ d Make an approximation;

[0077] Its fisheye camera model is:

[0078] θ d =k 0 θ+k 1 θ 3 +k 2 θ 5 +k 3 θ 7 +k 4 θ 9 +···

[0079] Where θ is the original incident angle.

[0080] Furthermore, in the step three, after performing preprocessing methods such as cropping and rotating on the image, convolutional neural networks such as ResNet50 and ResNet101 are used as the model backbone network to encode the surround fisheye image, extract important information in the image through multi-layer convolution operations, and compress the high-dimensional image data into a low-dimensional feature vector to obtain the hidden features F of the surround fisheye image as input for subsequent processing.

[0081] Furthermore, in step 4, a certain number of object queries are randomly initialized, and the initialized object queries Q initThe attention mechanism is used to interact the object query with the hidden features of the fisheye image when input into the non-temporal decoder, identify the potential target in the current frame, filter out the target information with high confidence, and take the three hundred object queries with the highest confidence to pass into the temporal decoder. The calculation formula is:

[0082]

[0083] Among them, M' i is the bias coefficient, i is the sequence number of the query vector, q i is the i-th randomly initialized query vector, is the i-th calculated query vector, v i is the i-th value vector, k i is the i-th key vector, d k is the dimension of the vector, W i are the parameter weights of the three-layer feed-forward network.

[0084] Furthermore, in step five, the output of the non-time series module is merged with the historical query saved at the previous moment and input into the time series decoder for decoding to obtain the result of target detection. The detection range is a circle with the vehicle as the center and a radius of 55 meters. The query passed in by the non-time series module is responsible for detecting newly appeared targets, and the query from the previous moment is responsible for tracking the detected targets. The output with high confidence in the detection result is selected, and the corresponding object query is saved as the time series information of the next moment.

[0085] The temporal decoder uses the time series model to combine the information of the current frame and the historical frame to achieve continuous tracking and detection of the target. The calculation formula is:

[0086]

[0087] Q = CrossAttention(Q,F,F)

[0088] Furthermore, in step 6, the target detection result is input into the trajectory prediction module, and the attention mechanism is used in the time sequence interaction module to interact the current frame target detection result with the historical information stored in the trajectory prediction module, so that the object query obtains the time sequence information. The historical object query stored in the trajectory prediction module is the trajectory predicted by the module in the past 2 seconds.

[0089] Furthermore, in step 7, the object queries output by the time series interaction module interact with each other so that each object query pays attention to the information of the surrounding targets. By interacting with each other, each object query obtains the interaction information between targets. Through the interaction between targets, the model can better understand the mutual influence between targets and improve the accuracy of prediction.

[0090] Furthermore, in step eight, the attention mechanism is used to interact the object query with the hidden features encoded by the convolutional neural network, so that the object query obtains global environmental information. Through the interaction with the global features, the model can combine the environmental information for more comprehensive trajectory prediction, thereby improving the trajectory prediction ability of the model.

[0091] Furthermore, in step nine, a multi-layer perceptron (MLP) is used to query the object output by the trajectory prediction module to output the trajectory prediction result. The trajectory prediction result of each detected target includes multiple possible future trajectories, each trajectory includes multiple unknown future coordinates of the target within a period of time in the future, and the output coordinates include the coordinates of the X-axis and the Y-axis in the map coordinate system (X map ,Y map ) Save the object query as the time series information of the subsequent moment. The trajectory prediction module generates possible future motion trajectories for each target by analyzing the historical trajectory data of the target. For example, the trajectory prediction module predicts 6 future motion trajectories of the target, each of which contains the position of the predicted target at [0.5, 1.0, 1.5, 2.0, 2.5, 3.0] seconds in the future. By predicting multiple future motion trajectories, it can infer the future multi-intention and multi-route of the target, thereby achieving accurate prediction of the target's future trajectory.

[0092] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera, characterized in that: include: Based on the existing public autonomous driving data set, the planar camera images in the data set are distorted to generate the original images of the surround fisheye camera; A non-temporal decoder and a temporal decoder are used to initialize an object query and extract hidden features of the original image, and the object query is interacted with the historical object query and the image features through an attention mechanism to detect the objects around the vehicle at the current moment; The object query of the detected target is input into the trajectory prediction module. First, the current object query interacts with the historically stored object query to obtain the predicted trajectory and historical information of the model history. Then, the objects interact with each other. Finally, the object query interacts with the image hidden features to enable the object query to obtain global environmental information. The object query after feature interaction is input into the multi-layer perceptron, and the trajectory prediction result is obtained as output.

2. The end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera according to claim 1, characterized in that: The process of performing distortion processing on the planar camera image in the data set includes: The radial distortion and tangential distortion of the planar camera images in the dataset are modeled. During the modeling process, the coordinates of the planar camera images are first normalized, and the coordinate (0,0) is used to represent the center of the image, and the coordinates (1,1), (1,-1), (-1,-1), and (-1,1) represent the coordinates of the four corners of the image. Radial distortion and tangential distortion are introduced, and the image pixels are moved to obtain the processed fisheye image.

3. The end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera according to claim 2, characterized in that: The corresponding position relationship between the original pixel coordinates (x, y) of the planar camera image and the pixel coordinates (x', y') after fisheye distortion is: The corresponding relationship between the coordinates (x', y') after distortion scaling and the initial distortion coordinates (x', y') is: Among them, r is the distance between the current pixel and the initial distortion coordinate (x', y'), and n is the distortion coefficient.

4. The end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera according to claim 1, characterized in that: Before initializing the object query and extracting the hidden features of the original image, the method further includes: Based on the fisheye camera model in OpenCV, an odd-order polynomial is used to model the incident angle θ. The fisheye camera model is unified in the form of an isometric projection model. The original image is calibrated to obtain the intrinsic and extrinsic parameters of the camera image. The formula expression is: r d =fθ d Where f is the focal length of the camera, θ d is the incident angle after distortion, and θ d Make an approximation; Its fisheye camera model is: i d =k0 θ+k1 θ 3 +k2 θ 5 +k3 θ 7 +k4 θ 9 +··· Where θ is the original incident angle.

5. The end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera according to claim 1, characterized in that: The process of using a non-temporal decoder and a temporal decoder to initialize an object query and extract hidden features of the original image, and interacting the object query with historical object queries and image features through an attention mechanism to detect targets around the vehicle at the current moment includes: Use convolutional neural network to encode the original image and obtain the hidden features of the image; Randomly initialize object queries, input the initialized object queries into the non-temporal decoder to interact with the image hidden features, and select high-confidence object queries to pass into the temporal decoder; The newly initialized object query with high confidence and the time series object query stored at the previous moment are merged and input into the time series decoder to detect all co-occurring targets to obtain the output of the detection module; the newly initialized object query is used to detect newly appearing targets, the time series object query is used to track the detected targets, and the high confidence object query is saved as the time series information of the next moment.

6. The end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera according to claim 5, characterized in that: The process of encoding the original image using a convolutional neural network and obtaining the hidden features of the image includes: After the original image is preprocessed by cropping and rotating, ResNet50 or ResNet101 is used as the model backbone convolutional neural network to encode the panoramic fisheye image, extract image information through multi-layer convolution operations, compress high-dimensional image data into low-dimensional feature vectors, and obtain the hidden features of the panoramic fisheye image.

7. The end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera according to claim 5, characterized in that: The object query is randomly initialized, and the initialized object query is input into the non-temporal decoder to interact with the image hidden features. The process of selecting high-confidence object queries and inputting them into the temporal decoder includes: Randomly initialize object query and query the initialized object Q init The attention mechanism is used to interact the object query with the hidden features of the fisheye image when input into the non-temporal decoder, identify the potential target in the current frame, filter out the target information with high confidence, and take the three hundred object queries with the highest confidence to pass into the temporal decoder. The calculation formula is: Among them, M' i is the bias coefficient, i is the sequence number of the query vector, q i is the i-th randomly initialized query vector, is the i-th calculated query vector, v i is the i-th value vector, k i is the i-th key vector, d k is the dimension of the vector, W i are the parameter weights of the three-layer feed-forward network.

8. The end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera according to claim 5, characterized in that: The temporal decoder uses the time series model to combine the information of the current frame and the historical frame to continuously track and detect the target. The formula expression is: Q = CrossAttention(Q,F,F) Among them, Q represents the query vector obtained after calculation, represents the 600 historical query vectors stored at the last moment, and F represents the hidden features of the surround fisheye image.

9. The end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera according to claim 1, characterized in that: The process of inputting the object query after feature interaction into the multi-layer perceptron and outputting the trajectory prediction result includes: The final object query output trajectory prediction results are used by a multi-layer perceptron. The trajectory prediction results of each detected target include multiple possible future trajectories. Each trajectory contains multiple unknown future coordinates of targets in a certain period of time in the future. The output coordinates include the coordinates of the X-axis and Y-axis in the map coordinate system (X map ,Y map )Saves the object query as timing information for subsequent moments.

Images