An End-to-End Trajectory Prediction Method for Autonomous Driving Based on Surround-Looking Fisheye Cameras

Abstract

This invention discloses an end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera, comprising: distorting a planar camera image to generate the original image from the surround-view fisheye camera; employing a non-temporal decoder and a temporal decoder, initializing an object query and extracting hidden features from the original image, interacting the object query with historical object queries and image features through an attention mechanism to detect targets around the vehicle at the current moment; inputting the detected object query into the trajectory prediction module to interact with the current object query and historically stored object queries to obtain the model's historical predicted trajectories and historical information, then performing interactions between targets, and finally interacting the object query with the image's hidden features to enable the object query to obtain global environmental information; inputting the feature-interacted object query into a multilayer perceptron to output the trajectory prediction result. This invention improves the performance and speed of trajectory prediction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of target detection and end-to-end trajectory prediction, and particularly relates to an automatic driving end-to-end trajectory prediction method based on a surround-view fisheye camera. BACKGROUND

[0002] With the continuous development of automatic driving technology, the ability of vehicles to perceive and predict the environment has become one of the key factors affecting the safety and reliability of automatic driving systems. Traditional environment perception systems usually rely on the fusion of multiple sensors such as lidar, millimeter wave radar, and ordinary cameras. However, these sensors have their own advantages and disadvantages. For example, lidar is expensive and its performance is affected in bad weather conditions; millimeter wave radar can work in bad weather but has low resolution; and ordinary cameras have low cost and can provide rich image information, but their field of view is limited.

[0003] In recent years, surround-view fisheye cameras have gradually been applied to automatic driving systems as a new type of visual sensor. Surround-view fisheye cameras have the advantages of wide viewing angle, large coverage range, and relatively low cost, and can effectively solve the problem of limited field of view of traditional cameras, thereby enhancing the environmental perception ability of vehicles. However, surround-view fisheye cameras also face some challenges in application. Due to their imaging principle, fisheye camera images have obvious distortion, making it difficult for traditional target detection and trajectory prediction algorithms to process such images. In addition, although the multi-sensor fusion method can improve the accuracy of environmental perception, the complex system architecture increases the computational and hardware costs, which is not conducive to the real-time performance and economy of the system.

[0004] Existing target detection and trajectory prediction methods based on surround-view fisheye cameras are mostly performed in steps, i.e., first performing target detection and then performing trajectory prediction based on the detection results. This method has the problems of information loss and error accumulation, making it difficult to meet the requirements of high precision and real-time performance. Therefore, there is an urgent need for a new method that can fully utilize the advantages of surround-view fisheye cameras, process distorted images, and realize integrated processing of target detection and trajectory prediction, thereby improving the environmental perception ability and trajectory prediction accuracy of automatic driving systems. SUMMARY

[0005] To solve the above technical problems, the present application proposes an end-to-end trajectory prediction method based on a surround-view fisheye camera, aiming to solve the problems in the prior art and provide an efficient and accurate environmental perception and trajectory prediction solution.

[0006] The automatic driving end-to-end trajectory prediction method based on a surround-view fisheye camera provided by the present application comprises:

[0007] Based on the existing public automatic driving public data set, the planar camera image in the data set is distorted to generate the original image of the all-around fisheye camera;

[0008] The non-timing decoder and the timing decoder are adopted, the hidden features of the original image are extracted through the initialization of the object query, the object query and the historical object query and the image features are interacted through the attention mechanism, and the target around the vehicle at the current time is detected;

[0009] The object query detecting the target is input to the trajectory prediction module, the current object query is first interacted with the historical stored object query to obtain the prediction trajectory and the historical information of the model, then the targets are interacted, and finally the object query is interacted with the image hidden features, so that the object query obtains the global environmental information;

[0010] The object query after the feature interaction is input to the multi-layer perception machine, and the trajectory prediction result is output.

[0011] Preferably, the process of distorting the planar camera image in the data set comprises:

[0012] The radial distortion and the tangential distortion of the planar camera image in the data set are modeled, in the modeling process, the planar camera image coordinates are first normalized, the (0, 0) coordinates represent the image center, the (1, 1), (1, -1), (-1, -1) and (-1, 1) coordinates represent the coordinates of the four corners of the image, the radial distortion and the tangential distortion are introduced, the image pixels are moved, and the processed fisheye image is obtained.

[0013] Preferably, the corresponding position relationship between the original pixel coordinates (x, y) of the planar camera image and the pixel coordinates (x', y') after the fisheye distortion is:

[0014]

[0015] The corresponding relationship between the scaled coordinates (x'', y'') after the distortion and the initial distortion coordinates (x', y') is:

[0016]

[0017] Wherein, r is the distance between the current pixel and the initial distortion coordinates (x', y'), and n is the distortion coefficient.

[0018] Preferably, before the initialization of the object query and the extraction of the hidden features of the original image, it further comprises:

[0019] Based on the fisheye camera model in OpenCV, an odd-degree polynomial is used to model the incident angle θ. The fisheye camera model is then represented by an isometric projection model. The original image is calibrated to obtain the intrinsic and extrinsic parameters of the camera image. The formula expression is as follows:

[0020] r d =fθ d

[0021] Where f is the camera focal length, θ d Let θ be the distorted angle of incidence. d Approximation;

[0022] Its fisheye camera model is as follows:

[0023] θ d =k0 θ+k1 θ 3 +k2 θ 5 +k3 θ 7 +k4θ 9 +···

[0024] Where θ is the original incident angle.

[0025] Preferably, employing a non-temporal decoder and a temporal decoder, the process of detecting targets around the vehicle at the current moment includes: initializing object queries and extracting hidden features from the original image, and then interacting with historical object queries and image features through an attention mechanism.

[0026] The original image is encoded using a convolutional neural network to obtain hidden features.

[0027] Randomly initialize object queries, input the initialized object queries into the non-temporal decoder to interact with the image hidden features, and select high-confidence object queries to be input into the temporal decoder;

[0028] The newly initialized object query with high confidence and the time-series object query stored in the previous time step are merged and input into the time-series decoder to detect all co-occurring targets, and the output of the detection module is obtained. The newly initialized object query is used to detect newly appearing targets, the time-series object query is used to track the detected targets, and the high-confidence object query is saved as the time-series information for the next time step.

[0029] Preferably, the process of encoding the original image using a convolutional neural network to obtain hidden image features includes:

[0030] After preprocessing the original image by cropping and rotating, ResNet50 or ResNet101 is used as the backbone convolutional neural network to encode the panoramic fisheye image. Image information is extracted through multi-layer convolution operations, and high-dimensional image data is compressed into low-dimensional feature vectors to obtain the hidden features of the panoramic fisheye image.

[0031] Preferably, the process of randomly initializing object queries, inputting the initialized object queries into a non-temporal decoder to interact with image hidden features, and selecting high-confidence object queries to be input into the temporal decoder includes:

[0032] Randomly initialize object query, and initialize object query Q. init The input to the non-temporal decoder uses an attention mechanism to interact with the hidden features of the fisheye image, identifying potential targets in the current frame, filtering out high-confidence target information, and taking the three hundred highest-confidence object queries to be fed into the temporal decoder. The calculation formula is as follows:

[0033]

[0034] Among them, M' i Here, q represents the bias coefficient, i is the index of the query vector, and q represents the bias coefficient. i Let i be the i-th randomly initialized query vector. Let v be the i-th computed query vector. i Let k be the i-th value vector. i Let d be the i-th key vector. k W is the dimension of the vector. i These are the parameter weights of a three-layer feedforward network.

[0035] Preferably, the temporal decoder uses a time series model, combining information from the current frame and historical frames, to continuously track and detect the target using the following formula:

[0036]

[0037]

[0038] Q = CrossAttention(Q, F, F)

[0039] Where Q represents the calculated query vector. This represents the 600 historical query vectors stored at the previous time step, and F represents the hidden features of the panoramic fisheye image.

[0040] Preferably, the process of inputting the object query after feature interaction into a multilayer perceptron and outputting the trajectory prediction result includes:

[0041] The multilayer perceptron is used to output the final object query trajectory prediction results. For each detected target, the trajectory prediction results include multiple possible future trajectories. Each trajectory contains multiple unknown future coordinates of the target within a future time period. The output coordinates include the X-axis and Y-axis coordinates in the map coordinate system (X...). map ,Y map Save the object query as time sequence information for subsequent moments.

[0042] Compared with the prior art, the present invention has the following advantages and technical effects:

[0043] This invention achieves accurate prediction of the future trajectory of targets around a vehicle by jointly optimizing the model end-to-end, thus avoiding the complex post-processing and accumulated errors required for trajectory prediction using multiple traditional models. The effectiveness of the proposed model is verified by comparative experiments with existing deep learning methods.

[0044] This invention is based on a surround-view fisheye camera. By detecting the target and predicting its future trajectory, the predicted trajectory of the target can be obtained directly from the input image, which greatly simplifies the intermediate processing of the model, avoids the cumulative error caused by using multiple models, and improves the performance and speed of trajectory prediction. Attached Figure Description

[0045] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a schematic diagram illustrating the fisheye distortion modeling principle of an embodiment of the present invention;

[0047] Figure 2 This is a diagram showing the overall structure of the model according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of a non-time-sequential decoder structure according to an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the timing decoder structure according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the detection module structure according to an embodiment of the present invention. Detailed Implementation

[0051] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0052] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0053] like Figures 1-5 As shown, this embodiment provides an end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera, including:

[0054] Based on existing publicly available autonomous driving datasets, distortion processing is performed on the planar camera images in the datasets to generate the original images from the surround-view fisheye camera.

[0055] Using both non-temporal and temporal decoders, the system initializes object queries and extracts hidden features from the surround-view fisheye image. It then uses an attention mechanism to interact with historical object queries and image features to detect targets around the vehicle at the current moment.

[0056] The object query that detects the target is input into the trajectory prediction module. First, the current object query interacts with the historically stored object queries to obtain the model's historical predicted trajectories and historical information. Then, the objects interact with each other to make them pay attention to each other's motion states. Finally, the object query interacts with the image hidden features to enable the object query to obtain global environmental information.

[0057] The object query, after feature interaction, is input into a multilayer perceptron, which outputs trajectory predictions. The predictions consist of multiple possible future trajectories of the target. Each trajectory includes the coordinates of the target at fixed time intervals within a future period. The object query is saved as temporal information for subsequent moments. By predicting multiple future trajectories, the system can infer the target's future intentions and routes, thereby achieving accurate prediction of the target's future trajectory.

[0058] A further optimization of the method of the present invention includes the following steps:

[0059] Step 1: Based on the existing publicly available autonomous driving dataset, perform distortion processing on the planar camera images in the dataset to generate the original images from the surround-view fisheye camera;

[0060] Step 2: Based on the fisheye camera model in OpenCV, calibrate the obtained original image to obtain the intrinsic and extrinsic parameters of the image;

[0061] Step 3: Encode the original image using a convolutional neural network to obtain hidden features;

[0062] Step 4: Randomly initialize object queries, input the initialized object queries into the non-temporal decoder to interact with the image hidden features, and select high-confidence object queries to be input into the temporal decoder;

[0063] Step 5: Merge the high-confidence newly initialized object query and the time-series object query stored in the previous time step, and input them into the time-series decoder to detect all co-occurring targets, and obtain the output of the detection module. The newly initialized object query is used to detect newly occurring targets, and the time-series object query is used to track the detected targets. The high-confidence object query is saved as the time-series information for the next time step.

[0064] Step 6: Input the object query of the detected target into the trajectory prediction module, interact with the historical object queries stored in the module, obtain time series information, and output the object query with obtained historical information.

[0065] Step 7: Query the output objects to enable them to interact with each other, so that each object query can obtain the interaction information between the targets;

[0066] Step 8: Interact the object query with the hidden features encoded by the convolutional neural network to enable the object query to obtain global environmental information;

[0067] Step 9: Use a multilayer perceptron to predict the final object query trajectory. The prediction content is multiple possible future trajectories of the target. Each trajectory includes the coordinates of the target at fixed time intervals within a certain period of the future. Save the object query as the temporal information for subsequent moments.

[0068] Furthermore, in step one, the planar camera images in the dataset are subjected to distortion processing. By modeling radial and tangential distortion of the planar camera images in the dataset, the image pixels are moved to obtain the processed fisheye image.

[0069] Specifically, the open-source NuScenes dataset was used to obtain raw images from existing public datasets. These images contain complex scenes with varying weather conditions, cities, and weather conditions. Each scene is 20 seconds long, with two keyframes labeled per second. Radial and tangential distortion models were then performed on the panoramic camera images obtained from the dataset. The modeling process first normalized the original image coordinates, using (0,0) to represent the image center and (1,1), (1,-1), (-1,-1), and (-1,1) to represent the coordinates of the four corners. Radial and tangential distortion were introduced. Specifically, the fisheye camera was considered a spherical model, and the degree of pixel distortion was closely related to the distance between the pixel and the center of the sphere. The closer the pixel was to the center, the smaller the distortion, and vice versa. The corresponding positional relationship between the original pixel coordinates (x,y) and the fisheye-distorted pixel coordinates (x',y') is as follows:

[0070]

[0071] To control the degree of distortion, a distortion coefficient is introduced based on the distortion model. The correspondence between the distorted scaled coordinates (x”, y”) and the initial distorted coordinates (x’, y’) is as follows:

[0072]

[0073] Where r is the distance between the current pixel and the initial distortion coordinates (x', y'), and n is the distortion coefficient.

[0074] Furthermore, in step two, a fisheye camera model in OpenCV is used, and an odd-degree polynomial is used to model the incident angle θ. The form of the fisheye camera model is unified into the form of an equidistant projection model. After calibration, the camera's intrinsic and extrinsic parameters are obtained.

[0075] r d =fθ d

[0076] Where f is the camera focal length, θ d Let θ be the distorted angle of incidence. d Approximation;

[0077] Its fisheye camera model is as follows:

[0078] θ d =k0 θ+k1 θ 3 +k2 θ 5 +k3 θ 7 +k4 θ 9 +···

[0079] Where θ is the original incident angle.

[0080] Furthermore, in step three, after preprocessing the image by cropping, rotating, etc., convolutional neural networks such as ResNet50 and ResNet101 are used as the backbone network of the model to encode the panoramic fisheye image. Important information in the image is extracted through multi-layer convolution operations, and the high-dimensional image data is compressed into a low-dimensional feature vector to obtain the hidden features F of the panoramic fisheye image, which is used as the input for subsequent processing.

[0081] Furthermore, in step four, a certain number of object queries are randomly initialized, and the initialized object queries Q are... init The input to the non-temporal decoder uses an attention mechanism to interact with the hidden features of the fisheye image, identifying potential targets in the current frame, filtering out high-confidence target information, and taking the three hundred highest-confidence object queries to be fed into the temporal decoder. The calculation formula is as follows:

[0082]

[0083] Among them, M' i Here, q represents the bias coefficient, i is the index of the query vector, and q represents the bias coefficient. i Let i be the i-th randomly initialized query vector. Let v be the i-th computed query vector. i Let k be the i-th value vector. i Let d be the i-th key vector. k W is the dimension of the vector. i These are the parameter weights of a three-layer feedforward network.

[0084] Furthermore, in step five, the output of the non-time-series module is merged with the historical query saved in the previous moment and input into the time-series decoder for decoding to obtain the target detection result. The detection range is a circle with a radius of 55 meters centered on the vehicle. The query from the non-time-series module is responsible for detecting newly appearing targets, while the query from the previous moment is responsible for tracking the detected targets. The output with high confidence in the detection result is selected, and the corresponding object query is saved as the time-series information for the next moment.

[0085] The temporal decoder uses a time-series model, combining information from the current frame and historical frames, to achieve continuous tracking and detection of the target. Its calculation formula is as follows:

[0086]

[0087] Q = CrossAttention(Q, F, F)

[0088] Furthermore, in step six, the target detection result is input into the trajectory prediction module. The temporal interaction module uses an attention mechanism to interact with the current frame's target detection result and the historical information stored in the trajectory prediction module, enabling the object query to obtain temporal information. The historical object query stored in the trajectory prediction module refers to the trajectories predicted by the module within the past two seconds.

[0089] Furthermore, in step seven, the object queries output by the time-series interaction module interact with each other, enabling each object query to focus on information about surrounding targets. By allowing the output object queries to interact with each other, each object query acquires interaction information between targets. Through the interaction between targets, the model can better understand the mutual influence between targets, thereby improving the accuracy of predictions.

[0090] Furthermore, in step eight, an attention mechanism is used to interact the object query with the hidden features encoded by the convolutional neural network, enabling the object query to acquire global environmental information. Through interaction with global features, the model can combine environmental information to make more comprehensive trajectory predictions, thereby improving the model's trajectory prediction capabilities.

[0091] Furthermore, in step nine, a multilayer perceptron (MLP) is used to query the trajectory prediction results of the object query output from the trajectory prediction module. The trajectory prediction result for each detected target includes multiple possible future trajectories. Each trajectory contains multiple unknown future coordinates of the target within a future time period. The output coordinates include the X-axis and Y-axis coordinates in the map coordinate system (X...). map ,Y map The object query is saved as time-series information for subsequent moments. The trajectory prediction module analyzes the target's historical trajectory data to generate possible future motion trajectories for each target. For example, the trajectory prediction module predicts 6 future motion trajectories for a target, each trajectory containing the predicted target's position at [0.5, 1.0, 1.5, 2.0, 2.5, 3.0] seconds in the future. By predicting multiple future motion trajectories, the module infers multiple intentions and routes of the target in the future, thereby achieving accurate prediction of the target's future trajectory.

[0092] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An end-to-end trajectory prediction method for autonomous driving based on a surround-view fisheye camera, characterized in that, The method comprises: Based on the existing public automatic driving public data set, the planar camera image in the data set is distorted to generate the original image of the surround view fisheye camera; Specifically, the radial distortion and tangential distortion of the planar camera image in the data set are modeled. In the modeling process, the planar camera image coordinates are first normalized, (0, 0) coordinates represent the image center, (1, 1), (1, -1), (-1, -1), and (-1, 1) represent the coordinates of the four corners of the image. Radial distortion and tangential distortion are introduced to move the image pixels to obtain the processed fisheye image. The non-sequential decoder and the sequential decoder are used to initialize the object query and extract the hidden features of the original image, and the object query and the historical object query and the image features are interacted through the attention mechanism to detect the target around the vehicle at the current time. Specifically, the convolutional neural network is used to encode the original image to obtain the image hidden features. Randomly initialize the object query, input the initialized object query into the non-sequential decoder to interact with the image hidden features, and select the high-confidence object query to input into the sequential decoder. The high-confidence new initialized object query and the stored sequential object query at the last time are combined and input into the sequential decoder to detect all the co-occurring targets to obtain the output of the detection module. The new initialized object query is used to detect the newly appearing target, and the sequential object query is used to track the detected target, and the high-confidence object query is saved as the sequential information at the next time. The object query detecting the target is input into the trajectory prediction module. First, the current object query and the historically stored object query are interacted to obtain the prediction trajectory and historical information of the model, then the targets are interacted, and finally the object query is interacted with the image hidden features to make the object query obtain the global environmental information. The object query after feature interaction is input into the multi-layer perception to output the trajectory prediction result.

2. The surround view fisheye camera-based automatic driving end-to-end trajectory prediction method according to claim 1, further comprising, before the initialization of the object query and the extraction of the hidden features of the original image: The original pixel coordinates of the planar camera image ( ) and pixel coordinates after fisheye distortion ( The corresponding positional relationship is as follows: The coordinate after distortion scaling corresponds to the initial distortion coordinate in the following relationship: Where r is the coordinate of the current pixel relative to the initial distortion coordinates ( The distance between and n is the distortion coefficient. 3.The method of claim 1, wherein, The fisheye camera model is: Based on the fisheye camera model in OpenCV, odd-degree polynomials are used to adjust the incident angle. Modeling is performed, unifying the fisheye camera model into an isometric projection model. The obtained original images are calibrated to obtain the intrinsic and extrinsic parameters of the camera image. The formula expression is as follows: wherein is the camera focal length, is the incident angle after distortion, and is approximated; The process of encoding the original image using the convolutional neural network to obtain the image hidden features comprises: wherein is the original angle of incidence. 4.The method of claim 1, wherein, After the preprocessing of cropping and rotating the original image, ResNet50 or ResNet101 is used as the model backbone convolutional neural network to encode the surround view fisheye image. The image information is extracted through multi-layer convolution operation to compress the high-dimensional image data into a low-dimensional feature vector to obtain the hidden features of the surround view fisheye image. The process of randomly initializing the object query, inputting the initialized object query into the non-sequential decoder to interact with the image hidden features, and selecting the high-confidence object query to input into the sequential decoder comprises:

5. The surround view fisheye camera based autonomous driving end-to-end trajectory prediction method according to claim 1, wherein, 6. The surround view fisheye camera-based automatic driving end-to-end trajectory prediction method according to claim 1, wherein Randomly initialize object queries, initialize the object query Input into the non-timing decoder, use attention mechanism to interact object query with hidden features of fisheye image, identify potential targets in current frame, filter out high confidence target information, take three hundred object queries with highest confidence to pass into timing decoder, its calculation formula is: wherein, is a bias coefficient, is a sequence number of the query vector, is the i-th randomly initialized query vector, is the i-th calculated query vector, is the i-th value vector, is the i-th key vector, is a dimension of the vector, is a parameter weight of the three-layer feedforward network. ​ The formula expression for the time sequence decoder to track and detect the target by combining the information of the current frame and the historical frame through a time sequence model is: wherein Q represents a query vector obtained after calculation, represents 600 historical query vectors stored at the last time, and F represents the hidden features of the surround-view fisheye image.

7. The surround view fisheye camera based autonomous driving end-to-end trajectory prediction method according to claim 1, wherein, The process of inputting the object query after the feature interaction into the multi-layer perception and outputting the trajectory prediction result includes: The final object query output trajectory prediction result is output using a multi-layer perception, and the trajectory prediction result of each detected target includes multiple possible future trajectories, each trajectory containing multiple future unknown coordinates of the target within a future period of time, and the output coordinates include X-axis and Y-axis coordinates in a map coordinate system The object query is saved as timing information at a subsequent time.

Images