Moving trajectory prediction method and device, electronic equipment and storage medium

Abstract

The invention provides a moving trajectory prediction method and device, electronic equipment and a storage medium. The method comprises the following steps: inputting a multi-view image sensed by a target vehicle and target data associated with the target vehicle into an end-to-end automatic driving model, wherein the end-to-end automatic driving model comprises a motion planning module and an image feature pyramid module; obtaining planning characteristics of the target vehicle in a target area, wherein the planning characteristics are obtained by processing the multi-view image by the motion planning module; projecting three-dimensional information corresponding to the planning features to a two-dimensional image based on camera parameters of a camera corresponding to the multi-view image to obtain two-dimensional projection features; and calling the image feature pyramid module to process the two-dimensional projection features according to the target data so as to obtain a predicted running track of the target vehicle. According to the invention, information errors or loss can be avoided, the accuracy of vehicle moving trajectory planning is improved, and the vehicle driving risk is reduced.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method, device, electronic device, and storage medium for predicting a running trajectory. Background Art

[0002] In the field of autonomous driving, sparsedrive can extract features from images and then interact with vehicle planning, obstacle prediction, and 2D space image features to obtain the environment surrounding the trajectory planning task.

[0003] However, relying solely on the interaction between obstacle prediction features and image features during planning, while ignoring the use of camera parameter information, can lead to information errors or loss. Current systems do not focus on key information related to planning and obstacle prediction, but instead process all input information indiscriminately. This causes valid information to be buried in a large amount of irrelevant data, resulting in low vehicle trajectory planning accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, electronic device, and storage medium for predicting vehicle trajectory to solve the problem that the prior art may cause information errors or losses and the accuracy of vehicle trajectory planning is low.

[0005] In order to solve the above technical problems, the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for predicting a running trajectory, comprising:

[0007] Inputting the multi-view images perceived by the target vehicle and the target data associated with the target vehicle into an end-to-end autonomous driving model, the end-to-end autonomous driving model comprising: a motion planning module and an image feature pyramid module;

[0008] Acquiring planning features of the target vehicle within the target area obtained by processing the multi-view images by the motion planning module;

[0009] Based on camera parameters of a camera corresponding to the multi-view image, projecting the three-dimensional information corresponding to the planning feature onto a two-dimensional image to obtain a two-dimensional projection feature;

[0010] The image feature pyramid module is called to process the two-dimensional projection features according to the target data to obtain a predicted running trajectory of the target vehicle.

[0011] Optionally, the acquiring, by processing the multi-view images by the motion planning module, of planning features of the target vehicle within the target area includes:

[0012] Acquire the planned trajectory features of the target vehicle in the target area and the obstacle prediction features of the obstacles obtained by processing the multi-view images by the motion planning module;

[0013] Widen the planned trajectory of the target vehicle within the target area according to the set widening value to obtain an expanded trajectory feature;

[0014] The planned trajectory feature, the obstacle prediction feature and the extended trajectory feature are used as the planning feature.

[0015] Optionally, projecting the three-dimensional information corresponding to the planning feature onto a two-dimensional image based on camera parameters of a camera corresponding to the multi-view image to obtain a two-dimensional projection feature includes:

[0016] Decoding the planned trajectory feature, the obstacle prediction feature, and the extended trajectory feature respectively to obtain corresponding planned trajectory decoding features, obstacle decoding features, and extended trajectory decoding features;

[0017] Based on the camera parameters of the camera, the planned trajectory decoding features, the obstacle decoding features, and the extended trajectory decoding features are respectively projected onto a two-dimensional image to obtain a two-dimensional trajectory projection feature, a two-dimensional obstacle feature, and a two-dimensional extended trajectory feature;

[0018] The two-dimensional trajectory projection feature, the two-dimensional obstacle feature and the two-dimensional extended trajectory feature are used as the two-dimensional projection feature.

[0019] Optionally, the image feature pyramid module includes: a temporal cross attention layer, a map cross attention layer, an image deformable attention layer, a bird's-eye view deformable attention layer, and a future deformable attention layer.

[0020] The calling of the image feature pyramid module to process the two-dimensional projection features according to the target data to obtain a predicted running trajectory of the target vehicle includes:

[0021] Calling the temporal cross attention layer to process the two-dimensional projection features based on historical obstacle information and historical host vehicle feature information to obtain temporal features;

[0022] Calling the map cross attention layer to perform cross attention processing on the temporal features based on the current map data output by the map perception module to obtain cross attention features;

[0023] Calling the image deformable attention layer to process the cross attention feature to obtain an attention feature;

[0024] Calling the bird's-eye view deformable attention layer to perform perspective conversion processing on the attention feature based on vehicle trajectory query information to obtain a perspective conversion feature;

[0025] The future deformable attention layer is called to process the perspective conversion features to obtain the predicted running trajectories of the target vehicle and obstacles.

[0026] Optionally, after calling the image feature pyramid module to process the two-dimensional projection features according to the target data to obtain the predicted running trajectory of the target vehicle, the method further includes:

[0027] generating a driving control instruction corresponding to the target vehicle according to the predicted running trajectory of the target vehicle and the obstacle;

[0028] The driving of the target vehicle is controlled based on the driving maneuver instruction.

[0029] In a second aspect, an embodiment of the present application provides a running trajectory prediction device, comprising:

[0030] a data input module, configured to input the multi-view images perceived by the target vehicle and target data associated with the target vehicle into an end-to-end autonomous driving model, wherein the end-to-end autonomous driving model includes a motion planning module and an image feature pyramid module;

[0031] a planning feature acquisition module, configured to acquire planning features of the target vehicle within the target area, obtained by processing the multi-view images by the motion planning module;

[0032] a projection feature acquisition module, configured to project the three-dimensional information corresponding to the planning feature onto a two-dimensional image based on camera parameters of a camera corresponding to the multi-view image, to obtain a two-dimensional projection feature;

[0033] The predicted trajectory acquisition module is used to call the image feature pyramid module to process the two-dimensional projection features according to the target data to obtain the predicted running trajectory of the target vehicle.

[0034] Optionally, the planning feature acquisition module includes:

[0035] a feature acquisition unit, configured to acquire the planned trajectory features of the target vehicle in the target area and the obstacle prediction features of the obstacles obtained by processing the multi-view images by the motion planning module;

[0036] an expansion feature acquisition unit, configured to perform expansion processing on the planned trajectory of the target vehicle within the target area according to a set expansion value to obtain an expanded trajectory feature;

[0037] A planning feature acquisition unit is configured to use the planning trajectory feature, the obstacle prediction feature, and the extended trajectory feature as the planning feature.

[0038] Optionally, the projection feature acquisition module includes:

[0039] A decoding feature acquisition unit, configured to decode the planned trajectory feature, the obstacle prediction feature, and the extended trajectory feature, respectively, to obtain corresponding planned trajectory decoding features, obstacle decoding features, and extended trajectory decoding features;

[0040] a two-dimensional feature acquisition unit, configured to project the planned trajectory decoding feature, the obstacle decoding feature, and the extended trajectory decoding feature onto a two-dimensional image based on camera parameters of the camera, to obtain a two-dimensional trajectory projection feature, a two-dimensional obstacle feature, and a two-dimensional extended trajectory feature;

[0041] The projection feature acquisition unit is configured to use the two-dimensional trajectory projection feature, the two-dimensional obstacle feature, and the two-dimensional extended trajectory feature as the two-dimensional projection feature.

[0042] Optionally, the image feature pyramid module includes: a temporal cross attention layer, a map cross attention layer, an image deformable attention layer, a bird's-eye view deformable attention layer, and a future deformable attention layer.

[0043] The predicted trajectory acquisition module includes:

[0044] A temporal feature acquisition unit, configured to call the temporal cross attention layer to process the two-dimensional projection features based on historical obstacle information and historical host vehicle feature information to obtain temporal features;

[0045] a cross-feature acquisition unit, configured to call the map cross-attention layer to perform cross-attention processing on the temporal features based on the current map data output by the map perception module, to obtain cross-attention features;

[0046] an attention feature acquisition unit, configured to call the image deformable attention layer to process the cross attention feature to obtain an attention feature;

[0047] a conversion feature acquisition unit, configured to call the bird's-eye view deformable attention layer to perform perspective conversion processing on the attention feature based on vehicle trajectory query information to obtain a perspective conversion feature;

[0048] A predicted trajectory acquisition unit is used to call the future deformable attention layer to process the perspective conversion feature to obtain the predicted running trajectory of the target vehicle and the obstacle.

[0049] Optionally, the device further comprises:

[0050] An operation instruction generating module, configured to generate a driving control instruction corresponding to the target vehicle according to the predicted running trajectory of the target vehicle and the obstacle;

[0051] A vehicle driving control module is used to control the driving of the target vehicle based on the driving control instruction.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, comprising:

[0053] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements any one of the above-described methods for predicting running trajectories when executed by the processor.

[0054] In a fourth aspect, an embodiment of the present application provides a readable storage medium, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute any of the above-mentioned operation trajectory prediction methods.

[0055] In an embodiment of the present application, the multi-perspective images perceived by the target vehicle and the target data associated with the target vehicle are input into an end-to-end autonomous driving model, and the end-to-end autonomous driving model includes: a motion planning module and an image feature pyramid module. The motion planning module processes the multi-perspective images to obtain the planning features of the target vehicle in the target area. Based on the camera parameters of the camera corresponding to the multi-perspective image, the three-dimensional information corresponding to the planning features is projected onto the two-dimensional image to obtain the two-dimensional projection features. The image feature pyramid module is called to process the two-dimensional projection features according to the target data to obtain the predicted running trajectory of the target vehicle. The embodiment of the present application uses camera parameters to map the planning features onto a two-dimensional image, which can avoid information errors or losses, can efficiently obtain the target area information of the future trajectory, improve the accuracy of vehicle running trajectory planning, and reduce vehicle driving risks.

[0056] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 A flowchart of a method for predicting a running trajectory according to an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of an end-to-end autonomous driving model provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of a vehicle planning trajectory information query provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the structure of a running trajectory prediction device provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0064] Reference Figure 1 , shows a flow chart of the steps of a running trajectory prediction method provided by an embodiment of the present application. Figure 1 As shown, the running trajectory prediction method may include: step 101, step 102, step 103 and step 104.

[0065] Step 101: Input the multi-perspective image perceived by the target vehicle and the target data associated with the target vehicle into an end-to-end autonomous driving model, where the end-to-end autonomous driving model includes: a motion planning module and an image feature pyramid module.

[0066] In this embodiment, the application of end-to-end autonomous driving models in the field of autonomous driving has gradually become a focus of industry exploration. End-to-end autonomous driving models are deep learning models that directly map raw input data (such as sensor data) to output results (such as driving instructions). In this example, the end-to-end autonomous driving model can be, but is not limited to, SparseDrive.

[0067] This end-to-end autonomous driving model includes a motion planning module and an image feature pyramid module. The image feature pyramid module is a technique commonly used in computer vision tasks, particularly in object detection, image segmentation, and target tracking. The motion planning module is the core component of the autonomous driving model, responsible for formulating the vehicle's driving strategy based on perception and prediction information and generating detailed driving trajectories.

[0068] When planning the trajectory of a target vehicle, multi-perspective images perceived by the target vehicle and target data associated with the target vehicle (which may include map data, vehicle speed, acceleration, etc.) can be obtained. Then, the multi-perspective images perceived by the target vehicle and the target data associated with the target vehicle can be input into an end-to-end autonomous driving model.

[0069] After the multi-view images perceived by the target vehicle and the target data associated with the target vehicle are input into the end-to-end autonomous driving model, step 102 is performed.

[0070] Step 102: Acquire planning features of the target vehicle within the target area obtained by processing the multi-view images by the motion planning module.

[0071] After the multi-view images perceived by the target vehicle and the target data associated with the target vehicle are input into the end-to-end autonomous driving model, the motion planning module can process the multi-view images to obtain the planning features of the target vehicle within the target area. The target area can be a custom area, i.e., an area used for obstacle recognition.

[0072] In practice, the motion planning module is the core component of the autonomous driving model. It is responsible for formulating the vehicle's driving strategy based on perceived and predicted information and generating a detailed driving trajectory. The following are the main functions and features of this module:

[0073] 1. Path search and decision-making:

[0074] The motion planning module receives the feature representation from the image feature pyramid module and the target data associated with the target vehicle (such as position, speed, etc.).

[0075] Based on this input information, the module performs path search and decision-making to select the best driving path.

[0076] 2. Trajectory generation and optimization:

[0077] After determining the driving path, the motion planning module generates a detailed driving trajectory, including control instructions such as speed, acceleration, and steering.

[0078] By optimizing the algorithm, the trajectory is smoothed to ensure that the vehicle can reach its destination safely and efficiently.

[0079] 3. Consider environmental constraints:

[0080] The motion planning module takes into account various environmental constraints when generating trajectories, such as road conditions, traffic regulations, and obstacle locations.

[0081] These constraints help ensure that the generated trajectories both comply with traffic regulations and avoid collisions with other vehicles or obstacles.

[0082] 4. Real-time and robustness:

[0083] The motion planning module needs to have high real-time performance to meet the real-time requirements of the autonomous driving system.

[0084] At the same time, the module also needs to be highly robust and able to cope with various complex and uncertain traffic scenarios.

[0085] In this embodiment, the planning features of the target vehicle in the target area may include: the planned trajectory features of the vehicle and the predicted features of obstacles. The acquisition process of the planning features can be described in detail in conjunction with the following specific implementation methods.

[0086] In a specific implementation of the present application, the above step 102 may include:

[0087] Sub-step A1: Acquire the motion planning module to process the multi-view images to obtain the planned trajectory features of the target vehicle in the target area and the obstacle prediction features of the obstacles.

[0088] In this embodiment, the planned trajectory feature refers to the future driving path of the target vehicle calculated based on the current state, target location, traffic regulations and other factors.

[0089] Obstacle prediction features involve the prediction of other vehicles, pedestrians, stationary objects, etc. in the surrounding environment, including their position, speed, direction, etc., to assess whether they may constitute obstacles.

[0090] The motion planning module processes the multi-view images to obtain the planned trajectory characteristics of the target vehicle within the target area and the obstacle prediction characteristics of any obstacles. Specifically, the motion planning module processes these images to identify the planned trajectory characteristics of the target vehicle (i.e., the host vehicle or the vehicle of interest) within the target area, as well as any possible obstacles in the surrounding environment and their obstacle prediction characteristics.

[0091] After obtaining the planned trajectory features of the target vehicle in the target area and the obstacle prediction features of the obstacles, sub-step A2 is executed.

[0092] Sub-step A2: widening the planned trajectory of the target vehicle within the target area according to the set widening value to obtain an expanded trajectory feature.

[0093] The set widening value may be a pre-set value for widening the planned trajectory. The specific value of the set widening value may be set based on factors such as vehicle size, driving speed, road conditions, etc., and this embodiment does not impose any restrictions on this.

[0094] After obtaining the planned trajectory features of the target vehicle within the target area, the planned trajectory of the target vehicle within the target area can be widened according to the set widening value to obtain the extended trajectory features. In other words, the planned trajectory is expanded outward along the edge of the planned trajectory to form a wider trajectory area, which is the extended trajectory feature.

[0095] Sub-step A3: using the planned trajectory feature, the obstacle prediction feature, and the extended trajectory feature as the planned feature.

[0096] After the planned trajectory features, obstacle prediction features, and extended trajectory features are obtained, the planned trajectory features, obstacle prediction features, and extended trajectory features may be used as planning features for subsequent processing.

[0097] The embodiments of the present application extract key information from multi-view images, widen the planned trajectory of the target vehicle, and integrate obstacle prediction features to generate planning features for subsequent decision-making and control.

[0098] After obtaining the planning features of the target vehicle in the target area obtained by processing the multi-view images by the motion planning module, step 103 is executed.

[0099] Step 103: Based on the camera parameters of the camera corresponding to the multi-view image, project the three-dimensional information corresponding to the planning feature onto the two-dimensional image to obtain a two-dimensional projection feature.

[0100] Camera parameters are key data that describe the internal structure and external position of the camera, including intrinsic parameters (such as focal length, optical center position, etc.) and extrinsic parameters (such as the position and orientation of the camera relative to a reference coordinate system).

[0101] After obtaining the planning features of the target vehicle in the target area obtained by processing the multi-view images by the motion planning module, the three-dimensional information corresponding to the planning features can be projected onto the two-dimensional image based on the camera parameters of the camera corresponding to the multi-view images to obtain two-dimensional projection features. Specifically, the camera parameters, especially the intrinsic parameter matrix, can be used to transform the points in the three-dimensional space into the pixel coordinate system of the camera. This process usually involves mathematical transformations such as perspective projection or orthogonal projection. For each point or area in the planning features, this projection transformation is applied to obtain their corresponding positions on the two-dimensional image plane. After the projection transformation, the three-dimensional information in the planning features is converted into points, lines or areas on the two-dimensional image. These two-dimensional projection features can be combined with the original image data for further image processing, target detection or decision making.

[0102] The projection process may be described in detail in conjunction with the following specific implementation methods.

[0103] In a specific implementation of the present application, the above step 103 may include:

[0104] Sub-step B1: Decoding the planned trajectory feature, the obstacle prediction feature, and the extended trajectory feature respectively to obtain corresponding planned trajectory decoding features, obstacle decoding features, and extended trajectory decoding features.

[0105] In this embodiment, after obtaining the planned trajectory features, obstacle prediction features, and extended trajectory features, they can be decoded to obtain corresponding planned trajectory decoded features, obstacle decoded features, and extended trajectory decoded features. This process can be performed by decoding the planned trajectory features, obstacle prediction features, and extended trajectory features into physical points in three-dimensional space using certain network parameters. Specifically, the planned trajectory decoded features, obstacle decoded features, and extended trajectory decoded features are converted into three-dimensional space point features.

[0106] After obtaining the corresponding planned trajectory decoding features, obstacle decoding features, and extended trajectory decoding features, sub-step B2 is executed.

[0107] Sub-step B2: Based on the camera parameters of the camera, the planned trajectory decoding features, the obstacle decoding features, and the extended trajectory decoding features are respectively projected onto a two-dimensional image to obtain a two-dimensional trajectory projection feature, a two-dimensional obstacle feature, and a two-dimensional extended trajectory feature.

[0108] After obtaining the corresponding planned trajectory decoded features, obstacle decoded features, and extended trajectory decoded features, they can be projected onto the 2D image based on the camera parameters, resulting in 2D trajectory projection features, 2D obstacle features, and 2D extended trajectory features. Camera parameters are the key bridge between the 3D world and the 2D image. They describe the camera's internal optical properties and external position and orientation. Using these parameters, the decoded 3D planned trajectory features, obstacle prediction features, and extended trajectory features can be projected onto the 2D image plane.

[0109] Sub-step B3: using the two-dimensional trajectory projection feature, the two-dimensional obstacle feature, and the two-dimensional extended trajectory feature as the two-dimensional projection feature.

[0110] After obtaining the two-dimensional trajectory projection features, the two-dimensional obstacle features, and the two-dimensional extended trajectory features, the two-dimensional trajectory projection features, the two-dimensional obstacle features, and the two-dimensional extended trajectory features may be used as two-dimensional projection features.

[0111] The present embodiment decodes the planned trajectory features, obstacle prediction features, and extended trajectory features and projects them onto a two-dimensional image to generate a series of two-dimensional projection features for the autonomous driving system. These features provide comprehensive environmental perception information for the autonomous vehicle, contributing to safer and more efficient autonomous driving.

[0112] After projecting the three-dimensional information corresponding to the planning features onto the two-dimensional image to obtain the two-dimensional projection features based on the camera parameters of the cameras corresponding to the multi-view images, step 104 is executed.

[0113] Step 104: calling the image feature pyramid module to process the two-dimensional projection features according to the target data to obtain the predicted running trajectory of the target vehicle.

[0114] After projecting the three-dimensional information corresponding to the planning features onto the two-dimensional image to obtain the two-dimensional projection features based on the camera parameters of the camera corresponding to the multi-view image, the image feature pyramid module can be called to process the two-dimensional projection features according to the target data to obtain the predicted running trajectory of the target vehicle.

[0115] In this embodiment, the image feature pyramid module obtains 3D spatial information. Specifically, the target 3D point to be queried is mapped to 2D image features using camera internal and external parameters and the deformable_attn module is used to extract information, thereby efficiently obtaining target area information. The improvements of this embodiment are:

[0116] 1. Obtain the intermediate results of the ego-vehicle planning features, output the planning results and decode them, project the decoded results into the image feature pyramid as deformable_attn, and output the features as the query for the next stage of planning.

[0117] 2. Obtain the intermediate results of the obstacle prediction features, output the obstacle prediction results and decode them, project the decoded results into the image feature pyramid as deformable_attn, and output the features as the query for the next stage.

[0118] 3. Obtain the intermediate results of the ego-vehicle planning features, and additionally output the decoded trajectory of the planning results. Expand the result trajectory vertically to a certain width on both sides, project the expanded result onto the image feature pyramid as deformable_attn, and output the features as the query for the next stage of planning.

[0119] The processing flow of the image feature pyramid module can be described in detail in conjunction with the following specific implementation methods.

[0120] In a specific implementation of the present application, the image feature pyramid module includes: a temporal cross-attention layer, a map cross-attention layer, an image deformable attention layer, a bird's-eye view deformable attention layer, and a future deformable attention layer. The above step 104 may include:

[0121] Sub-step C1: calling the temporal cross attention layer to process the two-dimensional projection features based on historical obstacle information and historical main vehicle feature information to obtain temporal features.

[0122] In this embodiment, the temporal cross attention layer is Figure 2 The Temporal Cross Attention mechanism shown in Figure 1 allows the model to interact between features at different time points, thereby capturing temporal dependencies. In autonomous driving, this helps the system understand the trajectory and speed changes of objects.

[0123] That is, after obtaining the two-dimensional projection features, the temporal cross attention layer can be called to process the two-dimensional projection features based on the historical obstacle information and the historical main vehicle feature information to obtain the temporal features.

[0124] Sub-step C2: calling the map cross-attention layer to perform cross-attention processing on the temporal features based on the current map data output by the map perception module to obtain cross-attention features.

[0125] The map cross attention layer is Figure 2MAP Cross Attention in

[15] , this mechanism may be used in specific scenarios of autonomous driving (such as map annotation scenarios) to perform cross-attention calculations on information from different sources (such as sensor data and map data) to improve the accuracy of scene understanding.

[0126] After obtaining the temporal features, the map cross-attention layer can be called to perform cross-attention processing on the temporal features based on the current map data output by the map perception module to obtain cross-attention features.

[0127] Sub-step C3: Call the image deformable attention layer to process the cross-attention feature to obtain the attention feature.

[0128] Image deformable attention layer is Figure 2 Image Deformable Attention in

[15] focuses on key sampling point features in an image, allowing the model to more flexibly focus on important areas in the image. In autonomous driving, this helps the system more accurately identify key elements such as roads, vehicles, and pedestrians.

[0129] After obtaining the cross-attention features, the image deformable attention layer can be called to process the cross-attention features to obtain the attention features.

[0130] Sub-step C4: calling the deformable attention layer of the bird's-eye view to perform perspective conversion processing on the attention feature based on the vehicle trajectory query information to obtain a perspective conversion feature.

[0131] Bird's-eye view deformable attention layer Figure 2 The BEV Deformable Attention mechanism in [1] transforms image features into a bird's-eye view and applies a deformable attention mechanism for feature aggregation. This helps autonomous driving systems understand the surrounding environment from multiple perspectives, improving the accuracy of tasks such as 3D object detection and map semantic segmentation.

[0132] After obtaining the attention features, the bird's-eye view deformable attention layer can be called to perform perspective conversion processing on the attention features based on the vehicle trajectory query information to obtain perspective conversion features.

[0133] Sub-step C5: calling the future deformable attention layer to process the perspective conversion features to obtain the predicted running trajectories of the target vehicle and obstacles.

[0134] The future deformable attention layer is Figure 2 Future Deformable Attention in

[15] can be used to predict key feature changes in the future, thereby assisting the decision-making and planning of the autonomous driving system.

[0135] After obtaining the perspective conversion features, the future deformable attention layer can be called to process the perspective conversion features to obtain the predicted running trajectories of the target vehicle and obstacles.

[0136] like Figure 2 As shown, the model can also include: Self Attention: The self-attention mechanism allows the model to automatically find the association between each element in the sequence when processing sequential data (such as images or text, etc.). In autonomous driving, this helps the system capture the association between objects in the image and enhance the accuracy of target detection and image classification. Motion Prediction: This module is responsible for predicting the motion trajectory of surrounding objects, including state information such as speed and direction. This is a key function for autonomous driving systems to plan safe driving paths and avoid collisions. Plan Prediction: It can predict the future driving path of the vehicle, which helps the autonomous driving system plan the driving route in advance to avoid congestion and dangerous situations. ConvGRU (Convolutional Gated Recurrent Unit) is a model that combines convolutional neural networks (CNN) and gated recurrent units (GRU) to process spatiotemporal data. In autonomous driving, it can be used to capture spatiotemporal features in video sequences and improve the vehicle's ability to adapt to environmental changes. World Model is an important concept in autonomous driving systems. It represents the system's understanding and representation of the surrounding environment. By integrating multiple sensor data and algorithm outputs, the world model provides the autonomous driving system with a comprehensive and dynamic view of the environment, which helps the system make more accurate decisions and planning.

[0137] The practical application of the above end-to-end autonomous driving model can be as follows Figure 3 As shown, the image feature pyramid module provided by this embodiment can obtain whether the planned trajectory points in the end-to-end model training phase are reasonable, query whether the vehicle is in the middle of the lane, and whether the predicted trajectory points are reasonable. It can also flexibly insert information about the location you want to view. Figure 3 The coordinates of the planned trajectory points shown in the figure can be used to obtain the look future (prospect information within the area of the future planned trajectory) through deformable_attn.

[0138] After obtaining the predicted running trajectory of the target vehicle, the target vehicle can be controlled according to the predicted running trajectory. Specifically, the corresponding driving control instructions of the target vehicle can be generated according to the predicted running trajectory of the target vehicle and obstacles, and the driving of the target vehicle can be controlled based on the driving control instructions. In actual applications, the central computing module of the vehicle can plan the best path according to the predicted running trajectory in combination with the navigation map and real-time traffic data, and according to the planned path, the control module of the autonomous driving vehicle will issue instructions. These instructions include target values of vehicle actuators such as throttle, brakes and steering. The generation of instructions may need to consider multiple factors, such as the current state of the vehicle, road conditions, traffic rules, etc. Finally, the specific actions of the vehicle can be controlled by the actuators of the vehicle (such as steering EPS, braking ABS, etc.) according to the received instructions.

[0139] The process of generating corresponding driving control instructions and controlling driving based on the predicted trajectories of the target vehicle and obstacles in this application embodiment is a highly integrated and complex process. It relies on advanced sensor technology, high-speed computer processors, and complex algorithms and artificial intelligence technologies. By continuously optimizing the performance of each link, such as perception, prediction, planning, and control, autonomous vehicles will be able to provide travel services more safely and efficiently.

[0140] The operation trajectory prediction method provided by the embodiment of the present application is to input the multi-perspective images perceived by the target vehicle and the target data associated with the target vehicle into the end-to-end autonomous driving model, and the end-to-end autonomous driving model includes: a motion planning module and an image feature pyramid module. The planning features of the target vehicle in the target area are obtained by processing the multi-perspective images by the motion planning module. Based on the camera parameters of the camera corresponding to the multi-perspective image, the three-dimensional information corresponding to the planning features is projected onto the two-dimensional image to obtain the two-dimensional projection features. The image feature pyramid module is called to process the two-dimensional projection features according to the target data to obtain the predicted operation trajectory of the target vehicle. The embodiment of the present application uses camera parameters to map the planning features onto the two-dimensional image, which can avoid information errors or losses, can efficiently obtain the target area information of the future trajectory, improve the accuracy of vehicle operation trajectory planning, and reduce vehicle driving risks.

[0141] Reference Figure 4 , which shows a schematic structural diagram of a running trajectory prediction device provided by an embodiment of the present application, such as Figure 4 As shown, the running trajectory prediction device 400 may include the following modules:

[0142] a data input module 410 for inputting the multi-view images perceived by the target vehicle and target data associated with the target vehicle into an end-to-end autonomous driving model, wherein the end-to-end autonomous driving model includes a motion planning module and an image feature pyramid module;

[0143] A planning feature acquisition module 420 is configured to acquire planning features of the target vehicle within the target area obtained by processing the multi-view images by the motion planning module;

[0144] A projection feature acquisition module 430 is configured to project the three-dimensional information corresponding to the planning feature onto a two-dimensional image based on camera parameters of the camera corresponding to the multi-view image to obtain a two-dimensional projection feature;

[0145] The predicted trajectory acquisition module 440 is configured to call the image feature pyramid module to process the two-dimensional projection features according to the target data to obtain the predicted running trajectory of the target vehicle.

[0146] Optionally, the planning feature acquisition module includes:

[0147] a feature acquisition unit, configured to acquire the planned trajectory features of the target vehicle in the target area and the obstacle prediction features of the obstacles obtained by processing the multi-view images by the motion planning module;

[0148] an expansion feature acquisition unit, configured to perform expansion processing on the planned trajectory of the target vehicle within the target area according to a set expansion value to obtain an expanded trajectory feature;

[0149] A planning feature acquisition unit is configured to use the planning trajectory feature, the obstacle prediction feature, and the extended trajectory feature as the planning feature.

[0150] Optionally, the projection feature acquisition module includes:

[0151] A decoding feature acquisition unit, configured to decode the planned trajectory feature, the obstacle prediction feature, and the extended trajectory feature, respectively, to obtain corresponding planned trajectory decoding features, obstacle decoding features, and extended trajectory decoding features;

[0152] a two-dimensional feature acquisition unit, configured to project the planned trajectory decoding feature, the obstacle decoding feature, and the extended trajectory decoding feature onto a two-dimensional image based on camera parameters of the camera, to obtain a two-dimensional trajectory projection feature, a two-dimensional obstacle feature, and a two-dimensional extended trajectory feature;

[0153] The projection feature acquisition unit is configured to use the two-dimensional trajectory projection feature, the two-dimensional obstacle feature, and the two-dimensional extended trajectory feature as the two-dimensional projection feature.

[0154] Optionally, the image feature pyramid module includes: a temporal cross attention layer, a map cross attention layer, an image deformable attention layer, a bird's-eye view deformable attention layer, and a future deformable attention layer.

[0155] The predicted trajectory acquisition module includes:

[0156] A temporal feature acquisition unit, configured to call the temporal cross attention layer to process the two-dimensional projection features based on historical obstacle information and historical host vehicle feature information to obtain temporal features;

[0157] a cross-feature acquisition unit, configured to call the map cross-attention layer to perform cross-attention processing on the temporal features based on the current map data output by the map perception module, to obtain cross-attention features;

[0158] an attention feature acquisition unit, configured to call the image deformable attention layer to process the cross attention feature to obtain an attention feature;

[0159] a conversion feature acquisition unit, configured to call the bird's-eye view deformable attention layer to perform perspective conversion processing on the attention feature based on vehicle trajectory query information to obtain a perspective conversion feature;

[0160] A predicted trajectory acquisition unit is used to call the future deformable attention layer to process the perspective conversion feature to obtain the predicted running trajectory of the target vehicle and the obstacle.

[0161] Optionally, the device further comprises:

[0162] An operation instruction generating module, configured to generate a driving control instruction corresponding to the target vehicle according to the predicted running trajectory of the target vehicle and the obstacle;

[0163] A vehicle driving control module is used to control the driving of the target vehicle based on the driving control instruction.

[0164] The trajectory prediction device provided by the embodiment of the present application inputs the multi-perspective images perceived by the target vehicle and the target data associated with the target vehicle into an end-to-end autonomous driving model, and the end-to-end autonomous driving model includes: a motion planning module and an image feature pyramid module. The planning features of the target vehicle in the target area are obtained by processing the multi-perspective images by the motion planning module. Based on the camera parameters of the camera corresponding to the multi-perspective image, the three-dimensional information corresponding to the planning features is projected onto the two-dimensional image to obtain the two-dimensional projection features. The image feature pyramid module is called to process the two-dimensional projection features according to the target data to obtain the predicted trajectory of the target vehicle. The embodiment of the present application uses camera parameters to map the planning features onto a two-dimensional image, which can avoid information errors or losses, can efficiently obtain the target area information of the future trajectory, improve the accuracy of vehicle trajectory planning, and reduce vehicle driving risks.

[0165] An embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-mentioned operation trajectory prediction method when executed by the processor.

[0166] Figure 5 FIG. 5 shows a schematic structural diagram of an electronic device 500 according to an embodiment of the present invention. Figure 5 As shown, electronic device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 502 or computer program instructions loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of electronic device 500 can also be stored in RAM 503. CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to bus 504.

[0167] Multiple components in the electronic device 500 are connected to the I / O interface 505, including an input unit 506, such as a keyboard, a mouse, a microphone, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0168] The various processes and processing described above may be performed by the processing unit 501. For example, the method of any of the above embodiments may be implemented as a computer software program, which is tangibly contained in a computer-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the CPU 501, one or more actions in the method described above may be performed.

[0169] The present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the various processes of the above-described operation trajectory prediction method embodiment are implemented, and the same technical effects are achieved. To avoid repetition, the details are not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0170] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0172] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0173] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0174] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0175] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or groups can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0178] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0179] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for predicting a running trajectory, characterized in that: include: Inputting the multi-view images perceived by the target vehicle and the target data associated with the target vehicle into an end-to-end autonomous driving model, the end-to-end autonomous driving model comprising: a motion planning module and an image feature pyramid module; Acquiring planning features of the target vehicle within the target area obtained by processing the multi-view images by the motion planning module; Based on camera parameters of a camera corresponding to the multi-view image, projecting the three-dimensional information corresponding to the planning feature onto a two-dimensional image to obtain a two-dimensional projection feature; The image feature pyramid module is called to process the two-dimensional projection features according to the target data to obtain a predicted running trajectory of the target vehicle.

2. The method according to claim 1, characterized in that The acquiring of planning features of the target vehicle within the target area obtained by processing the multi-view images by the motion planning module includes: Acquire the planned trajectory features of the target vehicle in the target area and the obstacle prediction features of the obstacles obtained by processing the multi-view images by the motion planning module; Widen the planned trajectory of the target vehicle within the target area according to the set widening value to obtain an expanded trajectory feature; The planned trajectory feature, the obstacle prediction feature and the extended trajectory feature are used as the planning feature.

3. The method according to claim 2, characterized in that The step of projecting the three-dimensional information corresponding to the planning feature onto a two-dimensional image based on camera parameters of a camera corresponding to the multi-view image to obtain a two-dimensional projection feature includes: Decoding the planned trajectory feature, the obstacle prediction feature, and the extended trajectory feature respectively to obtain corresponding planned trajectory decoding features, obstacle decoding features, and extended trajectory decoding features; Based on the camera parameters of the camera, the planned trajectory decoding features, the obstacle decoding features, and the extended trajectory decoding features are respectively projected onto a two-dimensional image to obtain a two-dimensional trajectory projection feature, a two-dimensional obstacle feature, and a two-dimensional extended trajectory feature; The two-dimensional trajectory projection feature, the two-dimensional obstacle feature and the two-dimensional extended trajectory feature are used as the two-dimensional projection feature.

4. The method according to claim 1, wherein The image feature pyramid module includes: a temporal cross attention layer, a map cross attention layer, an image deformable attention layer, a bird's-eye view deformable attention layer, and a future deformable attention layer. The calling of the image feature pyramid module to process the two-dimensional projection features according to the target data to obtain a predicted running trajectory of the target vehicle includes: Calling the temporal cross attention layer to process the two-dimensional projection features based on historical obstacle information and historical host vehicle feature information to obtain temporal features; Calling the map cross attention layer to perform cross attention processing on the temporal features based on the current map data output by the map perception module to obtain cross attention features; Calling the image deformable attention layer to process the cross attention feature to obtain an attention feature; Calling the bird's-eye view deformable attention layer to perform perspective conversion processing on the attention feature based on vehicle trajectory query information to obtain a perspective conversion feature; The future deformable attention layer is called to process the perspective conversion features to obtain the predicted running trajectories of the target vehicle and obstacles.

5. The method according to claim 4, characterized in that After calling the image feature pyramid module to process the two-dimensional projection features according to the target data to obtain the predicted running trajectory of the target vehicle, the method further includes: generating a driving control instruction corresponding to the target vehicle according to the predicted running trajectory of the target vehicle and the obstacle; The driving of the target vehicle is controlled based on the driving maneuver instruction.

6. A running trajectory prediction device, characterized in that: include: a data input module, configured to input the multi-view images perceived by the target vehicle and target data associated with the target vehicle into an end-to-end autonomous driving model, wherein the end-to-end autonomous driving model includes a motion planning module and an image feature pyramid module; a planning feature acquisition module, configured to acquire planning features of the target vehicle within the target area, obtained by processing the multi-view images by the motion planning module; a projection feature acquisition module, configured to project the three-dimensional information corresponding to the planning feature onto a two-dimensional image based on camera parameters of a camera corresponding to the multi-view image, to obtain a two-dimensional projection feature; The predicted trajectory acquisition module is used to call the image feature pyramid module to process the two-dimensional projection features according to the target data to obtain the predicted running trajectory of the target vehicle.

7. The device according to claim 6, characterized in that The planning feature acquisition module includes: a feature acquisition unit, configured to acquire the planned trajectory features of the target vehicle in the target area and the obstacle prediction features of the obstacles obtained by processing the multi-view images by the motion planning module; an expansion feature acquisition unit, configured to perform expansion processing on the planned trajectory of the target vehicle within the target area according to a set expansion value to obtain an expanded trajectory feature; A planning feature acquisition unit is configured to use the planning trajectory feature, the obstacle prediction feature, and the extended trajectory feature as the planning feature.

8. The device according to claim 7, characterized in that The projection feature acquisition module includes: A decoding feature acquisition unit, configured to decode the planned trajectory feature, the obstacle prediction feature, and the extended trajectory feature, respectively, to obtain corresponding planned trajectory decoding features, obstacle decoding features, and extended trajectory decoding features; a two-dimensional feature acquisition unit, configured to project the planned trajectory decoding feature, the obstacle decoding feature, and the extended trajectory decoding feature onto a two-dimensional image based on camera parameters of the camera, to obtain a two-dimensional trajectory projection feature, a two-dimensional obstacle feature, and a two-dimensional extended trajectory feature; The projection feature acquisition unit is configured to use the two-dimensional trajectory projection feature, the two-dimensional obstacle feature, and the two-dimensional extended trajectory feature as the two-dimensional projection feature.

9. The device according to claim 6, characterized in that The image feature pyramid module includes: a temporal cross attention layer, a map cross attention layer, an image deformable attention layer, a bird's-eye view deformable attention layer, and a future deformable attention layer. The predicted trajectory acquisition module includes: A temporal feature acquisition unit, configured to call the temporal cross attention layer to process the two-dimensional projection features based on historical obstacle information and historical host vehicle feature information to obtain temporal features; a cross-feature acquisition unit, configured to call the map cross-attention layer to perform cross-attention processing on the temporal features based on the current map data output by the map perception module, to obtain cross-attention features; an attention feature acquisition unit, configured to call the image deformable attention layer to process the cross attention feature to obtain an attention feature; a conversion feature acquisition unit, configured to call the bird's-eye view deformable attention layer to perform perspective conversion processing on the attention feature based on vehicle trajectory query information to obtain a perspective conversion feature; A predicted trajectory acquisition unit is used to call the future deformable attention layer to process the perspective conversion feature to obtain the predicted running trajectory of the target vehicle and the obstacle.

10. The device according to claim 9, characterized in that The device further comprises: An operation instruction generating module, configured to generate a driving control instruction corresponding to the target vehicle according to the predicted running trajectory of the target vehicle and the obstacle; A vehicle driving control module is used to control the driving of the target vehicle based on the driving control instruction.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the operation trajectory prediction method according to any one of claims 1 to 5.

12. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the operation trajectory prediction method according to any one of claims 1 to 5.

Images