Predicted trajectory optimization method, vehicle, electronic device and program product

Abstract

The invention provides a prediction trajectory optimization method, a vehicle, electronic equipment and a program product, and relates to the technical field of vehicles. According to the prediction trajectory optimization method, a local interaction mechanism is adopted, the initial prediction trajectory and the global aerial view feature are subjected to local interaction, the computing resource consumption is reduced, the computing efficiency is improved, meanwhile, the aerial view feature related to initial prediction trajectory optimization can be accurately captured, the accuracy of the local trajectory feature is improved, and the prediction trajectory optimization efficiency is improved. Furthermore, trajectory optimization is performed on the initial prediction trajectory based on the target state features and the local trajectory features to obtain a target prediction trajectory, and the precision of the target prediction trajectory is improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method for optimizing predicted trajectories, a vehicle, electronic devices, and software products. Background Technology

[0002] In autonomous driving technology, vehicles need to perceive their surroundings in real time and predict the future trajectories of other traffic participants (such as vehicles, pedestrians, bicycles, etc.) in order to make safe and efficient path planning and obstacle avoidance decisions.

[0003] In related technologies, end-to-end deep learning models are typically used to predict and optimize the future behavioral trajectories of vehicles and other traffic participants. Specifically, the initial bird's-eye view (BEV) features, along with the state information of the vehicle and other traffic participants (such as speed, acceleration, position, and orientation), are input into the end-to-end deep learning model for trajectory prediction. This yields the initial predicted trajectories of the vehicle and other traffic participants. Then, through feature fusion or attention mechanisms, the initial predicted trajectories are globally interacted with the global bird's-eye view features to optimize the accuracy of the initial predicted trajectories.

[0004] However, the inventors discovered that the above-mentioned optimization method, which involves global interaction between the initial predicted trajectory and global BEV features, suffers from low computational efficiency. Summary of the Invention

[0005] This application provides a method, vehicle, electronic device, and program product for optimizing predicted trajectories, in order to solve the problem of low computational efficiency in related technologies that involve global interaction between the initial predicted trajectory and global BEV features.

[0006] In a first aspect, embodiments of this application provide a method for optimizing a predicted trajectory, comprising: acquiring global bird's-eye view features, as well as an initial predicted trajectory and target state features of a target object; performing local interaction between the initial predicted trajectory and the global bird's-eye view features to generate local trajectory features; and optimizing the initial predicted trajectory based on the target state features and the local trajectory features to obtain a target predicted trajectory corresponding to the initial predicted trajectory.

[0007] In one possible implementation, the initial predicted trajectory is locally interacted with the global bird's-eye view features to generate local trajectory features, including: extracting local feature sampling regions from the global bird's-eye view features based on the coordinates of the trajectory points of the initial predicted trajectory; and fusing the features within the local feature sampling regions to generate local trajectory features.

[0008] In one possible implementation, based on the coordinates of the trajectory points of the initial predicted trajectory, a local feature sampling region is extracted from the global bird's-eye view features, including: mapping the coordinates of the trajectory points of the initial predicted trajectory to the global bird's-eye view features to obtain the pixel coordinates of each trajectory point in the global bird's-eye view features; and delineating the local feature sampling region in the global bird's-eye view features according to a preset specification, with the pixel coordinates as the center.

[0009] In one possible implementation, the initial predicted trajectory is optimized based on the target state features and local trajectory features to obtain the target predicted trajectory corresponding to the initial predicted trajectory. This includes: performing hierarchical fusion processing on the initial predicted trajectory, target state features, and local trajectory features to obtain a first trajectory fusion feature; and performing nonlinear processing on the first trajectory fusion feature to obtain the target predicted trajectory.

[0010] In one possible implementation, the initial predicted trajectory, target state features, and local trajectory features are subjected to hierarchical fusion processing to obtain a first trajectory fusion feature, including: fusing the initial predicted trajectory and target state features to obtain a second trajectory fusion feature; and fusing the second trajectory fusion feature and local trajectory features to obtain the first trajectory fusion feature.

[0011] In one possible implementation, obtaining global bird's-eye view features, as well as the initial predicted trajectory and target state features of the target object, includes: obtaining the initial bird's-eye view features and the initial state features of the target object; determining global bird's-eye view features based on the initial bird's-eye view features and the initial state features; extracting the state features of the target object from the global bird's-eye view features to obtain the target state features; and performing trajectory prediction on the target state features to obtain the initial predicted trajectory.

[0012] In one possible implementation, the target object includes a target vehicle. After obtaining the target predicted trajectory corresponding to the initial predicted trajectory, the optimization method of the predicted trajectory further includes: controlling the movement of the target vehicle according to the target predicted trajectory corresponding to the target vehicle.

[0013] Secondly, embodiments of this application provide an optimization apparatus for predicting trajectories, comprising:

[0014] The acquisition module is used to acquire global bird's-eye view features, as well as the initial predicted trajectory and target state features of the target object;

[0015] The generation module is used to interact locally with the initial predicted trajectory and global bird's-eye view features to generate local trajectory features;

[0016] The optimization module is used to optimize the initial predicted trajectory based on the target state features and local trajectory features, so as to obtain the target predicted trajectory corresponding to the initial predicted trajectory.

[0017] In one possible implementation, the generation module is specifically used to: extract local feature sampling regions from the global bird's-eye view features based on the coordinates of the trajectory points of the initial predicted trajectory; and perform fusion processing on the features within the local feature sampling regions to generate local trajectory features.

[0018] In one possible implementation, the generation module is further configured to: map the coordinates of the trajectory points of the initial predicted trajectory to the global bird's-eye view features, thereby obtaining the pixel coordinates of each trajectory point in the global bird's-eye view features; and delineate a local feature sampling area in the global bird's-eye view features according to a preset specification, with the pixel coordinates as the center.

[0019] In one possible implementation, the optimization module is specifically used to: perform hierarchical fusion processing on the initial predicted trajectory, target state features, and local trajectory features to obtain a first trajectory fusion feature; and perform nonlinear processing on the first trajectory fusion feature to obtain the target predicted trajectory.

[0020] In one possible implementation, the optimization module is further configured to: fuse the initial predicted trajectory and the target state features to obtain a second trajectory fusion feature; and fuse the second trajectory fusion feature and the local trajectory features to obtain a first trajectory fusion feature.

[0021] In one possible implementation, the acquisition module is used to: acquire initial bird's-eye view features and initial state features of the target object; determine global bird's-eye view features based on the initial bird's-eye view features and initial state features; extract the state features of the target object from the global bird's-eye view features to obtain target state features; and perform trajectory prediction on the target state features to obtain an initial predicted trajectory.

[0022] In one possible implementation, the target object includes a target vehicle. After obtaining the target predicted trajectory corresponding to the initial predicted trajectory, the optimization device for the predicted trajectory further includes a control module (not shown), which is used to control the movement of the target vehicle according to the target predicted trajectory corresponding to the target vehicle.

[0023] Thirdly, this application provides a vehicle, including: a vehicle body and a trajectory prediction system, the trajectory prediction system being used to implement the optimization method for predicting trajectories as provided in the first aspect above.

[0024] Fourthly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0025] The memory stores the instructions that the computer executes;

[0026] The processor executes computer execution instructions stored in memory to implement the method for optimizing the predicted trajectory as described in the first aspect above.

[0027] Fifthly, this application provides a mobile platform that includes the electronic device provided in the fourth aspect above.

[0028] In a sixth aspect, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the optimization method for predicting trajectories as provided in the first aspect above.

[0029] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method for optimizing the predicted trajectory as provided in the first aspect above.

[0030] The proposed trajectory optimization method, vehicle, electronic device, and program product provided in this application involve acquiring global bird's-eye view features, the initial predicted trajectory of the target object, and the target state features. The initial predicted trajectory is then locally interacted with the global bird's-eye view features to generate local trajectory features. Further trajectory optimization is performed on the initial predicted trajectory based on the target state features and the local trajectory features to obtain the target predicted trajectory. This application employs a local interaction mechanism, which reduces computational resource consumption and improves computational efficiency by locally interacting the initial predicted trajectory with the global bird's-eye view features. Simultaneously, it can accurately capture bird's-eye view features related to the initial predicted trajectory optimization, improving the accuracy of local trajectory features. Further trajectory optimization based on the target state features and local trajectory features yields the target predicted trajectory, thus enhancing the accuracy of the target predicted trajectory. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0032] Figure 1 This is an architecture diagram of the trajectory prediction system provided in the embodiments of this application;

[0033] Figure 2 A flowchart illustrating the method for optimizing predicted trajectories provided in this application embodiment. Figure 1 ;

[0034] Figure 3 A flowchart illustrating the method for optimizing predicted trajectories provided in this application embodiment. Figure 2 ;

[0035] Figure 4 A flowchart illustrating the method for optimizing predicted trajectories provided in this application embodiment. Figure 3 ;

[0036] Figure 5 A flowchart illustrating the method for optimizing predicted trajectories provided in this application embodiment. Figure 4 ;

[0037] Figure 6 A schematic diagram of the structure of the trajectory prediction optimization device provided in the embodiments of this application;

[0038] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0039] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] In related technologies, end-to-end deep learning models are typically used to predict and optimize the future behavioral trajectories of vehicles and other traffic participants. Specifically, initial bird's-eye view features, along with the state information of the vehicle and other traffic participants, are input into the end-to-end deep learning model for trajectory prediction, resulting in initial predicted trajectories for the vehicle and other traffic participants. Then, feature fusion or attention mechanisms are used to globally interact with the global bird's-eye view features to optimize the accuracy of the initial predicted trajectory. However, this global interaction between the initial predicted trajectory and the global bird's-eye view features through feature fusion or attention mechanisms has several drawbacks. First, it requires full processing of a large number of bird's-eye view features, leading to high computational resource consumption and low computational efficiency. Furthermore, it is difficult to adapt to the real-time requirements of automotive chips in scenarios where the computing power of automotive chips is limited. Second, the model needs to autonomously learn which regions in the global bird's-eye view features are critical for trajectory optimization, resulting in a time-consuming training process and unstable trajectory optimization effects, which may lead to inaccurate trajectory predictions.

[0042] Based on the technical problems existing in related technologies, the embodiments of this application adopt a local interaction mechanism. By locally interacting the initial predicted trajectory with the global bird's-eye view features, the consumption of computing resources is reduced and the computing efficiency is improved. At the same time, the bird's-eye view features related to the optimization of the initial predicted trajectory can be accurately captured, improving the accuracy of local trajectory features. Furthermore, the initial predicted trajectory is optimized based on the target state features and local trajectory features of the target object, thereby improving the accuracy of the optimized target predicted trajectory.

[0043] The application scenarios of the embodiments of this application will be described below first.

[0044] The trajectory prediction optimization method provided in this application is applicable to trajectory prediction and decision-making scenarios in autonomous driving systems, specifically to trajectory prediction and path planning scenarios in autonomous driving systems.

[0045] The following is combined Figure 1 The trajectory prediction system provided in the embodiments of this application will be described in detail.

[0046] Figure 1 This is an architecture diagram of the trajectory prediction system provided in an embodiment of this application. Figure 1 As shown, the trajectory prediction system includes a feature extraction module 11, a feature extraction module 12, a target detection module, a trajectory prediction module, a local interaction module, and a nonlinear layer.

[0047] The feature extraction module 11 is used to extract features from the initial bird's-eye view to obtain the first bird's-eye view features.

[0048] The second feature extraction module 12 is used to extract features from the initial state features of the target object (such as a car, other vehicles, or pedestrians) to obtain the first state features.

[0049] The target detection module is used to perform target detection on the second bird's-eye view features to obtain global bird's-eye view features. The second bird's-eye view features are obtained by fusing the first bird's-eye view features and the second state features. The second state features are obtained by embedding the first state features into a preset empty two-dimensional bird's-eye view features based on the position features corresponding to the target object. The dimension of the preset empty two-dimensional bird's-eye view features is the same as the dimension of the first bird's-eye view features.

[0050] The trajectory prediction module is used to predict the trajectory of the target state features and output an initial predicted trajectory set. The target state features are obtained by extracting the state features of the target object from the global bird's-eye view features based on the initial state features.

[0051] The local interaction module is used to locally interact with the initial predicted trajectory set and global bird's-eye view features to generate local trajectory features. The specific implementation of this local interaction between the initial predicted trajectory and global bird's-eye view features to generate local trajectory features is as follows: Figure 1 As shown in the dashed box 13.

[0052] The nonlinear layer is used to perform nonlinear processing on the first trajectory fusion feature obtained by layering and fusing the trajectory features, local trajectory features and target state features of the initial predicted trajectory, to obtain the optimized target predicted trajectory.

[0053] For example, the initial bird's-eye view features can be a three-dimensional feature tensor containing features of different semantic layers. For instance, the initial bird's-eye view features can be represented as (10+13+1+3)×208×128; where (10+13+1+3) represents the number of different semantic layers (such as 10 layers of road topology, 13 layers of lane lines, 1 layer of pedestrian crossings, 3 layers of curbs, etc.), and 208×128 represents the two-dimensional spatial grid size of the bird's-eye view.

[0054] For example, the initial predicted trajectory of the first bird's-eye view feature and the second state feature can be a multi-dimensional feature in the form of a three-dimensional tensor, such as a dimension of 64×208×128, where 64 represents the channel and 208×128 represents the two-dimensional spatial network size of the bird's-eye view.

[0055] For example, the second bird's-eye view feature and the global bird's-eye view feature can be multi-dimensional features in the form of a three-dimensional tensor, such as a dimension of 128×208×128.

[0056] For example, the initial state features can be a three-dimensional feature tensor containing the target objects, such as N_act×10×16, where N_act represents the number of target objects, 10 represents the time dimension (i.e., 10 frames of historical data), and 16 represents the feature dimension of each target object at each time step.

[0057] For example, the first state features, target state features, and local trajectory features can be two-dimensional feature tensors, such as N_act×64, where N_act represents the number of target objects and 64 represents the feature dimension.

[0058] For example, the initial predicted trajectory set can be a multi-dimensional feature in the form of a four-dimensional tensor, such as N_act×6×50×2, where N_act represents the number of target objects, 6 represents the number of modes of the initial predicted trajectory, 50 represents the number of trajectory points on each initial predicted trajectory, and 2 represents the two-dimensional coordinate dimension of each trajectory point.

[0059] from Figure 1As can be seen from the embodiments of this application, the trajectory prediction optimization method can be implemented by using multiple interconnected deep learning models or by using an end-to-end deep learning model to realize the complete process of trajectory prediction and trajectory optimization of the target object.

[0060] For example, the trajectory prediction system may include a multi-sensor fusion bird's-eye view perception system, which may be deployed on an onboard chip such as a graphics processing unit (GPU) or a dedicated AI acceleration chip.

[0061] The following is based on the above. Figure 1 The trajectory prediction system shown is the execution entity. In conjunction with specific embodiments, the specific implementation of the trajectory prediction optimization method provided in this application embodiment will be described in detail.

[0062] Figure 2 A flowchart illustrating the method for optimizing predicted trajectories provided in this application embodiment. Figure 1 .like Figure 2 As shown, a specific implementation of the trajectory prediction optimization method may include the following steps:

[0063] S201, Obtain global bird's-eye view features, as well as the initial predicted trajectory and target state features of the target object.

[0064] For example, the bird's-eye view feature can be a high-dimensional feature tensor (e.g., 128×208×128) obtained by projecting multi-view data collected by multiple sensors (such as cameras, lidar, millimeter-wave radar, etc.) integrated on the target vehicle onto a unified scene representation from a top-down perspective, i.e., the bird's-eye view, and extracting features based on a convolutional neural network.

[0065] For example, global bird's-eye view features can include different categories of semantic features, such as structured semantic features that characterize the environment (e.g., number of lanes, intersection shape, and lane connection relationships), basic geographic information semantic features (e.g., lane lines, pedestrian crossings, stop lines), and dynamic traffic participant semantic features (e.g., the location, category, and outline of traffic participants such as vehicles, pedestrians, and cyclists).

[0066] For example, the target object may include the target vehicle and other traffic participants adjacent to the target vehicle (such as pedestrians, cyclists, and other vehicles). Accordingly, the initial predicted trajectory of the target object may include the initial predicted trajectory of the target vehicle and the initial predicted trajectories of the other traffic participants.

[0067] For example, the initial predicted trajectory of the target vehicle may include multiple initial predicted trajectories of different modes. The different modes may include six modes such as going straight, turning left, turning right, making a U-turn, changing lanes to the left, and changing lanes to the right. The initial predicted trajectory corresponding to each mode may contain 50 trajectory points, and each trajectory point corresponds to a two-dimensional coordinate (x, y), that is, the trajectory point coordinates.

[0068] For example, the initial predicted trajectory of the target object can be used to represent the target object's travel path within a preset time period. The preset time period can be 5 seconds.

[0069] For example, the target state features of the target object can be a feature set obtained by fusing the target object's own state features and environmental semantic features; wherein, the target object's own state features include attributes such as orientation, speed, acceleration, and category, and the environmental semantic features include structured semantic features that characterize the environment (such as the number of lanes and the shape of the intersection) and basic geographic information semantic features (such as lane lines, pedestrian crossings, and stop lines).

[0070] The methods for obtaining global bird's-eye view features, target state features, and initial predicted trajectory are the same as those described above. Figure 1 Similar examples are shown below, and will not be repeated here.

[0071] S202, the initial predicted trajectory is locally interacted with the global bird's-eye view features to generate local trajectory features.

[0072] In this step, one possible implementation is as follows: input the initial predicted trajectory and global bird's-eye view features into the local interaction module, and use the local interaction mechanism to perform local interaction based on the trajectory point coordinates of the initial predicted trajectory and the spatial constraint information such as road topology, lane boundaries, and traffic facilities in the global bird's-eye view features, so as to obtain the local trajectory features of the initial predicted trajectory output by the local interaction module.

[0073] S203, optimize the initial predicted trajectory based on the target state characteristics and local trajectory characteristics to obtain the target predicted trajectory corresponding to the initial predicted trajectory.

[0074] In this step, one possible implementation is as follows: Feature extraction is performed on the initial predicted trajectory to obtain its trajectory features. Then, an attention fusion mechanism is employed to fuse the target state features with the trajectory features of the initial predicted trajectory and local trajectory features, resulting in the first trajectory fusion feature of the initial predicted trajectory. This first trajectory fusion feature is then input into... Figure 1 The nonlinear layer shown in the figure performs nonlinear processing by fitting the complex relationship between the trajectory and the environment and the state of the target object, and obtains the target predicted trajectory after optimization of the initial predicted trajectory.

[0075] For example, the nonlinear layer can be a multi-layer perceptron (MLP). An MLP can contain multiple linear layers, and these different linear layers are connected by activation functions (such as ReLU or Sigmoid). This application does not limit the number of linear layers in the MLP; it can be determined based on the specific application requirements.

[0076] This application adopts a local interaction mechanism. By interacting the initial predicted trajectory with the global bird's-eye view features locally, the consumption of computing resources is reduced and the computing efficiency is improved. At the same time, the bird's-eye view features related to the optimization of the initial predicted trajectory can be accurately captured, improving the accuracy of local trajectory features. Furthermore, the initial predicted trajectory is optimized based on the target state features and local trajectory features to obtain the target predicted trajectory, thereby improving the accuracy of the target predicted trajectory.

[0077] Understandably, compared to related technologies that require full processing of large-scale bird's-eye view features when globally interacting the initial predicted trajectory with global bird's-eye view features, resulting in high computational resource consumption and low computational efficiency, and are difficult to adapt to the real-time requirements of automotive chips in scenarios with limited computing power, the prediction trajectory optimization method provided in this application adopts a local interaction mechanism to locally interact the initial predicted trajectory with global bird's-eye view features to generate local trajectory features. This significantly improves computational efficiency, reduces computational resource consumption, and at the same time, has lower requirements for the computing power of automotive chips, improving compatibility with automotive chips and meeting the real-time requirements of autonomous driving systems.

[0078] The following is combined Figure 3 The specific implementation method of step S202, which involves locally interacting the initial predicted trajectory with the global bird's-eye view features to generate local trajectory features, is explained in detail.

[0079] Figure 3 A flowchart illustrating the method for optimizing predicted trajectories provided in this application embodiment. Figure 2 .like Figure 3 As shown, the optimization method for this predicted trajectory involves local interaction between the initial predicted trajectory and global bird's-eye view features to generate local trajectory features. A specific implementation of this method may include the following steps:

[0080] S301, based on the coordinates of the trajectory points of the initial predicted trajectory, extract the local feature sampling area from the global bird's-eye view features.

[0081] For example, the coordinates of the trajectory points of the initial predicted trajectory can be represented as (x, y), i.e., two-dimensional coordinates.

[0082] Optionally, this step may include the following steps:

[0083] S3011, map the coordinates of the trajectory points of the initial predicted trajectory to the global bird's-eye view features to obtain the pixel coordinates of each trajectory point in the global bird's-eye view features.

[0084] For example, when the initial predicted trajectory contains 50 trajectory points, mapping the initial predicted trajectory onto the global bird's-eye view feature yields 50 pixel coordinate points. By connecting all the pixel coordinate points in chronological order, the trajectory corresponding to the initial predicted trajectory can be obtained in the global bird's-eye view feature.

[0085] S3012, using pixel coordinates as the center, delineates local feature sampling areas in the global bird's-eye view features according to preset specifications.

[0086] For example, the preset specification can be a rectangular area of ​​a preset size. This application does not limit the preset specification; it can be determined according to actual application requirements.

[0087] In one possible implementation, the local sampling sub-region of each trajectory point coordinate can be defined in the global bird's-eye view feature according to a preset specification, with the pixel coordinate point corresponding to each trajectory point coordinate as the center. Then, the area synthesized from all layout sampling sub-regions can be used as the local sampling area of ​​the initial predicted trajectory.

[0088] In another possible implementation, the pixel coordinates corresponding to each trajectory point are connected according to the time sequence of the trajectory points to obtain the trajectory corresponding to the global bird's-eye view feature of the initial predicted trajectory. Then, a local feature sampling area is delineated in the global bird's-eye view feature according to a preset specification, with the trajectory as the center.

[0089] S302 performs fusion processing on features within the local feature sampling area to generate local trajectory features.

[0090] For example, the features within the local feature sampling area can be local bird's-eye view features (such as lane lines) that are related to the optimization of the initial predicted trajectory.

[0091] In one possible implementation, different weights are assigned to features within a local feature sampling region, and then the features within the local feature sampling region are subjected to weighted fusion processing. The weights of different features can be allocated based on the relative distance between the feature location and the pixel coordinates corresponding to the trajectory point (the closer the distance, the higher the weight), or the weights of features strongly correlated with the initial predicted trajectory can be learned through an attention network.

[0092] In this embodiment, local feature sampling areas are extracted from the global bird's-eye view features based on the coordinates of the trajectory points of the initial predicted trajectory. Then, the features within the local feature sampling areas are fused to generate local trajectory features, which reduces computational resource consumption and improves computational efficiency. At the same time, it can accurately capture bird's-eye view features related to the optimization of the initial predicted trajectory, thereby improving the accuracy of local trajectory features.

[0093] The following is combined Figure 4 The following is a detailed explanation of a specific implementation method for step S203, which optimizes the initial predicted trajectory based on the target state features and local trajectory features to obtain the target predicted trajectory corresponding to the initial predicted trajectory.

[0094] Figure 4 A flowchart illustrating the method for optimizing predicted trajectories provided in this application embodiment. Figure 3 .like Figure 4 As shown, the optimization method for the predicted trajectory optimizes the initial predicted trajectory based on the target state features and local trajectory features to obtain the target predicted trajectory corresponding to the initial predicted trajectory. A specific implementation of this method may include the following steps:

[0095] S401, perform hierarchical fusion processing on the initial predicted trajectory, target state features and local trajectory features to obtain the first trajectory fusion feature.

[0096] Optionally, in this step, one possible implementation is to fuse the initial predicted trajectory and the target state features to obtain the second trajectory fusion feature; and to fuse the second trajectory fusion feature and the local trajectory features to obtain the first trajectory fusion feature.

[0097] For example, as described above Figure 1 As shown, firstly, the initial predicted trajectory is extracted using a deep learning model such as a convolutional neural network to obtain the trajectory features corresponding to the initial predicted trajectory. Secondly, the trajectory features and the target state features are fused to obtain the second trajectory fusion features. Then, the second trajectory fusion features and the local trajectory features are fused to obtain the first trajectory fusion features.

[0098] For example, the feature fusion processing method can be splicing or element-by-element addition, etc. This application embodiment does not limit this, and the specific method can be determined according to the actual application requirements.

[0099] S402, perform nonlinear processing on the first trajectory fusion features to obtain the target predicted trajectory.

[0100] The specific implementation method is similar to that described above, and will not be repeated here.

[0101] In this embodiment, the initial predicted trajectory, target state features, and local trajectory features are fused in a hierarchical manner to obtain a first trajectory fusion feature. Then, the first trajectory fusion feature is subjected to nonlinear processing to obtain the target predicted trajectory, thereby improving the trajectory prediction accuracy and computational efficiency.

[0102] The following is combined Figure 5 A detailed explanation is provided on a specific implementation method for obtaining global bird's-eye view features, as well as the initial predicted trajectory and target state features of the target object in step S201.

[0103] Figure 5 A flowchart illustrating the method for optimizing predicted trajectories provided in this application embodiment. Figure 4 .like Figure 5 As shown, a specific implementation of the method for optimizing the predicted trajectory, which obtains global bird's-eye view features, as well as the initial predicted trajectory and target state features of the target object, may include the following steps:

[0104] S501, Obtain the initial bird's-eye view features and the initial state features of the target object.

[0105] For example, the initial bird's-eye view features may include the initial semantic features of different lane lines (such as solid lines, dashed lines, double yellow lines), the initial semantic features of different lane elements (such as lane boundaries, lane types, lane turning attributes, lane connection relationships, etc.), and the initial semantic features of dynamic traffic participants (such as the position, category, outline, etc. of traffic participants such as vehicles, pedestrians, cyclists, etc.).

[0106] For example, the initial state characteristics of the target object may include position, velocity, and orientation.

[0107] The method for obtaining the initial bird's-eye view features is similar to that described above, and will not be repeated here.

[0108] The initial state features of the target object can be obtained in the following way: obtain the state data of the target object, extract features from the state data through a deep learning model, and obtain the initial state features.

[0109] S502, Determine the global bird's-eye view features based on the initial bird's-eye view features and the initial state features.

[0110] As mentioned above Figure 1As shown, one possible implementation of this step is as follows: First, the initial bird's-eye view features are input into the feature extraction module 11 for feature extraction to obtain the first bird's-eye view features, and the initial state features are input into the feature extraction module 12 for feature extraction to obtain the first state features; second, the position features of the target object are extracted from the first state features, and the first state features are embedded into a preset empty two-dimensional bird's-eye view features based on the position features to obtain the second state features; then, the second state features and the first bird's-eye view features are fused to obtain the second bird's-eye view features; finally, the second bird's-eye view features are input into the target detection module, and the target detection is performed on the second bird's-eye view features based on the target detection model to obtain the global bird's-eye view features output by the target detection module.

[0111] S503 extracts the state features of the target object from the global bird's-eye view features to obtain the target state features.

[0112] It is understandable that the target state features are state features that incorporate global bird's-eye view features.

[0113] S504, perform trajectory prediction on the target state features to obtain the initial predicted trajectory.

[0114] As mentioned above Figure 1 As shown, one possible implementation of this step is to input the target state features into the trajectory prediction module, perform trajectory prediction based on the trajectory prediction model, and obtain the initial predicted trajectory output by the trajectory prediction module.

[0115] Optionally, in the trajectory optimization method provided in this application embodiment, when the target object includes a target vehicle, after obtaining the target predicted trajectory corresponding to the initial predicted trajectory, the trajectory optimization method further includes: controlling the target vehicle to move according to the target predicted trajectory corresponding to the target vehicle.

[0116] For example, the target predicted trajectory corresponding to the target vehicle can be the predicted trajectory obtained by optimizing the initial predicted trajectory of the target vehicle using the prediction trajectory optimization method provided in the embodiments of this application.

[0117] In summary, the trajectory prediction optimization method provided in this application can be embedded in any trajectory prediction and decision-making algorithm. Research shows that the trajectory prediction optimization method provided in this application can significantly improve trajectory prediction and decision-making metrics, such as Final Displacement Error (FDE) or Average Displacement Error (ADE), achieving a 10% improvement in prediction accuracy compared to baseline models such as LSTM-based trajectory prediction models.

[0118] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0119] Figure 6 This is a schematic diagram of the structure of the trajectory prediction optimization device provided in an embodiment of this application. Figure 6 As shown, the trajectory prediction optimization device 60 includes an acquisition module 610, a generation module 620, and an optimization module 630.

[0120] Among them, the acquisition module 610 is used to acquire global bird's-eye view features, as well as the initial predicted trajectory and target state features of the target object;

[0121] The generation module 620 is used to locally interact with the initial predicted trajectory and the global bird's-eye view features to generate local trajectory features;

[0122] The optimization module 630 is used to optimize the initial predicted trajectory based on the target state features and local trajectory features to obtain the target predicted trajectory corresponding to the initial predicted trajectory.

[0123] In one possible implementation, the generation module 620 is specifically used to: extract local feature sampling regions from the global bird's-eye view features based on the coordinates of the trajectory points of the initial predicted trajectory; and perform fusion processing on the features within the local feature sampling regions to generate local trajectory features.

[0124] In one possible implementation, the generation module 620 is further configured to: map the coordinates of the trajectory points of the initial predicted trajectory to the global bird's-eye view features to obtain the pixel coordinates of each trajectory point in the global bird's-eye view features; and delineate a local feature sampling area in the global bird's-eye view features according to a preset specification, with the pixel coordinates as the center.

[0125] In one possible implementation, the optimization module 630 is specifically used to: perform hierarchical fusion processing on the initial predicted trajectory, target state features, and local trajectory features to obtain a first trajectory fusion feature; and perform nonlinear processing on the first trajectory fusion feature to obtain the target predicted trajectory.

[0126] In one possible implementation, the optimization module 630 is further configured to: fuse the initial predicted trajectory and the target state features to obtain a second trajectory fusion feature; and fuse the second trajectory fusion feature and the local trajectory features to obtain a first trajectory fusion feature.

[0127] In one possible implementation, the acquisition module 610 is used to: acquire initial bird's-eye view features and initial state features of the target object; determine global bird's-eye view features based on the initial bird's-eye view features and initial state features; extract the state features of the target object from the global bird's-eye view features to obtain target state features; and perform trajectory prediction on the target state features to obtain an initial predicted trajectory.

[0128] In one possible implementation, the target object includes a target vehicle. After obtaining the target predicted trajectory corresponding to the initial predicted trajectory, the optimization device for the predicted trajectory further includes a control module (not shown), which is used to control the movement of the target vehicle according to the target predicted trajectory corresponding to the target vehicle.

[0129] The trajectory prediction optimization device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0130] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the electronic device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0131] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0132] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0133] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0134] The memory may include random access memory (RAM) and non-volatile memory (NVM), such as at least one disk storage device.

[0135] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0136] This application also provides a mobile platform, which includes electronic devices. Exemplary examples include vehicles, robotic platforms (such as server robots, exploratory robots, scientific research robots, etc.), drones, and other devices, but are not limited thereto.

[0137] This application also provides a vehicle, including: a vehicle body and a trajectory prediction system, the trajectory prediction system being used to implement the method described above. The trajectory prediction system is as described above. Figure 1 As shown in the figure, it will not be repeated here.

[0138] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0139] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0140] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0141] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0142] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0145] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0147] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An optimization method for predicting trajectories, characterized in that, include: Obtain global bird's-eye view features, as well as the initial predicted trajectory and target state features of the target object; The initial predicted trajectory is locally interacted with the global bird's-eye view features to generate local trajectory features; Based on the target state features and the local trajectory features, the initial predicted trajectory is optimized to obtain the target predicted trajectory corresponding to the initial predicted trajectory.

2. The method for optimizing predicted trajectories according to claim 1, characterized in that, The step of locally interacting the initial predicted trajectory with the global bird's-eye view features to generate local trajectory features includes: Based on the coordinates of the trajectory points of the initial predicted trajectory, local feature sampling regions are extracted from the global bird's-eye view features; The features within the local feature sampling area are fused to generate the local trajectory features.

3. The method for optimizing predicted trajectories according to claim 2, characterized in that, The step of extracting local feature sampling regions from the global bird's-eye view features based on the trajectory point coordinates of the initial predicted trajectory includes: Map the coordinates of the trajectory points of the initial predicted trajectory to the global bird's-eye view feature to obtain the pixel coordinates of each trajectory point in the global bird's-eye view feature; Centered on the pixel coordinates, the local feature sampling area is delineated in the global bird's-eye view feature according to a preset specification.

4. The method for optimizing the predicted trajectory according to any one of claims 1 to 3, characterized in that, Based on the target state features and the local trajectory features, the initial predicted trajectory is optimized to obtain the target predicted trajectory corresponding to the initial predicted trajectory, including: The initial predicted trajectory, the target state features, and the local trajectory features are subjected to hierarchical fusion processing to obtain the first trajectory fusion feature; The first trajectory fusion feature is subjected to nonlinear processing to obtain the target predicted trajectory.

5. The method for optimizing predicted trajectories according to claim 4, characterized in that, The step of performing a hierarchical fusion process on the initial predicted trajectory, the target state features, and the local trajectory features to obtain the first trajectory fusion feature includes: The initial predicted trajectory and the target state features are fused to obtain the second trajectory fusion feature; The second trajectory fusion feature and the local trajectory feature are subjected to feature fusion processing to obtain the first trajectory fusion feature.

6. The method for optimizing the predicted trajectory according to any one of claims 1 to 3, characterized in that, The acquisition of global bird's-eye view features, as well as the initial predicted trajectory and target state features of the target object, includes: Obtain the initial bird's-eye view features and the initial state features of the target object; The global bird's-eye view features are determined based on the initial bird's-eye view features and the initial state features; Extract the state features of the target object from the global bird's-eye view features to obtain the target state features; The initial predicted trajectory is obtained by performing trajectory prediction on the target state features.

7. The method for optimizing the predicted trajectory according to any one of claims 1 to 3, characterized in that, The target object includes the target vehicle, and after obtaining the target predicted trajectory corresponding to the initial predicted trajectory, it also includes: The target vehicle is controlled to move based on the target predicted trajectory corresponding to the target vehicle.

8. A vehicle, characterized in that, It includes a vehicle body and a trajectory prediction system, the trajectory prediction system being used to implement the method for optimizing the predicted trajectory as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method for optimizing the predicted trajectory as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, include: A computer program, when executed by a processor, implements the method for optimizing the predicted trajectory as described in any one of claims 1 to 7.

Images