Transform architecture-based automatic driving track prediction method and system

Abstract

The invention relates to the technical field of vehicles, and discloses an automatic driving track prediction method and system based on a transformer architecture, electronic equipment, a storage medium and a vehicle, and the method comprises the steps: responding to multi-mode input information needed by automatic driving motion prediction, and carrying out the corresponding feature representation; in response to the multi-modal input information after feature representation, extracting deep features of the environment and encoding the deep features to obtain environment feature expression; in response to the encoded environment feature expression, track features are extracted based on a cross attention mechanism, and the track features comprise key information used for track prediction; in response to the trajectory features, a multi-modal trajectory is predicted, where the prediction includes a probability of the trajectory, a mean value of trajectory points, and a variance. According to the invention, complex and heterogeneous architecture design can be avoided, and expansion is facilitated; and meanwhile, by changing architecture parameters, the model capacity and the computing power consumption are controlled, and the balance between the model precision and the performance is solved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to an automatic driving trajectory prediction method, system, electronic device, storage medium and vehicle based on a transformer architecture. Background Art

[0002] At present, motion prediction for autonomous driving is a challenging task, the core of which lies in scene understanding. The complexity of driving scenes leads to the input of this task being a heterogeneous mixture of static and dynamic information. Specifically, how to perform feature representation, feature extraction, and feature fusion on multimodal input information such as road geometry, lane connectivity, time-varying traffic light status, historical information of dynamic obstacles, and the interactions between them is a very critical and unresolved issue.

[0003] In the existing technical solutions, it is necessary to manually design different feature representation modules, feature pre-processing modules, and subsequent feature fusion modules for different types of input information. This makes it difficult to expand and extend the entire system, and it is difficult to adjust between accuracy and performance.

[0004] One approach is to render the scene input information as a multi-channel rasterized top-down view [4, 2, 28, 29, 7, 23], where the relationship between scene input elements is presented on top-down orthogonal planes and modeled through a spatiotemporal convolutional network. However, the local structure of the convolution is well suited for processing image inputs, but cannot effectively capture long-term spatiotemporal relationships.

[0005] Another approach is to use time-series processing techniques (RNN or time-series CNN) to model the historical state of the agent, approximate road elements to basic primitives (such as piecewise linear segments) and encode them with posture and semantic information. Then, the relationship modeling between all entities is usually regarded as an information aggregation process. The model can be implemented using structures such as pooling [23, 34, 31, 35, 10, 28], soft attention [10, 23], or graph neural networks [36, 33, 30]. However, the disadvantage of time-series processing techniques such as RNN or time-series CNN is that they cannot be calculated in parallel, which makes the calculation and reasoning time-consuming; at the same time, RNN is not good at processing too long sequences, and CNN has a limited receptive field, which is also not conducive to the relationship modeling between agents.

[0006] Therefore, the present application provides an autonomous driving trajectory prediction method based on a transformer architecture to solve the above technical problems. Summary of the invention

[0007] The purpose of the present invention is to provide an autonomous driving trajectory prediction method, system, electronic device, storage medium and vehicle based on a transformer architecture, so as to solve the technical problems in the prior art that manual feature characterization is required, which makes it difficult to expand and extend, and it is difficult to adjust between accuracy and performance.

[0008] In order to solve the above technical problems, the present invention provides an automatic driving trajectory prediction method based on a transformer architecture, comprising:

[0009] A feature expression step, including responding to multimodal input information required for autonomous driving motion prediction and performing corresponding feature representation;

[0010] The encoding processing step includes extracting and encoding the deep features of the environment in response to the multimodal input information after the feature representation to obtain an environmental feature expression;

[0011] A trajectory feature extraction step, comprising extracting trajectory features based on a cross-attention mechanism in response to the encoded environmental feature expression, wherein the trajectory features include key information for trajectory prediction;

[0012] The prediction output step includes predicting a multimodal trajectory in response to the trajectory features, wherein the prediction includes the probability of the trajectory, the mean and variance of the trajectory points.

[0013] In some specific embodiments, the feature expression step includes responding to multimodal input information required for autonomous driving motion prediction and performing corresponding feature characterization, further comprising:

[0014] Determine the agent history, which is used to characterize the historical motion state of the agent, including the position, velocity, and acceleration data of each agent in a continuous time series, forming historical state information;

[0015] Determine agent interaction, which is used to characterize the interaction relationship between agents, including analyzing the relative position, velocity, and acceleration of each agent with respect to its surrounding agents to determine the interaction relationship;

[0016] Determine the roadmap, which is used to represent the road information around the agent, including extracting the geometric information of the road around the agent and converting the geometric information into the representation in the agent coordinate system;

[0017] Determine the traffic light state, which is used to represent the historical information of the traffic lights around the agent, including collecting the state information of the traffic lights around the agent, including the position and color, and converting it into the representation in the agent coordinate system;

[0018] Based on each determined feature, corresponding mask information is generated to identify and process missing or incomplete data to ensure the completeness and accuracy of feature characterization.

[0019] In some specific embodiments, the encoding processing step includes extracting and encoding the deep features of the environment in response to the multimodal input information after the feature representation to obtain the environmental feature expression, and further includes:

[0020] Inputting the feature-characterized multimodal input information into a scene encoder, wherein the scene encoder is composed of a plurality of transformer encoder blocks;

[0021] The input of each modality is subjected to feature extraction using a transformer encoder block. Each block includes a multi-head self-attention mechanism and a feedforward neural network.

[0022] Aggregating input information of different modalities according to a fusion strategy, wherein the fusion strategy includes late fusion, early fusion or hierarchical fusion;

[0023] The model complexity and inference time are controlled by adjusting the number of transformer encoder blocks, sequence length, and the number of heads of multi-head self-attention.

[0024] In some specific embodiments, the trajectory feature extraction step includes extracting trajectory features based on a cross-attention mechanism in response to the encoded environmental feature expression, wherein the trajectory features include key information for trajectory prediction, and further includes:

[0025] Inputting the encoded environmental feature expression into a trajectory decoder;

[0026] Use multiple transformer decoder layers for feature extraction, each layer includes mask multihead self attention, cross multihead self attention and feedforward neural network;

[0027] Extract trajectory features from environmental feature expressions through the cross-attention mechanism;

[0028] A learnable query is used as input to the decoder to extract key information for trajectory prediction.

[0029] In some specific embodiments, the prediction output step includes predicting a multimodal trajectory in response to the trajectory features, wherein the prediction includes the probability of the trajectory, the mean and variance of the trajectory points, and further includes:

[0030] Input the output trajectory features into the GMM prediction head;

[0031] The three fully connected layer branches based on the GMM prediction head predict the probability of the trajectory, the mean and variance of the trajectory points respectively;

[0032] Generate a multimodal prediction trajectory based on the prediction results;

[0033] During the model training process, loss calculation is used to combine the prediction results and the actual trajectory data to calculate the trajectory classification loss and trajectory regression loss to optimize the model parameters.

[0034] In some specific embodiments, before the feature expression step, the method further includes:

[0035] Before model training, use open source datasets for data processing to obtain input data that meets feature characterization requirements;

[0036] Collect real-car data in specific environments and fine-tune the model to adapt to actual driving scenarios;

[0037] Based on the model training results, adjust the model parameters to optimize prediction accuracy and performance.

[0038] Based on the same concept, the present invention also provides an automatic driving trajectory prediction system based on a transformer architecture, comprising:

[0039] A feature expression module, configured to respond to multimodal input information required for autonomous driving motion prediction and perform corresponding feature representation;

[0040] A coding processing module, configured to extract and encode the deep-level features of the environment in response to the multimodal input information after the feature representation, so as to obtain an environmental feature expression;

[0041] A trajectory feature extraction module is configured to extract trajectory features in response to the encoded environmental feature expression based on a cross-attention mechanism, wherein the trajectory features include key information for trajectory prediction;

[0042] A prediction output module is configured to predict a multimodal trajectory in response to the trajectory features, wherein the prediction includes a probability of the trajectory, a mean and a variance of the trajectory points.

[0043] Based on the same concept, the present invention also provides an electronic device, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the autonomous driving trajectory prediction method based on the transformer architecture.

[0044] Based on the same concept, the present invention also provides a computer-readable storage medium, which stores a computer program that can be executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the autonomous driving trajectory prediction method based on the transformer architecture.

[0045] Based on the same concept, the present invention also provides a vehicle, which is provided with the automatic driving trajectory prediction system based on the transformer architecture as described above.

[0046] Compared with the prior art, the beneficial effect lies in that the present invention discloses an autonomous driving trajectory prediction method, system, electronic device, storage medium and vehicle based on transformer architecture, which can avoid complex and heterogeneous architecture design and facilitate expansion; at the same time, by changing the architecture parameters, the model capacity and computing power consumption are controlled, and a balance is achieved between model accuracy and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of some specific embodiments of an automatic driving trajectory prediction method based on a transformer architecture of the present invention;

[0048] Figure 2 It is a schematic diagram of the architecture of an autonomous driving trajectory prediction method based on a transformer architecture in some applications of the present invention;

[0049] Figure 3 It is a schematic diagram of late fusion in scene encoder in some applications of an autonomous driving trajectory prediction method based on transformer architecture of the present invention;

[0050] Figure 4 It is a schematic diagram of early fusion in scene encoder of an automatic driving trajectory prediction method based on transformer architecture in some applications of the present invention;

[0051] Figure 5It is a schematic diagram of hierarchical fusion in scene encoder of an autonomous driving trajectory prediction method based on transformer architecture in some applications of the present invention;

[0052] Figure 6 It is a schematic diagram of a trajectory decoder of an autonomous driving trajectory prediction method based on a transformer architecture in some applications of the present invention;

[0053] Figure 7 It is a schematic diagram of a GMM prediction head of an automatic driving trajectory prediction method based on a transformer architecture in some applications of the present invention;

[0054] Figure 8 It is a schematic diagram of loss calculation of an autonomous driving trajectory prediction method based on a transformer architecture in some applications of the present invention;

[0055] Fig. 9 It is a schematic diagram of the structure of an autonomous driving trajectory prediction system based on a transformer architecture in some specific embodiments of the present invention;

[0056] Fig.10 It is a schematic diagram of the structure of an electronic device in some specific embodiments of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0058] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, and unless the context clearly indicates other meanings, "multiple" generally includes at least two.

[0059] It should be understood that the term "and / or" used in this article is only a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0060] It should be understood that although the terms first, second, third, etc. may be used to describe in the embodiments of the present application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0061] Depending on the context, the words "if" and "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting." Similarly, depending on the context, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)."

[0062] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "including a ..." do not exclude the presence of other identical elements in the commodity or device including the elements.

[0063] It should be particularly noted that any symbols and / or numbers in the specification that are not marked in the accompanying drawings are not drawing marks.

[0064] Reference Figure 1 , a method for autonomous driving trajectory prediction based on transformer architecture, comprising:

[0065] S101, a feature expression step, comprising responding to multimodal input information required for autonomous driving motion prediction and performing corresponding feature representation;

[0066] It is understandable that trajectory prediction requires predicting the future motion trajectory of a vehicle (i.e., an agent). To achieve this goal, complex data from multiple sources needs to be processed and analyzed.

[0067] In some of these applications, in response to multimodal input information required for autonomous driving motion prediction and corresponding feature characterization, an agent history is determined to characterize the historical motion state of the agent, including the position, velocity and acceleration data of each agent in a continuous time series to form historical state information; an agent interaction is determined to characterize the interaction relationship between agents, including analyzing the relative position, velocity and acceleration of each agent with respect to its surrounding agents to determine the interaction relationship; a roadgraph is determined to characterize the road information around the agent, including extracting geometric information of the road around the agent and converting the geometric information into a representation in the agent coordinate system; a traffic light state is determined to characterize the historical information of the traffic lights around the agent, including collecting the state information of the traffic lights around the agent, including the position and color and converting it into a representation in the agent coordinate system; based on each determined feature, corresponding mask information is generated to identify and process missing or incomplete data to ensure the integrity and accuracy of the feature characterization.

[0068] It is understandable that the characteristics represent:

[0069] 1.Agent history: Characterizes the historical motion state of each agent (e.g., vehicle).

[0070] Data Collection: Collect position, velocity, and acceleration data for each agent in a continuous time series.

[0071] Data processing: Integrate the data into a unified representation. For example, for each agent, collect data within the past 1 second with a sampling frequency of 10 Hz, so the historical state of each agent consists of the position (x, y), velocity (vx, vy), and acceleration (ax, ay) data of 10 time points.

[0072] For example, an agent's position at time t is (5m, 5m), its velocity is (1m / s, 0m / s), and its acceleration is (0m / s 2 , 0m / s 2 ), the data will be encoded into a 10x6 matrix, with each row representing the data at one time point.

[0073] 2.Agent interaction: characterizes the interaction relationship between agents.

[0074] Data Collection: Analyze the relative position, velocity, and acceleration of each agent to its surrounding agents.

[0075] Data processing: For example, for an agent, consider the 5 nearest agents around it and calculate the relative position change and relative speed between them.

[0076] For example, if the position of an agent at time point t is (10m, 10m), and the positions of its surrounding agents are (11m, 10m), (9m, 11m), etc., the relative positions are calculated and encoded into a 5x2 matrix representing the position of each surrounding agent relative to the central agent.

[0077] 3.roadgragh: represents the road information around the agent.

[0078] Data collection: Extract geometric information of the road around the agent, such as lane lines, intersections, etc.

[0079] Data processing: Convert this geometric information into a representation in the agent's coordinate system, for example, converting the center line of a road into coordinates relative to the agent's current position.

[0080] For example, if an agent is located at an intersection, the centerline of the road around it can be represented as a series of relative coordinates, such as (1m, 0m), (-1m, 0m), etc., indicating the relative position to the agent's position.

[0081] 4. Traffic light state: represents the historical information of traffic lights around the agent.

[0082] Data Collection: Collects information about the state of traffic lights around the agent, including their position and color.

[0083] Data processing: Convert the traffic light status information into the agent coordinate system.

[0084] For example, if an agent is located near a traffic light, the position of the traffic light is (20m, -5m) and the current state is green, the information is encoded as a vector containing the position and state, such as (20m, -5m, "Green").

[0085] 5. Mask Information: Identify and handle missing or incomplete data.

[0086] Data processing: For each feature, generate corresponding mask information to identify which data is complete and which is missing.

[0087] For example, in the agent history, if the data at a certain time point is missing, the corresponding mask value will be set to False to instruct the model to ignore this data when processing.

[0088] In some embodiments, for example, the agent history feature collects information about a specific vehicle (agent A) at an intersection. The location data of agent A is collected in 0.1 second intervals over the past 1 second. At t = 0s, the location of agent A is (x = 10.0m, y = 5.0m), the velocity is (vx = 2.0m / s, vy = 0.5m / s), and the acceleration is (ax = 0.1m / s 2 , ay=0.0m / s 2 ). The data is recorded every 0.1 seconds to form the historical motion state of agent A. The data is integrated into a historical state information tensor containing 11 time points, each of which contains position, velocity and acceleration information, forming historical state information.

[0089] Characterization of agent interaction, analyzing the relative position, velocity and acceleration of agent A and other surrounding vehicles (agent B, agent C, etc.). At t = 0.5s, the position of agent B relative to agent A is (x = 2.0m, y = -1.0m), and the speed difference is (Δvx = -0.2m / s, Δvy = 0.3m / s); the position of agent C relative to agent A is (x = -3.0m, y = 2.0m), and the speed difference is (Δvx = 0.1m / s, Δvy = -0.4m / s). Based on the above data, the interaction relationship between agent A and the surrounding vehicles is determined, for example, agent B is approaching agent A, while agent C is moving away from agent A.

[0090] The feature representation of roadgraph extracts the road geometry information around agent A, such as lane lines, sidewalks, and intersection layout. Assume that agent A is located at a T-junction with a straight lane and a left turn lane in front of it. The above information is converted into the representation in the coordinate system of agent A. For example, the center line of the straight lane is located at (x=5.0m, y=0m) in the coordinate system of agent A.

[0091] Traffic light state feature representation, collects traffic light state information around agent A, including position and color. In agent A's field of view, there is a traffic light located on its right side, at (x=10.0m, y=10.0m), and its current state is green. The above information is converted into representation in agent A's coordinate system.

[0092] Mask information generation,There may be missing data in the collected data, for example, the acceleration data at a certain time point is missing. At t = 0.3s, the acceleration data of agent A is missing. In the formed historical state information tensor, the acceleration value at this time point is marked as missing, and the corresponding mask information is generated to instruct the model to ignore this missing data point when processing.

[0093] S102, a coding processing step, comprising extracting and encoding deep features of the environment in response to the multimodal input information after the feature representation, to obtain an environmental feature expression;

[0094] It can be understood that this step is used to extract deep-level features of the environment from the multimodal input information after feature representation, and encode it to obtain the environmental feature expression.

[0095] In some of the applications, in response to multimodal input information after feature representation, deep features of the environment are extracted and encoded to obtain environmental feature expressions, and the multimodal input information after feature representation is input into a scene encoder, wherein the scene encoder is composed of multiple transformer encoder blocks; a transformer encoder block is used to extract features from the input of each modality, and each block includes a multi-head self-attention mechanism and a feedforward neural network; input information of different modalities is aggregated according to a fusion strategy, wherein the fusion strategy includes late fusion, early fusion or hierarchical fusion; model complexity and inference time are controlled based on adjusting the number of transformer encoder blocks, sequence length and the number of heads of multi-head self-attention parameters.

[0096] It can be understood that feature extraction and encoding:

[0097] 1. Input information:

[0098] The multimodal input information after feature representation includes: agent history, agent interaction, roadmap, traffic light state and its corresponding mask information.

[0099] 2.scene encoder structure:

[0100] Composition: The scene encoder consists of multiple transformer encoder blocks, each of which is responsible for processing input information of one modality.

[0101] For example, use 4 transformer encoder blocks, each block processes input of one modality.

[0102] 3. Feature extraction process:

[0103] Multi-head self-attention mechanism: Each transformer encoder block contains a multi-head self-attention mechanism to capture the dependencies between different parts of the input information.

[0104] For example, each block has 8 attention heads, each of which processes a subset of the input information and extracts features in parallel.

[0105] Feedforward neural network: Each block also contains a feedforward neural network for further feature extraction.

[0106] For example, a feed-forward network can be a sequence of two fully connected layers, the first layer has 128 neurons and the second layer has 64 neurons.

[0107] 4. Fusion strategy:

[0108] Late fusion: The input information of each modality is first independently extracted through its own transformer encoder block and then aggregated.

[0109] For example, there are four modal inputs. After extracting features from each modality independently, the obtained feature vectors are concatenated.

[0110] Early fusion: The input information of all modalities is first concatenated and then input into a transformer encoder block for feature extraction.

[0111] For example, the input vectors of all modalities are concatenated into a large input vector and then input into an encoderblock.

[0112] Hierarchical fusion: First, use the multi-head attention mechanism to extract features from the input of each modality, then aggregate the feature extraction results of the above modalities together in the sequence dimension, and finally use the multi-head attention mechanism to extract features from the aggregated features again.

[0113] For example, the feature extraction results of agent history and agent interaction are first concatenated, and then concatenated with the feature extraction results of roadmap and traffic light state.

[0114] 5. Model complexity and reasoning time control:

[0115] Adjust parameters: Control the complexity and inference time of the model by adjusting the number of transformer encoder blocks, sequence length, and the number of heads of multi-head self-attention.

[0116] For example, if you want to reduce the amount of computation, you can reduce the number of transformer encoder blocks from 4 to 2, or reduce the number of attention heads in each block from 8 to 4.

[0117] In some embodiments, for example, the following characterizations have been achieved:

[0118] Agent history: The historical motion state of each agent (vehicle).

[0119] Agent interaction: The interaction between agents.

[0120] roadgragh: Road information around the agent.

[0121] traffic light state: The state of the traffic lights around the agent.

[0122] The scene encoder consists of 3 transformer encoder blocks, each of which is responsible for processing the input information of one modality. The input sequence length of each modality is 50, and the feature dimension of each modality is 128. Each transformer encoder block contains 8 attention heads to capture the dependencies between different parts of the input information. Each attention head processes a subset of the input sequence, so the 8 heads can process different parts of the sequence in parallel. For example, at a specific point in time, the historical position information of agent A is assigned to head 1 and head 2 for processing, while the interaction information of agent B is assigned to head 3 and head 4 for processing. Each block also contains a feedforward neural network for further feature extraction. The feedforward network can be a sequence of two fully connected layers, with the first layer having 256 neurons and the second layer having 128 neurons. For example, the first fully connected layer can use the ReLU activation function and the second fully connected layer can use the tanh activation function.

[0123] In latefusion, the input information of each modality is first independently extracted through its own transformer encoder block, and then aggregated. Assuming there are 4 modal inputs, each modality extracts features independently, and then the obtained feature vectors are concatenated. For example, the feature vector of agent history is [0.5, 0.3, 0.2], the feature vector of agent interaction is [0.4, 0.4, 0.2], the feature vector of roadgraph is [0.6, 0.2, 0.2], and the feature vector of traffic light state is [0.3, 0.3, 0.4], then the concatenated feature vector is [0.5, 0.3, 0.2, 0.4, 0.4, 0.2, 0.6, 0.2, 0.2, 0.3, 0.3, 0.4].

[0124] Early fusion first concatenates the input information of all modalities and then inputs it into a transformer encoder block for feature extraction. Concatenate the input vectors of all modalities into a large input vector and then input it into an encoder block. For example, if there are 4 agents and each agent has 4 modal inputs, then the concatenated input vector is a tensor with a shape of [4, 4*50, 128].

[0125] Hierarchical fusion, first use the multi-head attention mechanism to extract features from the input of each modality, then aggregate the feature extraction results of the above modalities together in the sequence dimension, and finally use the multi-head attention mechanism to extract features from the aggregated features. First, concatenate the feature extraction results of agent history and agent interaction, and then concatenate them with the feature extraction results of roadgraph and traffic light state. For example, if the feature vectors of agenthistory and agent interaction are concatenated first, and then concatenated with the feature vectors of roadgraph and traffic light state, the final concatenated feature vector is a tensor with a shape of [4, 4*50, 128].

[0126] The model complexity and inference time can be controlled by adjusting the number of transformer encoder blocks, sequence length, and the number of heads of multi-head self-attention. If you want to reduce the amount of computation, you can reduce the number of transformer encoder blocks from 3 to 2, or reduce the number of attention heads in each block from 8 to 4. For example, if you reduce the number of blocks, the model will first use one block to process the fused features of agent history and agent interaction, and then use the second block to process the fused features of roadmap and traffic light state.

[0127] S103, a trajectory feature extraction step, comprising extracting trajectory features in response to the encoded environmental feature expression based on a cross attention mechanism, wherein the trajectory features include key information for trajectory prediction;

[0128] It can be understood that this step is used to extract trajectory features from the encoded environmental feature expression.

[0129] In some of these applications, in response to the encoded environmental feature expression, trajectory features are extracted based on a cross-attention mechanism, and the trajectory features include key information for trajectory prediction. The encoded environmental feature expression is input into a trajectory decoder; multiple transformer decoder layers are used for feature extraction, each layer including mask multihead self attention, cross multihead selfattention and a feedforward neural network; trajectory features are extracted from the environmental feature expression through a cross-attention mechanism; and a learnable query is used as an input to a decoder to extract key information for trajectory prediction.

[0130] It is understandable that trajectory feature extraction:

[0131] 1. Expression of environmental characteristics:

[0132] The encoded environmental feature expression is generated by the scene encoder and contains the deep feature information of the environment.

[0133] 2. Trajectory decoder structure:

[0134] Composition: The trajectory decoder consists of multiple transformer decoder layers, each of which is responsible for further extracting and decoding features to generate the features required for trajectory prediction.

[0135] For example, using 3 transformer decoder layers, each layer contains 8 attention heads.

[0136] 3. Feature extraction process:

[0137] Mask multihead self attention: The first component of each decoder layer is a mask multihead self attention, which allows the model to establish dependencies between different parts of the sequence while ignoring irrelevant parts.

[0138] For example, each attention head processes a part of the sequence, then 8 heads can process different parts of the sequence in parallel.

[0139] Cross multihead self attention: The cross attention mechanism allows the model to compare the internal representation of the decoder with the output of the encoder to extract information relevant to trajectory prediction.

[0140] For example, if the dimension of the environmental feature expression is 512, then the cross attention layer will receive 512-dimensional query, key, and value vectors.

[0141] Feedforward neural network: The last component of each decoder layer is a feedforward neural network, which performs further nonlinear transformation on the output of the attention layer.

[0142] For example, a feed-forward network can be a sequence of two fully connected layers, the first layer has 256 neurons and the second layer has 128 neurons.

[0143] 4. Cross-attention mechanism, extracting trajectory features from environmental feature expressions:

[0144] The learnable query vector is used as the input of the decoder to extract key information for trajectory prediction through cross-attention with the environmental features.

[0145] For example, there are 6 learnable query vectors, each with a dimension of 512, then the cross attention mechanism compares these queries with the environmental feature representation to generate 6 trajectory feature vectors.

[0146] 5. Learning and applying query vectors to extract key information for trajectory prediction:

[0147] The query vector is learned through training, so that the model can identify and extract the features most relevant to trajectory prediction.

[0148] For example, during training, the model learns that some query vectors focus more on the state of a traffic light, while other query vectors might focus more on the interactions between agents.

[0149] In some embodiments, for example, the environmental feature expression has been generated by the scene encoder, which contains deep feature information of the environment, and its shape is [batch_size, sequence_length, feature_dimension], for example [32, 50, 256], where 32 represents the batch size, 50 represents the sequence length, and 256 represents the feature dimension.

[0150] The trajectory decoder consists of 3 transformer decoder layers, each of which is responsible for further extracting and decoding features to generate the features required for trajectory prediction. Each decoder layer contains 8 attention heads with a feature dimension of 256.

[0151] The first component of each decoder layer is the mask multihead selfattention. If the data at the 10th time point in a sequence is missing, the model can be made to ignore the data at this time point by setting the mask information. For example, if there is a fill value such as -1 in the sequence, the mask can be a Boolean array indicating whether the data at each time point is valid.

[0152] Cross multihead self attention, assuming that the dimension of the environmental feature expression is 256, the crossattention layer will receive a 256-dimensional query, key, and value vector. For example, if the model is paying attention to a specific agent (such as agent A), the trajectory history of agent A is used as keys and the environmental features are used as values ​​to predict the future position of agent A through the query.

[0153] The last component of each decoder layer is a feed-forward neural network that performs further nonlinear transformations on the output of the attention layer. The feed-forward network can be a sequence of two fully connected layers, with the first layer having 512 neurons and the second layer having 256 neurons. For example, the first fully connected layer applies a ReLU activation function to increase the nonlinearity, while the second fully connected layer applies a tanh activation function to limit the output to a reasonable range.

[0154] Using a learnable query vector as the input of the decoder, the key information for trajectory prediction is extracted through cross attention with the environmental feature expression. Assuming there are 6 learnable query vectors, each with a dimension of 256, the cross attention mechanism compares these queries with the environmental feature expression to generate 6 trajectory feature vectors. The query vector can be learned through training so that the model can focus on the most important information for trajectory prediction. For example, the model learns that at an intersection, the agent's future trajectory is highly correlated with the state of the traffic lights and the movement trends of the surrounding agents.

[0155] S104, a prediction output step, comprising predicting a multimodal trajectory in response to the trajectory features, wherein the prediction includes a probability of the trajectory, a mean value and a variance of the trajectory points.

[0156] It can be understood that this step is used to predict the multimodal trajectory from the trajectory features, including the probability of the trajectory, the mean and variance of the trajectory points.

[0157] In some of these applications, in response to trajectory features, multimodal trajectories are predicted, including the probability of the trajectory, the mean and variance of the trajectory points, and the output trajectory features are input into the GMM prediction head; the probability of the trajectory, the mean and variance of the trajectory points are predicted respectively based on the three fully connected layer branches of the GMM prediction head; a multimodal predicted trajectory is generated based on the prediction results; during the model training process, the trajectory classification loss and trajectory regression loss are calculated using loss calculation combined with the prediction results and the actual trajectory data to optimize the model parameters.

[0158] It is understandable that multimodal trajectory prediction:

[0159] 1. Trajectory characteristics:

[0160] The trajectory feature is output by the trajectory decoder and contains key information for trajectory prediction.

[0161] 2.GMM prediction head structure:

[0162] Composition: The GMM prediction head contains three fully connected layer branches, which are used to predict the probability of the trajectory, the mean and variance of the trajectory points.

[0163] For example, the dimension of trajectory features is 256, and the output dimensions of each fully connected layer branch are:

[0164] Trajectory probability branch: The output dimension is 1, indicating the probability of each trajectory.

[0165] Trajectory point mean branch: The output dimension is 2 (x, y coordinates), which represents the mean position of the trajectory point.

[0166] Track point variance branch: The output dimension is 2 (x, y coordinates), which represents the position variance of the track point.

[0167] 3. Prediction process:

[0168] Trajectory probability prediction: The first fully connected layer branch receives trajectory features as input and outputs the probability of the trajectory through an activation function (such as sig moid).

[0169] For example, if the trajectory feature vector is [0.2, 0.3, ..., 0.5], the layer outputs a probability value of 0.6, indicating the likelihood of the trajectory occurring.

[0170] Trajectory point mean prediction: The second fully connected layer branch outputs the mean position of the trajectory point.

[0171] For example, given the same trajectory feature vector, the layer outputs the mean position as (x=5.0m, y=3.0m).

[0172] Trajectory point variance prediction: The third fully connected layer branch outputs the position variance of the trajectory point, indicating the uncertainty of the predicted position.

[0173] For example, also based on the trajectory feature vector, the layer output position variance is (x=0.5m, y=0.5m).

[0174] 4. Multimodal prediction trajectory generation:

[0175] Combining the prediction results of the above three branches, multiple trajectories that each agent may take in the future are generated.

[0176] For example, 5 possible trajectories are predicted, each with a probability value, a mean position, and a position variance.

[0177] 5. Model training and optimization:

[0178] Loss calculation: During model training, the loss calculation module is used to calculate the trajectory classification loss and trajectory regression loss by combining the prediction results and the actual trajectory data.

[0179] Trajectory classification loss: Calculate the difference between the predicted trajectory probability and the true trajectory probability.

[0180] Trajectory regression loss: Calculate the difference between the mean and variance of the predicted trajectory points and the mean and variance of the true trajectory points.

[0181] For example, the probability of the true trajectory is 0.7, the mean position is (x=5.2m, y=3.1m), and the position variance is (x=0.4m, y=0.4m). The difference between the trajectory predicted by the model and the true trajectory is quantified through the loss calculation module, and the model parameters are optimized through back propagation.

[0182] In some embodiments, for example, the trajectory feature has been output by the trajectory decoder, contains key information for trajectory prediction, and has a shape of [batch_size, feature_dimension], for example, [32, 256], where 32 represents the batch size and 256 represents the feature dimension.

[0183] The GMM prediction head contains three fully connected layer branches, each of which is responsible for predicting different aspects of the trajectory. Assuming that the dimension of the trajectory feature is 256, the output dimensions of each fully connected layer branch are:

[0184] Trajectory probability branch: The output dimension is 1, indicating the probability of each trajectory.

[0185] Trajectory point mean branch: The output dimension is 2 (x, y coordinates), which represents the mean position of the trajectory point.

[0186] Track point variance branch: The output dimension is 2 (x, y coordinates), which represents the position variance of the track point.

[0187] The first fully connected layer branch receives the trajectory features as input and outputs the probability of the trajectory through an activation function (such as sigmoid, where the weight matrix is ​​randomly initialized and the bias term is also a learnable parameter). If the trajectory feature vector is [0.2, 0.3, ..., 0.5], the layer outputs a probability value of 0.6, indicating the possibility of the trajectory occurring.

[0188] The second fully connected layer branch outputs the mean position of the trajectory points. Given the same trajectory feature vector, the layer outputs the mean position (x=5.0m, y=3.0m).

[0189] The third fully connected layer branch outputs the position variance of the trajectory point, which is also based on the trajectory feature vector. The output position variance of this layer is (x=0.5m, y=0.5m).

[0190] Combining the prediction results of the above three branches, multiple trajectories that each agent may take in the future are generated. Assume that 5 possible trajectories are predicted, each trajectory has a probability value, a mean position, and a position variance. For example, a trajectory has a probability of 0.7, a mean position of (x=5.0m, y=3.0m), and a position variance of (x=0.5m, y=0.5m). The model predicts that the agent has a 70% probability of being at the position of (5.0m, 3.0m) at some point in the future, and the uncertainty of the prediction is relatively low.

[0191] During the model training process, the Loss calculation module is used to combine the prediction results and the actual trajectory data to calculate the trajectory classification loss and trajectory regression loss.

[0192] Trajectory classification loss can be calculated by using the meta-cross entropy loss function to calculate the difference between the predicted trajectory probability and the true trajectory probability. For example, if the true probability is very close to 1 and the model predicts a probability of 0.8, the loss function will calculate the difference between these two values.

[0193] Trajectory regression loss, which can be calculated by the mean square error (MSE) loss function, is the difference between the mean and variance of the predicted trajectory points and the mean and variance of the true trajectory points. For example, if the difference between the mean position predicted by the model and the true position is large, the MSE loss will be high, which will prompt the model parameters to adjust in the direction of reducing this difference during training. Assume that the probability of the true trajectory is 0.9, the mean position is (x=5.2m, y=3.1m), and the position variance is (x=0.4m, y=0.4m). The difference between the trajectory predicted by the model and the true trajectory is quantified by the Loss calculation module. For example, if the model predicts a probability of 0.8, the mean position is (x=5.0m, y=3.0m), and the position variance is (x=0.5m, y=0.5m), the loss function calculates the difference between the two probability values, the two sets of position values, and the two sets of variance values, and optimizes the model parameters through back propagation.

[0194] Some of these applications also include using open source datasets for data processing before model training to obtain input data that meets feature characterization requirements; collecting real vehicle data under specific environments and fine-tuning the model to adapt to actual driving scenarios; and adjusting model parameters based on model training results to optimize prediction accuracy and performance.

[0195] Understandably, this step is used to ensure that the model can adapt to actual driving scenarios and optimize prediction accuracy and performance.

[0196] 1. Dataset selection:

[0197] Choose open source datasets, such as Waymo Open Dataset or nuScenes, which contain rich driving scene information such as vehicle position, speed, acceleration, road geometry, traffic light status, etc.

[0198] 2. Data preprocessing:

[0199] Data cleaning: Remove noise and outliers from a dataset, for example, by removing data points that are mislabeled or have missing signals.

[0200] Data formatting: Convert data into the format required for model input, for example, convert data into historical state information and interaction information for a specific time window.

[0201] For example, processing the Waymo Open Dataset, the position, velocity, and acceleration data of each agent (vehicle) are converted into a tensor of shape [A, T, 1, Dh], where A = 10 (number of agents), T = 10 (number of historical frames), and Dh = 5 (feature length, including x, y position, vx, vy velocity, ax, ay acceleration).

[0202] 3. Feature extraction:

[0203] According to the feature representation requirements, features such as agent history, agent interaction, roadmap and traffic light state are extracted, and the corresponding mask information is generated.

[0204] Collect real vehicle data in specific environments:

[0205] 1. Data Collection:

[0206] Deploy test vehicles in actual driving environments and collect vehicle sensor data, such as lidar, cameras, GPS, etc.

[0207] For example, collect 100 hours of real-car driving data, including different scenarios such as urban roads, highways, and rural roads.

[0208] 2. Data annotation:

[0209] The collected data are labeled, including the agent's movement trajectory, interaction relationship, road information, and traffic light status.

[0210] Model fine-tuning:

[0211] 1. Model training:

[0212] Use the preprocessed open source dataset to perform preliminary training on the model and obtain the initial weights of the model.

[0213] 2. Fine-tune strategy:

[0214] Data fusion: Fusion of real vehicle data with open source datasets to fine-tune the model.

[0215] For example, a model is trained using an open source dataset and then fine-tuned on a dataset containing 30% real-car data.

[0216] 3. Model evaluation and adjustment:

[0217] Evaluation metrics: Use metrics such as accuracy, recall, and F1 score to evaluate model performance.

[0218] Parameter adjustment: Based on the evaluation results, adjust the model parameters, such as learning rate, regularization coefficient, number of network layers, etc., to optimize the prediction accuracy and performance.

[0219] For example, if the accuracy of the model on real car data is lower than 0.9, adjust the learning rate from 0.001 to 0.0001 and increase the dropout rate from 0.2 to 0.3 to improve the generalization ability of the model.

[0220] Combine the following Figures 2 to 8 The following describes some examples of the automatic driving trajectory prediction method based on the transformer architecture in some applications:

[0221] This embodiment uses masked vectors to uniformly represent the features of multimodal input information, designs an encoding and decoding neural network framework based on the transformer structure that excels at sequence modeling, and outputs multimodal motion trajectories.

[0222] The input information and feature representation required for autonomous driving motion prediction:

[0223] The agent history is used to characterize the historical motion state of the agent. The data dimension is [A, T, 1, Dh], where A is the number of agents modeled, T is the number of historical frames used (including the current frame), 1 is a placeholder to unify the rest of the input into 4 dimensions, and Dh is the feature length (including position coordinates x, y, velocity, acceleration, obstacle bounding box); the agent history mask is used to mask out the missing information in the agent history. Its data dimension is [A, T, 1].

[0224] Agent interaction is used to characterize the interaction relationship between agents. The data dimension is [A, T, Si, Di], where A is the number of modeled agents, T is the number of historical frames used (including the current frame), Si is the number of agents closest to the modeled agent, and Di is the physical feature length of the agent closest to the modeled agent (position coordinates x, y, velocity, acceleration, obstacle bounding box), which needs to be converted to the modeled agent coordinate system; agent interaction mask is used to mask out the missing information in agent interaction. Its data dimension is [A, T, Si].

[0225] Roadgraph is used to represent the road information around the agent, and its data dimension is [A, 1, Sr, Dr]. A is the number of modeled agents. Here, the road around the agent is segmented into many polylines. Sr is the number of Sr polylines closest to the modeled agent. Dr is the characteristic length of the polyline (including its endpoints and type), which needs to be converted to the modeled agent coordinate system. Roadgraph mask is used to mask out the missing information in the roadgraph, and its data dimension is [A, 1, Sr].

[0226] Traffic light state is used to characterize the historical information of traffic lights around the agent. The data dimension is [A, T, Stls, Dtls], where A is the number of modeled agents, T is the number of historical frames used (including the current frame), Stls is the number of traffic lights closest to the modeled agent, and Dtls is the characteristic length of the traffic light (the location coordinates x, y, and color of the traffic light), which needs to be converted to the coordinate system of the modeled agent. Traffic light state mask is used to mask out the missing information in the traffic light state. The data dimension is [A, T, Stls].

[0227] like Figure 2As shown in the figure, the network contains 8 inputs, namely agent history, agent history mask; agent interaction, agent interaction mask; roadgraph, roadgraph mask; traffic lightstate, traffic light statemask. The input information is then encoded by the scene encoder, and then cross-attention is performed with the trajectory decoder to obtain the trajectory feature embedding, which is finally sent to the GMM predictionhead submodule to generate a multimodal predicted trajectory.

[0228] like Figure 3 , Figure 4 and Figure 5 As shown in the figure, the scene encoder is mainly used to extract features from multimodal input information to obtain environmental feature expressions. It uses a similar structure to process the input of each model, making the network structure design simple and avoiding the need to design a separate network structure for each modal input. Secondly, the complexity of the model can be adjusted by adjusting the structural parameters (such as the number of times the transformer encoder block is stacked, the length of the sequence, the number of heads of multi-head self-attention, etc.), so it is very easy to find a balance between model complexity and reasoning time / computing power consumption. According to the timing of multimodal input information aggregation, it can be roughly divided into the following three structures, namely late fusion (each modal input extracts features separately, and then multimodal information is aggregated), early fusion (multimodal information is aggregated first, and then feature extraction is performed on the aggregated information), and hierarchical fusion (each modal input extracts features separately, and then multimodal information is aggregated, and then feature extraction is performed again).

[0229] like Figure 6As shown in the figure, trajectory decoder: stack multiple standard transformer decoder layers (mask multihead self attention+add+layernorm, cross multihead self attention+add+layernorm, ffn+add+layernorm), whose input is multiple learnable queries, which are used to obtain the feature expression of multimodal trajectories from the encoded information of the encoder through the cross-attention structure, and are used for subsequent prediction of multimodal trajectories; these learnable query inputs are all valid, and their mask values ​​are all True.

[0230] like Figure 7 As shown in the figure, GMM prediction head: the input is the multimodal trajectory embedding after the query passes through the trajectory decoder, and its shape is (K, D); the structure in this module is a branch of three fully connected layers, which are used to predict the probability of the trajectory, the mean and variance of the trajectory points respectively.

[0231] like Figure 8 As shown in the figure, the loss calculation module is only needed in the model training process, not in the model inference process; the input of this module includes the output of the GMM prediction head and the true value of the trajectory in the future T time period, which are combined to calculate the trajectory classification loss and trajectory regression loss.

[0232] As described above, the network framework of this embodiment is simple, efficient and easy to expand; it is independent of the specific input modality; it can avoid complex, heterogeneous architecture design; and it can easily control the model capacity and computing power consumption by changing the architecture parameters, thereby achieving a balance between model accuracy and performance.

[0233] For the method steps disclosed in the above embodiments, the method steps are described as a series of action combinations for the purpose of simple description, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0234] like Fig. 9 As shown, the present invention also provides an automatic driving trajectory prediction system based on a transformer architecture, comprising:

[0235] A feature expression module 201 is configured to respond to multimodal input information required for autonomous driving motion prediction and perform corresponding feature representation;

[0236] The encoding processing module 202 is configured to extract and encode the deep-level features of the environment in response to the multimodal input information after the feature representation to obtain an environmental feature expression;

[0237] A trajectory feature extraction module 203 is configured to extract trajectory features in response to the encoded environmental feature expression based on a cross-attention mechanism, wherein the trajectory features include key information for trajectory prediction;

[0238] The prediction output module 204 is configured to predict a multimodal trajectory in response to the trajectory features, wherein the prediction includes a probability of the trajectory, a mean value and a variance of the trajectory points.

[0239] It is worth noting that although only some basic functional modules are disclosed in the embodiments of the present invention, it does not mean that the composition of the present system is limited to the above basic functional modules. On the contrary, what the present embodiment wants to express is that on the basis of the above basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with the prior art to form an infinite number of embodiments or technical solutions. In other words, the present system is open rather than closed. Just because the present embodiment only discloses individual basic functional modules, it cannot be considered that the protection scope of the claims of the present invention is limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above devices are described in terms of functions and are described separately in various units and modules. Of course, when implementing the present invention, the functions of each unit and module can be implemented in the same or one or more software and / or hardware.

[0240] like Fig.10 As shown, the present invention also provides an electronic device, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the autonomous driving trajectory prediction method based on the transformer architecture.

[0241] Fig.10 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig.10 The electronic device provided in the embodiment of the present invention includes: one or more processors 710 and a storage device 720; the processor 710 in the electronic device may be one or more, Fig.10A processor 710 is taken as an example; the storage device 720 is used to store one or more programs; the one or more programs are executed by the one or more processors 710, so that the one or more processors 710 implement the autonomous driving trajectory prediction method based on the transformer architecture as described in any one of the embodiments of the present invention.

[0242] The electronic device may further include: an input device 730 and an output device 740 .

[0243] The processor 710, storage device 720, input device 730 and output device 740 in the electronic device can be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0244] The storage device 720 in the electronic device is a computer-readable storage medium that can be used to store one or more programs, which can be software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the automatic driving trajectory prediction method based on the transformer architecture provided in the embodiment of the present invention. The processor 710 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the storage device 720, that is, the automatic driving trajectory prediction method based on the transformer architecture in the above method embodiment is implemented.

[0245] The storage device 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the storage device 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 720 may further include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0246] The input device 730 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 740 may include a display device such as a display screen.

[0247] The present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of an autonomous driving trajectory prediction method based on a transformer architecture.

[0248] Specifically, the computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0249] The present invention also provides a vehicle, which is provided with the automatic driving trajectory prediction system based on the transformer architecture as described above.

[0250] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatic driving trajectory prediction based on transformer architecture, characterized in that: include: A feature expression step, including responding to multimodal input information required for autonomous driving motion prediction and performing corresponding feature representation; The encoding processing step includes extracting and encoding the deep features of the environment in response to the multimodal input information after the feature representation to obtain an environmental feature expression; A trajectory feature extraction step, comprising extracting trajectory features based on a cross-attention mechanism in response to the encoded environmental feature expression, wherein the trajectory features include key information for trajectory prediction; The prediction output step includes predicting a multimodal trajectory in response to the trajectory features, wherein the prediction includes the probability of the trajectory, the mean and variance of the trajectory points.

2. The automatic driving trajectory prediction method based on transformer architecture according to claim 1, characterized in that: The feature expression step includes responding to the multimodal input information required for the autonomous driving motion prediction and performing corresponding feature characterization, further including: Determine the agent history, which is used to characterize the historical motion state of the agent, including the position, velocity, and acceleration data of each agent in a continuous time series, forming historical state information; Determine agentinteraction, which is used to characterize the interaction relationship between agents, including analyzing the relative position, velocity and acceleration of each agent with its surrounding agents to determine the interaction relationship; Determine the roadmap, which is used to represent the road information around the agent, including extracting the geometric information of the road around the agent and converting the geometric information into the representation in the agent coordinate system; Determine the traffic light state, which is used to represent the historical information of the traffic lights around the agent, including collecting the state information of the traffic lights around the agent, including the position and color, and converting it into the representation in the agent coordinate system; Based on each determined feature, corresponding mask information is generated to identify and process missing or incomplete data to ensure the completeness and accuracy of feature characterization.

3. The automatic driving trajectory prediction method based on transformer architecture according to claim 1, characterized in that: The encoding processing step includes extracting and encoding the deep features of the environment in response to the multimodal input information after the feature representation to obtain the environmental feature expression, further including: Inputting the feature-characterized multimodal input information into a scene encoder, wherein the scene encoder is composed of a plurality of transformer encoder blocks; The input of each modality is subjected to feature extraction using a transformer encoder block. Each block includes a multi-head self-attention mechanism and a feedforward neural network. Aggregating input information of different modalities according to a fusion strategy, wherein the fusion strategy includes late fusion, early fusion or hierarchical fusion; The model complexity and inference time are controlled by adjusting the number of transformer encoder blocks, sequence length, and the number of heads of multi-head self-attention.

4. The automatic driving trajectory prediction method based on transformer architecture according to claim 1, characterized in that: The step of extracting trajectory features comprises extracting trajectory features based on a cross-attention mechanism in response to the encoded environmental feature expression, wherein the trajectory features include key information for trajectory prediction, and further comprises: Inputting the encoded environmental feature expression into a trajectory decoder; Use multiple transformer decoder layers for feature extraction, each layer includes mask multihead self attention, cross multihead self attention and feedforward neural network; Extract trajectory features from environmental feature expressions through the cross-attention mechanism; A learnable query is used as input to the decoder to extract key information for trajectory prediction.

5. The automatic driving trajectory prediction method based on transformer architecture according to claim 1, characterized in that: The prediction output step includes predicting a multimodal trajectory in response to the trajectory features, wherein the prediction includes the probability of the trajectory, the mean and variance of the trajectory points, and further includes: Input the output trajectory features into the GMM prediction head; The three fully connected layer branches based on the GMM prediction head predict the probability of the trajectory, the mean and variance of the trajectory points respectively; Generate a multimodal prediction trajectory based on the prediction results; During the model training process, loss calculation is used to combine the prediction results and the actual trajectory data to calculate the trajectory classification loss and trajectory regression loss to optimize the model parameters.

6. The automatic driving trajectory prediction method based on transformer architecture according to claim 1, characterized in that: Before the feature expression step, the method further includes: Before model training, use open source datasets for data processing to obtain input data that meets feature characterization requirements; Collect real-car data in specific environments and fine-tune the model to adapt to actual driving scenarios; Based on the model training results, adjust the model parameters to optimize prediction accuracy and performance.

7. An autonomous driving trajectory prediction system based on transformer architecture, characterized in that: include: A feature expression module, configured to respond to multimodal input information required for autonomous driving motion prediction and perform corresponding feature representation; A coding processing module, configured to extract and encode the deep-level features of the environment in response to the multimodal input information after the feature representation, so as to obtain an environmental feature expression; A trajectory feature extraction module is configured to extract trajectory features in response to the encoded environmental feature expression based on a cross-attention mechanism, wherein the trajectory features include key information for trajectory prediction; A prediction output module is configured to predict a multimodal trajectory in response to the trajectory features, wherein the prediction includes a probability of the trajectory, a mean and a variance of the trajectory points.

8. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: It stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method described in any one of claims 1 to 6.

10. A vehicle, characterized in that: The vehicle is provided with an automatic driving trajectory prediction system based on a transformer architecture as described in claim 7.

Images