Vehicle trajectory prediction method based on pre-trained large language model

Through the vehicle trajectory prediction method based on pre-trained large language model, combined with the Mamba layer and multimodal Gaussian decoder, the decision-making process of human drivers is simulated, and the problem of insufficient scenario cognition and understanding ability in vehicle trajectory prediction is solved, and the problem of improved Predictive capabilities and mobility of autonomous vehicles.

CN119089287APending Publication Date: 2024-12-06BEIHANG UNIV

Patent Information

Application Number
CN202411159060.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing vehicle trajectory prediction methods have a gap compared with human drivers in terms of scene cognition and understanding, and it is difficult to effectively predict the movement of surrounding vehicles.

Method used

Using a vehicle trajectory prediction method based on a pre-trained large language model, by obtaining space-time scenarios of vehicle state and lane information, using an embedded network to extract high-dimensional features, fuse vehicle information and lane information, build a large language model and introduce trainable lateral directions. The layer, combined with the Mamba layer and a multimodal Gaussian decoder, performs sparse context joint encoding and lane-aware learning, simulates the decision-making process of human drivers.

Benefits of technology

The scenario cognition and understanding ability of the prediction model is enhanced, the trajectory prediction is close to human choice, and the prediction ability and maneuverability of autonomous driving vehicles in complex traffic scenarios is improved, so as to effectively avoid collisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089287A_ABST
    Figure CN119089287A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of intelligent traffic and automatic driving, and particularly relates to a pre-trained large language model-based vehicle trajectory prediction method, which comprises the following steps of: obtaining a space-time scene of a vehicle state and lane information; encoding the space-time scene of the vehicle state and the lane information, including extracting high-dimensional features through an embedded network, fusing the high-dimensional features to perform information exchange of the vehicle information and the lane information, and outputting sparse context joint codes; constructing a large language model; the sparse context joint coding obtains an advanced interaction state based on a pre-trained large language model; the motion and lane information of the target vehicle are synchronized through the Mama layer at each time step, prediction scores of a plurality of lane segments in the corresponding time step are obtained, the prediction score of the lane segment closest to the real track position is 1, and the prediction scores of the other lane segments are 0; the second input is processed by a multi-modal Gaussian decoder, resulting in a vehicle trajectory in consecutive time steps.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure belongs to the field of intelligent transportation and autonomous driving technology, and specifically relates to a vehicle trajectory prediction method based on a pre-trained large language model. Background Art

[0002] Vehicle trajectory prediction plays a vital role in autonomous driving systems, providing rich perception cues for downstream planning and decision-making modules. However, due to the inherent randomness of driving behavior, it is still challenging to enable autonomous vehicles to have the ability to predict the movement of surrounding vehicles like humans.

[0003] Existing research mainly uses different deep network modules to explore temporal dynamics and spatial interactions. Specifically, most studies use RNN and Transformer to encode temporal dependencies, and apply pooling mechanisms and GCN to integrate information from surrounding vehicles. Although these methods can significantly improve trajectory prediction performance, there is still a gap in scene cognition and understanding compared to human drivers.

[0004] Recently, large language models (LLMs) have shown great potential in many tasks that mimic human understanding and reasoning capabilities. At the same time, LLMs can provide high-level semantics and extensive knowledge, and this emergent ability has expanded its application beyond the field of language processing. Exploring the application of LLMs in trajectory prediction will enhance the scene recognition and understanding capabilities of the prediction model. Summary of the invention

[0005] The present disclosure is proposed based on the above-mentioned needs of the prior art. The technical problem to be solved by the present disclosure is to provide a vehicle trajectory prediction method based on a pre-trained large language model to enhance the scene recognition and understanding capabilities of the prediction model, thereby making the trajectory prediction close to human selection.

[0006] In order to solve the above problems, the technical solutions provided by the present disclosure include:

[0007] A vehicle trajectory prediction method based on a pre-trained large language model is provided, comprising: obtaining a spatiotemporal scenario of vehicle status and lane information; encoding the spatiotemporal scenario of the vehicle status and lane information, including extracting high-dimensional features through an embedding network, wherein the embedding network is composed of a gated loop unit and a multi-layer perceptron; fusing the high-dimensional features to exchange vehicle information and lane information, and outputting sparse context joint coding; constructing a large language model, freezing some pre-trained parameters, introducing a trainable lateral layer, and using a low-rank adaptive update attention layer to form a large language model; the sparse context joint coding obtains a high-level interaction state based on the pre-trained large language model; synchronizing the motion and lane information of the target vehicle through a Mamba layer at each time step, wherein the Mamba layer includes a Mamba block, a normalization layer, a point-by-point feedforward network, and a residual connection; the Mamba layer performs a sparse context joint coding on the high-level interaction state and The first input formed by partial sparse context joint coding is batch normalized to obtain a first output; the first output is mapped through the Mamba block to obtain a second output; based on the first output and the second output, the implicit state is obtained by combining batch normalization and residual connection; the lane perception estimation modeling in the hidden layer dimension is improved by a point-by-point feedforward network to obtain a third output; based on the implicit state and the third output, the lane perception learning vector is obtained by batch normalization and residual connection; the lane perception learning vector enters the multi-layer perceptron to obtain the prediction scores of multiple lane segments in the corresponding time step, the prediction score of the lane segment closest to the true trajectory position is 1, and the prediction scores of the remaining lane segments are 0; the second input is processed by a multimodal Gaussian decoder, the second input is composed of sparse context joint coding, high-level interaction features and latent vectors sampled by multivariate normal distribution, and the vehicle trajectory in continuous time steps is obtained.

[0008] Preferably, the fusing of high-dimensional features to exchange vehicle information and lane information and outputting sparse context joint coding includes: fusing features between vehicles through a multi-head self-attention mechanism and a gated linear unit; updating vehicles and lanes through a multi-head attention mechanism and a residual connection to achieve feature fusion between vehicles and between vehicles and lanes; expressed as: , where h i and f l is the high-dimensional feature proposed by embedding the spatiotemporal scene of vehicle status and vehicle information through the network, MultiSelfAtt() is the multi-head self-attention mechanism, GLU() is the gated linear unit, N is the number of vehicles, and L' is the number of vehicle segments; and Connect them to form the coefficient context joint coding g i .

[0009] Preferably, using low-rank adaptive updating of the attention layer includes: assuming that the rank of the low-rank adaptive LoRA is r, the j-th attention layer network θ j The input is a j , whose dimension is d, and the output is Its dimension is k, and LoRA approximates the network θ by j The pre-trained weight matrix W∈R in d×k : Among them, the rank decomposition matrix B∈R d×r , A∈R r×k , and r<<min(d,k); during the training process, freeze W∈R d×k , B and A are trainable parameters, initialized to 0 and Gaussian distribution respectively.

[0010] Preferably, the forward transfer function of LoRA is expressed as: The context is jointly encoded g i Input into the large language model LLMs(), and the output is: z i =LLMs(g i ), z i Transformed by a multi-layer perceptron to match g i , thereby generating the final high-level interaction state s i .

[0011] Preferably, the Mamba layer performs batch normalization on the first input to obtain a first output; and maps the first output through the Mamba block to obtain a second output, including: Q=Mamba(BatchNorm(F)), wherein Q is the second output, BatchNorm() is batch normalization, and Mamba() represents the output after being processed by the Mamba block. F is the first input, s i For advanced interaction status, is the sparse context joint encoding of the parts, B is the batch size, L is the lane segment length, and D is the hidden layer dimension.

[0012] Preferably, the processing of the Mamba block includes expanding the input dimension by a linear projection with an expansion factor of E, generating different representations of two parallel branches, denoted as m and n respectively; capturing the lane perception dependency m′ by a one-dimensional convolution and a SiLU activation function, linearly projecting m′ into O, C and Δ respectively, Δ is used to transform O and generate the structured state matrix U and The selective state model accepts m′, and C are used as input to generate the modified lane perception feature q; the SiLU activation function is introduced to generate the gating signal n′ that can filter out irrelevant information; q is multiplied by n′ and output Q after linear projection.

[0013] Preferably, the obtaining of the implicit state based on the first output and the second output by combining batch normalization and residual connection comprises: in is an implicit state; the lane perception estimation modeling in the hidden layer dimension is improved by the point-by-point feedforward network, and the third output includes: in, For the third output, W (0) , W (1) , b (0) and b (1) is a trainable parameter ReLU() is a nonlinear activation function; the lane perception learning vector obtained by batch normalization and residual connection based on the implicit state and the third output includes: Where S is the lane perception learning vector.

[0014] Preferably, the lane perception learning vector enters a multi-layer perceptron to obtain prediction scores of multiple lane segments in a corresponding time step, including: Among them, p l,t is the prediction score of the lth lane segment at time t; S l is the lane perception learning vector of the lth lane segment, S j is the lane perception learning vector of the j-th lane segment, and exp() is an exponential function with the natural constant e as the base.

[0015] Preferably, lane perception estimation modeling optimizes the probability through a binary cross entropy loss function, expressed as: Among them, L lane is the binary cross entropy loss function, L CE Expressed as a binary cross entropy function, p t Expressed as the prediction score, Represented as the actual score label, each time step t∈{1,...,t f}.

[0016] Preferably, the second input e is processed by a multimodal Gaussian decoder i , the second input consists of sparse context joint encoding, high-level interaction features and latent vectors sampled from multivariate normal distribution, obtaining the vehicle trajectory in consecutive time steps s t For the trajectory, is the mean, N is the Gaussian distribution, p i is the modal likelihood, with a mean of The covariance is The likelihood p of K modes 1:K The correlation in time is shown, i By sparse context joint encoding gi , advanced interactive features for lane-aware guidance and latent vectors o sampled from a multivariate normal distribution, s i and M′ perform cross attention generation M′ is formed by connecting the candidate lane segments with the highest test scores.

[0017] Compared with the prior art, the present invention predicts trajectories by introducing the advanced capabilities of a pre-trained large language model in trajectory prediction tasks, and drives lane perception learning based on the Mamba module to simulate human cognitive activities of paying attention to potential lanes during driving decision-making, thereby enhancing the understanding ability of the pre-trained large language model for the spatiotemporal scene of lane information in vehicle trajectory prediction, and facilitating the guidance of the vehicle's motion state and potential lane segments, so that the final vehicle trajectory prediction result can be as close as possible to the driving of a human driver, so that the autonomous driving vehicle has a human-like ability to predict the movement of surrounding vehicles, so as to effectively avoid collisions, while improving maneuverability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0019] Figure 1 This is a step flow section of a vehicle trajectory prediction method based on a pre-trained large language model in an embodiment of the present invention;

[0020] Figure 2 Schematic diagram of the processing process of the Mamba block in an embodiment of the present invention;

[0021] Figure 3 Schematic diagram of the structure of the pre-trained large language model in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] In the description of the embodiments of the present disclosure, it should be noted that, unless otherwise clearly specified and limited, the term "connected" should be understood in a broad sense, for example, it can be a fixed connection, or a detachable connection, or an integral connection can be a mechanical connection, or an electrical connection can be a direct connection, or it can be indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.

[0024] The terms "top", "bottom", "above", "lower", and "on" used throughout the description are relative to the relative positions of components of a device, such as the relative positions of the top and bottom substrates within a device. It is understood that devices are multifunctional regardless of their orientation in space.

[0025] To facilitate understanding of the embodiments of the present application, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present application.

[0026] This embodiment provides a vehicle trajectory prediction method based on a pre-trained large language model. Figure 1-Figure 3 shown.

[0027] Get the spatiotemporal scenario of vehicle status and lane information.

[0028] Encoding the spatiotemporal scenario of the vehicle state and lane information includes extracting high-dimensional features through an embedding network, wherein the embedding network is composed of a gated loop unit and a multi-layer perceptron; fusing the high-dimensional features to exchange information between the vehicle information and the lane information, and outputting a sparse context joint code.

[0029] For each input, an embedding network consisting of GRU and MLP is used to extract high-dimensional features. For example, when the input is the vehicle state, the embedding network outputs the tensor h i ; For the spatiotemporal scenario where the input is lane information, the output tensor through the embedding network is f l .

[0030] The generated tensor h i and f l It is then fed into the fusion submodule to facilitate complex information exchange of vehicle status and lane information in local areas. This process is similar to the working principle of token embedding in LLMs (large language models).

[0031] The method of fusing high-dimensional features to exchange vehicle information and lane information and outputting sparse context joint coding includes: fusing features between vehicles through a multi-head self-attention mechanism MultiSelfAtt and a gated linear unit GLU; updating vehicles and lanes through a multi-head attention mechanism and residual connections to achieve feature fusion between vehicles and between vehicles and lanes.

[0032] Specifically expressed as:

[0033] It is expressed as:

[0034]

[0035]

[0036] where h i and f l It is a high-dimensional feature proposed after the spatiotemporal scene of vehicle status and vehicle information is embedded in the network. MultiSelfAtt() is a multi-head self-attention mechanism, GLU() is a gated linear unit, N is the number of vehicles, and L' is the number of vehicle sections.

[0037] Will get and Connect them to generate sparse context joint coding g i These encodings intuitively reflect the local receptive field correlations between vectorized entities.

[0038] Trajectory transitions follow patterns governed by high-level constraints arising from various factors in the scene. To learn these high-level interactions, the present invention explores the ability of LLMs to model various inherent dependencies in trajectory prediction tasks. Despite the similarities between trajectory data and natural language text, it is inappropriate to directly use LLMs to process sparse context joint encoding because general pre-trained LLMs are mainly used for text data processing. An alternative is to fully retrain the entire LLMs, but this process requires a lot of computing resources and is therefore less feasible. Another more effective solution is to fine-tune the pre-trained LLMs using parameter efficient fine-tuning (PEFT) technology. PEFT can greatly optimize the downstream tasks of pre-trained LLMs by adjusting or introducing trainable parameters.

[0039] Build large language models. Use pre-trained LLMs for natural language processing based on the Transformer architecture for advanced interaction modeling, such as Figure 3 shown.

[0040] Specifically, some pre-trained parameters are frozen, and trainable lateral layers are introduced. The attention layer is updated using low-rank adaptiveness to form a large language model. Sparse context joint coding obtains high-level interaction states based on the pre-trained large language model.

[0041] Assume that the rank of low-rank adaptive LoRA is r, and the j-th attention layer network θ j The input is a j , whose dimension is d, and the output is Its dimension is k, and LoRA approximates the network θ by j The pre-trained weight matrix W∈R in d×k :

[0042]

[0043] Among them, the rank decomposition matrix B∈R d×r , A∈R r×k , and r<<min(d,k). During the training process, freeze W∈R d ×k , B and A are trainable parameters, initialized to 0 and Gaussian distribution respectively.

[0044] The forward transfer function of LoRA is expressed as:

[0045]

[0046] On this basis, the context joint encoding g i Input into the large language model LLMs() and the output is:

[0047] z i =LLMs(g i )

[0048] z i Transformed by a multi-layer perceptron (MLP) to match g i , thereby generating the final high-level interaction state s i .

[0049] Most experienced drivers will directly select potential lane segments based on their driving experience, rather than indiscriminately traversing all information. In order to simulate human cognitive functions and further enhance the scene understanding ability of Traj-LLM, the present invention adopts lane perception probability learning to continuously estimate the probability of motion state and lane segment alignment.

[0050] Specifically:

[0051] The target vehicle’s motion and lane information are synchronized through the Mamba layer at each time step.

[0052] Mamba has a selective attention mechanism that can correct and summarize relevant information through input-dependent adaptation and embedded hidden states. This gated selection mechanism is similar to the complex decision-making process of human drivers and can promote lane perception probabilistic learning.

[0053] The Mamba layer includes a Mamba block, a normalization layer, a point-by-point feedforward network and a residual connection. Further, the Mamba layer has three normalization layers and one point-by-point feedforward network.

[0054] like Figure 2 As shown, the Mamba layer performs batch normalization on the first input formed by jointly encoding the high-level interaction state and part of the sparse context to obtain a first output; the first output is mapped through the Mamba block to obtain a second output.

[0055] Specifically expressed as:

[0056] Q = Mamba(BatchNorm(F))

[0057] Among them, Q is the second output, BatchNorm() is batch normalization, and Mamba() represents the output after processing by the Mamba block. F is the first input, s i For advanced interaction status, is the sparse context joint encoding of the parts, B is the batch size, L is the lane segment length, and D is the hidden layer dimension.

[0058] Furthermore, the processing of the Mamba block includes expanding the input dimension by a factor of E using linear projection to generate different representations of two parallel branches, denoted as m and n. One of the branches captures the lane perception dependency m′ through a one-dimensional convolution and SiLU activation function, and linearly projects m′ into O, C, and Δ, respectively. Δ is used to transform O and the structured state matrix U to generate and The selective state model accepts m′, and C are used as input to generate the modified lane perception feature q. At the same time, the other branch introduces the SiLU activation function to generate a gating signal n′ that can filter out irrelevant information; q is multiplied by n′ and output Q after linear projection.

[0059] To enhance the robustness of the model, based on the first output and the second output, batch normalization and residual connection are combined to obtain the implicit state, which is expressed as: in It is an implicit state.

[0060] The lane perception estimation modeling in the hidden layer dimension is improved by a point-by-point feedforward network, and the third output is obtained, which is expressed as: in, For the third output, W (0) , W (1) , b (0) and b (1) is a trainable parameter and ReLU() is a nonlinear activation function.

[0061] Based on the implicit state and the third output, the lane perception learning vector is obtained through batch normalization and residual connection:

[0062] The lane perception learning vector enters the multi-layer perceptron to obtain the prediction scores of multiple lane segments in the corresponding time step. The prediction score of the lane segment closest to the actual trajectory position is 1, and the prediction scores of the remaining lane segments are 0.

[0063] The prediction score for the lth lane segment at time t:

[0064]

[0065] Among them, p l,t is the prediction score of the lth lane segment at time t; S l is the lane perception learning vector of the lth lane segment, S j is the lane perception learning vector of the j-th lane segment, and exp() is an exponential function with the natural constant e as the base.

[0066] Experienced drivers will pay attention to multiple potential lane segments to make effective decisions. To this end, the present invention selects the c lane segments {v1,v2,...,v c} as candidate lane segments, and the scores of these lane segments are {p1,p2,...,p c Furthermore, M′ is formed by connecting the candidate lane segments with the highest test scores.

[0067] Lane perception estimation modeling optimizes the probability through a binary cross entropy loss function, expressed as:

[0068]

[0069] Among them, L lane is the binary cross entropy loss function, L CE Expressed as a binary cross entropy function, p t Expressed as the prediction score, Represented as the actual score label, each time step t∈{1,...,t f}.

[0070] Here, the ground truth score of the lane segment closest to the true trajectory position is set to 1, while the ground truth scores of other lane segments are set to 0. The distance d(l,y0) between lane segment l and the true position is calculated by the Euclidean distance: d(l,y0) = ||l-y0|| 2 .

[0071] The second input, which consists of a sparse contextual joint encoding, high-level interaction features, parameters formed by the highest-scoring candidate lanes, and a latent vector sampled from a multivariate normal distribution, is processed through a multimodal Gaussian decoder to obtain the vehicle trajectory in consecutive time steps.

[0072] The second input i By sparse context joint encoding g i , advanced interactive features for lane-aware guidance and a latent vector o sampled from a multivariate normal distribution. By i and M′ perform cross attention generation, which can guide the target vehicle to select candidate lane segments like a human driver. This embodiment uses an MLP and a Softmax function to predict the mixing coefficients, and applies two parallel MLPs to generate and

[0073] After receiving the second input, the multimodal Gaussian decoder outputs a set of trajectories Among them, s t For the trajectory, is the mean, N is the Gaussian distribution, p i is the modal likelihood, with a mean of The covariance is The likelihood p of K modes 1:K Shows correlation in time.

[0074] The TrajPred-LLM framework constructed in this embodiment is the first trajectory prediction method based on pre-trained LLMs that does not require explicit prompting engineering. By injecting the advanced capabilities of LLMs into the trajectory prediction task, TrajPred-LLM is able to understand and utilize rich high-level knowledge, such as scene context and interaction information, significantly improving the accuracy of vehicle trajectory prediction. The introduction of lane perception probability learning methods imitates the cognitive function of human drivers to focus on critical lanes during the decision-making process, thereby improving the scene understanding ability of the model. The present invention provides a general and adaptable vehicle trajectory prediction solution, which significantly improves the prediction ability of autonomous driving systems in complex traffic scenarios.

[0075] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A vehicle trajectory prediction method based on a pre-trained large language model, characterized in that: include: Get the spatiotemporal scenario of vehicle status and lane information; encoding the spatiotemporal scenario of the vehicle state and lane information, including extracting high-dimensional features through an embedding network, the embedding network consisting of a gated loop unit and a multi-layer perceptron; fusing the high-dimensional features to exchange information between the vehicle information and the lane information, and outputting a sparse context joint code; Build a large language model, freeze some pre-trained parameters, introduce trainable lateral layers, and use low-rank adaptive update of the attention layer to form a large language model; sparse context joint coding obtains high-level interaction states based on the pre-trained large language model; The motion and lane information of the target vehicle are synchronized at each time step through the Mamba layer, and the Mamba layer includes a Mamba block, a normalization layer, a point-by-point feedforward network and a residual connection; the Mamba layer performs batch normalization processing on the first input formed by the joint encoding of the high-level interaction state and part of the sparse context to obtain a first output; the first output is mapped through the Mamba block to obtain a second output; based on the first output and the second output, the implicit state is obtained by combining batch normalization and residual connection; the lane perception estimation modeling in the hidden layer dimension is improved through the point-by-point feedforward network to obtain a third output; based on the implicit state and the third output, the lane perception learning vector is obtained through batch normalization and residual connection; the lane perception learning vector enters the multi-layer perceptron to obtain the prediction scores of multiple lane segments in the corresponding time step, the prediction score of the lane segment closest to the true trajectory position is 1, and the prediction scores of the remaining lane segments are 0; The second input, which consists of a sparse contextual joint encoding, high-level interaction features, parameters formed by the highest-scoring candidate lanes, and a latent vector sampled from a multivariate normal distribution, is processed through a multimodal Gaussian decoder to obtain the vehicle trajectory in consecutive time steps.

2. The vehicle trajectory prediction method based on a pre-trained large language model according to claim 1, characterized in that: The fusing of high-dimensional features to exchange vehicle information and lane information and outputting sparse context joint coding comprises: Feature fusion between vehicles through multi-head self-attention mechanism and gated linear unit; Through the multi-head attention mechanism and residual connection, vehicles and lanes are updated to achieve feature fusion between vehicles and between vehicles and lanes; It is expressed as: where h i and f l It is the high-dimensional features proposed by embedding the spatiotemporal scene of vehicle status and vehicle information through the network, MultiSelfAtt() is the multi-head self-attention mechanism, GLU() is the gated linear unit, N is the number of vehicles, and L' is the number of vehicle segments; Will and Connect them to form the coefficient context joint encoding gi.

3. The vehicle trajectory prediction method based on a pre-trained large language model according to claim 1, characterized in that: Using low-rank adaptive updating of the attention layer includes: Assume that the rank of low-rank adaptive LoRA is r, and the j-th attention layer network θ j The input is a j , whose dimension is d, and the output is Its dimension is k, and LoRA approximates the network θ by j The pre-trained weight matrix W∈R in d×k : Among them, the rank decomposition matrix B∈R d×r , A∈R r×k , and r<<min(d,k); During training, freeze W∈R d×k , B and A are trainable parameters, initialized to 0 and Gaussian distribution respectively.

4. The vehicle trajectory prediction method based on a pre-trained large language model according to claim 3 is characterized in that: The forward transfer function of LoRA is expressed as: The context joint encoding gi is input into the large language model LLMs(), and the output is: z i =LLMs(g i ) z i Transformed by a multi-layer perceptron to match g i , thereby generating the final high-level interaction state s i .

5. The vehicle trajectory prediction method based on a pre-trained large language model according to claim 1, characterized in that: The Mamba layer performs batch normalization on the first input to obtain a first output; the first output is mapped through the Mamba block to obtain a second output, including: Q = Mamba(BatchNorm(F)) Among them, Q is the second output, BatchNorm() is batch normalization, and Mamba() represents the output after processing by the Mamba block. F is the first input, s i For advanced interaction status, is the sparse context joint encoding of the parts, B is the batch size, L is the lane segment length, and D is the hidden layer dimension.

6. The vehicle trajectory prediction method based on a pre-trained large language model according to claim 5, characterized in that: The processing of the Mamba block involves expanding the input dimension by a factor of E using linear projection, generating two different representations of parallel branches, denoted as m and n; The lane perception dependency m′ is captured by one-dimensional convolution and SiLU activation function, and m′ is linearly projected into O, C and Δ respectively. Δ is used to transform O and generate the structured state matrix U and The selective state model accepts m′, and C as input to generate the modified lane perception feature q; The SiLU activation function is introduced to generate a gating signal n′ that can filter out irrelevant information; q is multiplied by n′ and then linearly projected to output Q.

7. The vehicle trajectory prediction method based on a pre-trained large language model according to claim 6 is characterized in that: The step of obtaining an implicit state based on the first output and the second output by combining batch normalization and residual connection includes: in It is an implicit state; The lane perception estimation modeling in the hidden layer dimension is improved by the point-by-point feedforward network to obtain the third output including: in, For the third output, W (0) , W (1) , b (0) and b (1) ReLU() is a trainable parameter and a nonlinear activation function; The lane perception learning vector obtained by batch normalization and residual connection based on the implicit state and the third output includes: Where S is the lane perception learning vector.

8. The vehicle trajectory prediction method based on a pre-trained large language model according to claim 7, characterized in that: The lane perception learning vector enters the multi-layer perceptron to obtain the prediction scores of multiple lane segments in the corresponding time step, including: Among them, p l,t is the prediction score of the lth lane segment at time t; S l is the lane perception learning vector of the lth lane segment, S j is the lane perception learning vector of the j-th lane segment, and exp() is an exponential function with the natural constant e as the base.

9. The vehicle trajectory prediction method based on a pre-trained large language model according to claim 8, characterized in that: Lane perception estimation modeling optimizes the probability through a binary cross entropy loss function, expressed as: Among them, L lane is the binary cross entropy loss function, L CE Expressed as a binary cross entropy function, p t Expressed as the prediction score, Represented as the actual score label, each time step t∈{1,...,t f }.

10. The vehicle trajectory prediction method based on a pre-trained large language model according to claim 1, characterized in that: Process the second input e through the multimodal Gaussian decoder i , the second input consists of sparse context joint encoding, high-level interaction features and latent vectors sampled from multivariate normal distribution, obtaining the vehicle trajectory in consecutive time steps s t For the trajectory, is the mean, N is the Gaussian distribution, p i is the modal likelihood, with a mean of The covariance is The likelihood p of K modes 1:K The correlation in time is shown, i By sparse context joint encoding g i , advanced interactive features for lane-aware guidance and latent vectors o sampled from a multivariate normal distribution, s i and M′ perform cross attention generation M′ is formed by connecting the candidate lane segments with the highest test scores.

Citation Information

Patent Citations

  • Vehicle trajectory prediction method based on lane point future trajectory offset auxiliary supervision

    CN116403176A

  • Target vehicle track prediction method for multi-lane scene

    CN116758727A

  • Vehicle scene arrangement service method and system, intelligent vehicle and storage medium

    CN117519684A

  • Artificial Intelligence for Vehicle Performance and Tracking

    US20240144016A1

Cited By

  • Urban traffic flow prediction method and system fusing spatial-temporal characteristics and large language model

    CN119649599A

  • Transform network-based goods stacking track prediction and control method for clamping and holding vehicle

    CN119960438A

  • Traffic flow prediction method and system based on space-time large language model, and medium

    CN120071606A

  • Digital mechanical equipment data management system based on cloud computing

    CN120180251A