Traffic large model construction and decision-making method and device based on multi-modal two-way map reasoning

Abstract

The invention discloses a traffic large model construction and decision-making method and device based on multi-modal two-way map reasoning, and the method comprises the steps: constructing a multi-modal data set of a text, an image and a track, generating fusion features through spatial-temporal clustering and cross-modal Transform coding, carrying out the two-way map reasoning in combination with a traffic knowledge map, and carrying out the decision-making of the traffic large model. The method comprises the following steps: generating an embedded representation through a forward graph neural network, reversely mapping a decision scheme generated by a language model to a graph to verify consistency, outputting knowledge to enhance embedding, fusing multi-modal features and knowledge embedding by adopting an LoRA multi-task joint fine tuning technology, adapting to traffic field tasks, deploying a real-time inference engine, and carrying out real-time inference on the traffic field. And processing the dynamic data flow through an aging perception attention mechanism, and outputting traffic event identification, path planning and scene question and answer results in parallel. Compared with the prior art, the method has the advantages that the problems of insufficient multi-source heterogeneous data fusion, low knowledge utilization efficiency and poor real-time decision consistency can be solved, and the semantic understanding and decision accuracy of the traffic large model is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and intelligent transportation, in particular to a traffic large model construction and decision method and device based on multi-modal bidirectional graph reasoning, which is suitable for traffic event analysis, real-time path planning and scene decision support. BACKGROUND

[0002] Large-scale pre-training language models (LLM) have made significant progress in open-domain text understanding and generation. Models such as GPT-4, Gemini, LLaMA-3, etc. have reached a quasi-commercial level in dialogue, code completion, and reasoning tasks, and with the help of LoRA and other parameter-efficient fine-tuning technologies, they are gradually moving towards industry landing. When landing in vertical industries such as the transportation field, general LLMs still have three outstanding limitations: first, they lack the ability to model "strong structure" traffic knowledge such as road topology, signal control rules, and event handling processes, making it easy to produce "hallucination" outputs that violate industry common sense; second, they are difficult to handle multi-source heterogeneous data such as text, surveillance images, and vehicle trajectories, resulting in insufficient comprehensive reasoning capabilities for dynamic scenarios; third, they lack an automated arbitration mechanism for multi-task conflicts when performing tasks such as event identification and path planning in real-time parallel execution, resulting in reduced credibility of decision results.

[0003] To address the above problems, the academic and industrial communities have launched preliminary explorations. Multi-modal large models such as LLaVA and GPT-4V have achieved image-text alignment by introducing visual encoders, and the Retrieval-Augmented Generation (RAG) framework uses vector databases to retrieve external knowledge to reduce hallucination risk. In the transportation field, there are also prototype verification systems such as TransGPT and TrafficGPT. However, existing solutions typically focus on a single modality or static text and do not establish a spatiotemporal consistency encoding framework, making it impossible to capture the collaborative evolution of trajectories, images, and text in traffic events. Static retrieval mechanisms cannot achieve closed-loop verification of decisions and knowledge, and knowledge updates are lagging and prone to logical contradictions, making it impossible to meet the comprehensive needs of complex transportation decisions for high timeliness and strong expertise. SUMMARY

[0004] The present application provides a traffic large model construction and decision method and device based on multi-modal bidirectional graph reasoning, which solves the fusion bias problem caused by spatiotemporal misalignment by fusing cross-modal spatiotemporal encoding and sliding window attention. In addition, a bidirectional graph reasoning mechanism is designed to ensure the consistency of knowledge embedding and dynamic decision-making through a forward propagation + decision-making backward verification reasoning mechanism. LoRA multi-task fine-tuning is used to reduce computational overhead, and a timeliness-aware attention mechanism and path reliability arbitration are introduced to ensure decision accuracy and multi-task output consistency in intelligent transportation scenarios.

[0005] Technical Solution: This invention proposes a traffic model construction and decision-making method based on multimodal bidirectional graph reasoning, which includes the following steps:

[0006] Step 1: Based on traffic events, a multimodal dataset of text, images, and trajectories is constructed. The fused feature representation H is generated through spatiotemporal clustering and cross-modal Transformer encoding.

[0007] Step 2: Based on the fused feature representation H from step 1, bidirectional graph reasoning is performed in conjunction with the traffic knowledge graph. An embedding representation is generated through a forward graph neural network. The decision solution generated by the language model is then reverse-mapped to the graph based on the fused feature representation H and the embedding representation to verify consistency, and the knowledge-enhanced embedding representation set E is output.

[0008] Step 3: Using LoRA multi-task joint fine-tuning technology, the fused feature representation H and the knowledge-enhanced embedding representation set E are jointly modeled and optimized. Through the multi-task learning framework, domain-specific model parameters adapted to the traffic domain are obtained and the domain-adapted traffic model M is derived.

[0009] Step 4: Obtain the traffic data to be processed and use the spatiotemporal clustering algorithm to generate a multimodal spatiotemporal feature sequence U real Initialize the parallel reasoning channel of the three tasks of event classifier, path planning generator and scene question answering generator, and use the time-aware attention mechanism to process the dynamic data stream to obtain the multimodal fusion feature H with dynamic time constraints. live , for the multimodal fusion feature H live Generate knowledge enhancement sequence H′ based on knowledge enhancement decision making based on knowledge enhancement embedding representation set E live , and output the results of three tasks: traffic event recognition, path planning and scenario question answering in parallel.

[0010] Furthermore, the step 1 is as follows:

[0011] Step 1.1: For each traffic event, construct a multimodal dataset D consisting of three modalities: text, image, and trajectory. The text includes the traffic event description, sample unique identifier, text content, and corresponding timestamp; the image includes the RGB color image pixel matrix and the latitude and longitude coordinates of the shooting location; the trajectory includes the set of trajectory points arranged in time sequence for the same event;

[0012] Step 1.2: Preprocess the data of each modality, extract features, and construct spatiotemporal position encoding to form a unified multimodal input unit set Z;

[0013] The feature extraction calls the BERT-base pre-trained model, the ViT-base model, and the road node embedding model Embed based on graph neural network training. geo, obtain semantic feature representation text feature embedding, image feature embedding, trajectory feature embedding; the spatiotemporal position encoding includes spatial position encoding P geo and time position encoding P time , the text feature embedding, the image feature embedding, the trajectory feature embedding and the spatial position encoding vector P geo , the time position encoding vector P time are spliced in the channel dimension to form a multi-modal input unit, and a plurality of multi-modal input units form a multi-modal input unit set Z;

[0014] Step 1.3: Based on the multi-modal input unit obtained in step 1.2, the time threshold and the space threshold are aggregated into a spatiotemporal cluster, and the latest time in the cluster and the intra-cluster order are unfolded to generate a unified multi-modal spatiotemporal feature sequence U, and based on the multi-modal spatiotemporal feature sequence U, the multi-head self-attention fusion is performed in parallel in the sliding window cluster to obtain the local fused multi-modal spatiotemporal feature sequence U', and the multi-layer Transformer encoder is used to cross-modal modeling to obtain the fusion feature representation H of the fused semantic, spatiotemporal and modal features;

[0015] Step 1.4: Based on the fusion feature representation H, a cross-modal time alignment loss function L align is designed, and the loss is minimized in the self-supervised training.

[0016] Further, in step 1.3, the multi-modal input unit vector obtained in step 1.2 is aggregated into a spatiotemporal cluster with a preset time threshold and a space threshold, and the latest time in the cluster and the intra-cluster order are unfolded to generate a unified multi-modal spatiotemporal feature sequence U, which is as follows:

[0017] The multi-modal input unit vector z i is unfolded according to the collection timestamp tsi and the geographic coordinates gps i , the time window width Δt and the space radius Δd are set, and all units z k that satisfy the condition |ts i -ts k |≤Δt∧dist(gps i ,gps k )≤Δd are constructed into a spatiotemporal neighborhood cluster:

[0018] C m ={z k ∈Z∣|ts k -ts i |≤Δt,dist(gps k ,gps i )≤Δd},m∈[1,M]

[0019] Where M is the total number of spatiotemporal neighborhood clusters, tsi is the collection timestamp of the i-th unit, and gps k, ts j represent two sampling time stamps, gps k , gps j represent the geographical longitude and latitude coordinates corresponding to the two sampling time stamps respectively, and the maximum time stamp of each cluster C m is taken as the cluster ordering basis, and all clusters are arranged in ascending order, then all vectors z m in each cluster C k are flattened in the original time order to generate a multi-modal spatio-temporal feature sequence U = {u1, u2,..., u N}, |U| = N, which is equal to the total number of original input units, that is, the total number of multi-modal input unit vectors z i in the modal input unit Z;

[0020] The cross-modal alignment loss function L align is specifically:

[0021] For any two different encoding feature vectors (h i , h j ) in the feature set H, i≠j, if |ts i -ts j |≤Δt and dist(gps i , gps j )≤Δd, then (h i , h j ) is regarded as a positive example pair; otherwise, it is regarded as a negative example pair, and the cross-modal alignment loss is defined as:

[0022]

[0023] where 1(·) is an indicator function, only for the pair (i,j) that meets the spatio-temporal neighborhood condition is included in the loss, ts i , ts j and gps i , gps j correspond to the original collection time stamps and geographical coordinates of h i and h j respectively, Δt represents the time window threshold, Δd represents the spatial distance threshold, dist(gps i , gps j ) is the distance function between two geographical coordinates, Proj: is a shared linear mapping, is the feature vector after projection alignment, is the Euclidean norm square, which is used to measure the squared distance between feature vectors.

[0024] Further, the specific process of outputting the knowledge enhanced embedding representation E in step 2 is:

[0025] Step 2.1: Construct a multi-granularity traffic knowledge graph G = (V, E, R), where the node set V contains four types of entity nodes: roads, intersections, regions, and cities. The edge set E represents the connection between entities. The relationship type set R includes but is not limited to "connection", "adjacency", "attribution", "coverage", "restriction", "accident", and "construction".

[0026] Step 2.2: Based on the spatial position encoding of the multimodal feature fusion representation H and its corresponding index, align each fusion feature representation in the multimodal feature fusion representation H to a node set that matches the multi-granularity traffic knowledge graph and the spatial position encoding. Use a graph neural network with a multi-scale gated propagation mechanism and a relational attention aggregation strategy to perform forward inference on the graph and generate a forward embedding representation E. f ;

[0027] Step 2.3: Use the large language model LLM to compare H and E f Processing generates a traffic decision plan, and maps the traffic decision plan back to the knowledge graph G, triggering reverse graph reasoning to generate a reverse embedding representation E b ;

[0028] Step 2.4: Fusion of reverse embedding and forward embedding generates the final knowledge-enhanced embedding E, E = {e v ∣v∈V}.

[0029] Furthermore, the graph neural network with multi-scale gated propagation mechanism and relational attention aggregation strategy in step 2.2 is a heterogeneous graph neural network, specifically as follows:

[0030] Step 2.2.1: Traverse all nodes v in the knowledge graph G = (V, E, R);

[0031] Step 2.2.2: Representation based on multimodal feature fusion and its corresponding index spatial position code P geo , each Aligned to the traffic knowledge graph G and P geo The set of matching nodes And the initial embedding of these nodes Set to the corresponding l∈[1,L], T represents the number of layers of the Transformer encoder;

[0032] Step 2.2.3: Define a learnable gating function in is the embedding vector of node v in round t-1, P geo (v) is the spatial position encoding of node v, σ(·) is the element-wise Sigmoid function, bg ∈R d are learnable parameters, the output dimension is the same as the node embedding dimension; by the neighbor set of control node v where d hop (v, u) represents the shortest hop number between nodes v and u in the graph;

[0033] Step 2.2.4: For each edge (u→v) ∈ E in the graph, read the edge type r uv ∈R, and map it to a learnable relationship embedding vector e(r uv ) ∈R d , the last round of embedding after splicing with the relationship embedding e(r uv ) mapping:

[0034]

[0035] where W r ∈R 1×3d is a learnable linear mapping matrix, and the candidate neighbor set of node v perform soft maximum normalization to calculate the relationship attention weight:

[0036]

[0037] Step 2.2.5: For each node v, initialize the 0th round embedding is the node feature, for the tth iteration, t = 1, …, T, perform the following aggregation and update, neighbor information weighted aggregation:

[0038]

[0039] where W h ∈R d×d , b h ∈R d are learnable parameters, and the gate fusion update is:

[0040]

[0041] where is element multiplication, which ensures dynamic balance between new and old information for nodes;

[0042] Step 2.2.6: After T rounds of iteration, collect the final embedding of each node v ∈ V in the traffic knowledge graph G = (V, E, R), and organize all node final embedding vectors into a forward embedding set

[0043] Further, the step 2.3 is specifically operated as:

[0044] Step 2.3.1: Let multimodal feature fusion representation with forward graph embedding As the input of large language model (LLM), prompt instruction is designed to prompt LLM to combine H and E f Generate a series of preliminary traffic decision schemes denoted as: Natural language traffic decision instruction, t'∈[1,T'], T'∈N + , N + represents a set of positive integers, l∈[1,L], and T represents the number of layers of the Transformer encoder;

[0045] Step 2.3.2: For each traffic decision instruction in M , use natural language parsing or rule template technology to parse into a structured triple mapping set where each (v, r, u) represents node v acting on node u through relation r, based on M t′ , extract the corresponding subgraph G t′ = (V t′ , E t′ ) in the multi-granularity traffic knowledge graph G = (V, E, R)

[0046] V t′ = {v | (v, r, u) ∈ M t′} ∪ {u | (v, r, u) ∈ M t′},

[0047] E t′ = {(v, u) | (v, r, u) ∈ M t′}

[0048] For each subgraph node v ∈ V t′ , its initial embedding is taken from the forward embedding If v does not exist in E f , use static attribute mapping to generate

[0049] Step 2.3.3: For each G t′ = (V t′ , E t′ ), perform T b rounds of heterogeneous graph neural network backpropagation on the subgraph to obtain the backward embedding of each node v ∈ V t′ . Only iterate over the subset V t′ and E t′ : for t = 1, 2, …, Tb , for each v ∈ V t′ , first, the gating computation is performed as follows:

[0050]

[0051] Filtering candidate neighbors:

[0052]

[0053] where dist G (u, v) is the shortest hop number between u and v in graph G, is the initial propagation radius corresponding to node v; for each (u, v) ∈ E r , read its relation type r, and project the embedding of the last round concatenated with the relation embedding r:

[0054]

[0055] Normalize the relation list using Softmax:

[0056]

[0057] where a ∈ R d is a learnable vector, after T b rounds of iteration, the backward embedding of each node v in subgraph G t′ is obtained:

[0058]

[0059] Form the set of reverse embeddings Each time the subgraph V t′ generates a set of

[0060] Further, the specific process in step 3 is:

[0061] Step 3.1: Based on the fusion feature representation and the knowledge-enhanced embedding representation set E = {e v | v ∈ V}, for each feature vector h l in the sequence, according to its spatio-temporal position encoding, calculate the cosine similarity s v with each knowledge embedding e l,v , retrieve the node v * with the maximum similarity in the embedding library = argmax v s l,v and take the corresponding knowledge embedding as the fusion source, taking h l as the query, The sequence H′={h1′,...,h L ′} as the multimodal fusion representation after knowledge enhancement;

[0062] Step 3.2: Construct the LoRA enhancement module and set the main weight set W of the LLaMA-3-8B pre-trained model base ={W Q ,W K ,W V ,W FFN ,...} is loaded into the fine-tuning framework, where each weight matrix d is the model hidden layer dimension, for the query projection matrix W Q Introducing a pair of trainable low-rank matrices r<<d, and define the incremental matrix Similarly, for the key projection matrix W K , value projection matrix W V and the feedforward network weight W FFN Construct low-rank increments ΔW respectively K ,ΔW V ,ΔW FFN , after fine-tuning, the overall parameter set is:

[0063]

[0064] Keep W during fine-tuning base The original weights in are frozen, and only the incremental matrix group {A * ,B *} Perform gradient update;

[0065] Step 3.3: Construct a multi-task learning framework. In the downstream task set T = {T1, T2, T3} in the traffic field, define the traffic event recognition task T1, the classification head and cross entropy loss L1; define the travel path planning task T2, generate the head and sequence negative log-likelihood loss L2; ​​define traffic scene question answering task T3, generate head And the negative log-likelihood loss of the conditional language model; the loss of each task is calculated according to the preset weight λ i Combined into a unified multi-task objective function

[0066] Step 3.4: In the GPU training environment, perform multi-task joint optimization on the lightweight fine-tuning parameters and load the LoRA-enhanced model parameters Synchronously load the knowledge-enhanced fusion representation sequence H′={h1′,...,h L′} and its corresponding supervision labels, divided into several training subsets according to the batch size 14; for each training subset, according to Perform forward reasoning on H′ to obtain the output of each task. After the training is completed, the final LoRA incremental parameter ΔW and the basic model weight W are output. base Save together;

[0067] Step 3.5: Compare the fine-tuned LoRA incremental parameters ΔW with the base model weights W base Fusion, generate complete domain-specific model parameters W final , and get the final traffic model M.

[0068] Furthermore, in step 4, the time-sensitive attention mechanism is used to process the dynamic data stream to obtain the dynamic time-constrained multimodal fusion feature H live , as follows:

[0069] For the multimodal spatiotemporal feature sequence U real , introduce the time-sensitive attention mechanism and construct the dynamic attenuation mask matrix B live ∈R T×T :

[0070]

[0071] Among them, when |ts i -ts j |>τ live The mask value is -∞ when ts is set, otherwise it is 0. i 、ts j Represents two sampling timestamps, τ live To preset the timeliness threshold, the mask is injected into the attention calculation of each layer of the Transformer of the traffic model M. The calculation formula of the timeliness-aware attention head is defined as follows:

[0072]

[0073] The features output by the time-sensitive attention head are concatenated to obtain the dynamic time-constrained multimodal fusion feature H live ={h live,1 ,...,u live,L′}, whose semantic level is consistent with the multimodal fusion representation H in the training stage.

[0074] Furthermore, in step 4, the multimodal fusion feature H live Generate knowledge enhancement sequence H′ based on knowledge enhancement decision making based on knowledge enhancement embedding representation set E live , as follows:

[0075] Perform knowledge enhancement decision-making for each feature unit h in the sequence live,iRetrieving the most similar embedding in the knowledge-enhanced embedding representation set E of the knowledge graph:

[0076]

[0077] Using a cross-attention mechanism on each feature unit h live,i And the most similar embedding realizes feature depth fusion:

[0078]

[0079] Generate a knowledge-enhanced sequence H' live ={h' 1ive,1 ,...,u' live,L′} to the three-task channel.

[0080] The application also discloses a traffic large model construction and decision device based on multi-modal and bidirectional graph reasoning, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the computer program realizes the traffic large model construction and decision method based on multi-modal bidirectional graph reasoning when loaded into the processor.

[0081] Advantages

[0082] 1. The application significantly improves the semantic alignment capability of multi-source heterogeneous data such as text, image and trajectory through cross-modal spatio-temporal coding and sliding window multi-head attention mechanism, effectively overcomes the spatio-temporal misalignment problem in traditional methods, and substantially breaks through the accuracy of traffic event understanding.

[0083] 2. The innovative bidirectional graph reasoning architecture of the application deeply integrates structured knowledge into the decision-making closed loop, and through the cooperative mechanism of forward propagation and decision-making reverse verification, significantly reduces the abnormal output that violates traffic rules and road network constraints, and significantly enhances the decision reliability driven by knowledge.

[0084] 3. The application adopts LoRA multi-task joint fine-tuning technology, which significantly reduces the computational resource threshold of field adaptation while ensuring model performance, making it possible to efficiently deploy traffic large models under limited hardware conditions; the integrated time-sensitive attention mechanism and path reliability arbitration engine effectively solve the conflict problem of multi-task output in real-time scenarios, generate explainable decision basis through structured evidence chain, and comprehensively improve the timeliness, consistency and reliability of dynamic traffic decision-making. This scheme systematically solves the core bottleneck of traffic intelligence, providing key technical support for building a high-robustness, low-latency cognitive traffic system. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 The overall flowchart of the application;

[0086] Figure 2 is a multi-modal data fusion schematic diagram;

[0087] Figure 3 is a bidirectional graph reasoning structure schematic diagram;

[0088] Figure 4 is the loss function change of the multi-task learning framework of step 3 in the present application under different training rounds. DETAILED DESCRIPTION

[0089] The present application will be further illustrated below in conjunction with specific embodiments, which should be understood as merely illustrating the present application and not limiting the scope of the present application, and after reading the present application, various equivalent modifications of the present application by those skilled in the art fall within the scope defined by the appended claims.

[0090] The present application proposes a traffic large model construction and decision method based on multi-modal bidirectional graph reasoning, comprising the following steps:

[0091] Step 1: For each traffic event, a multi-modal data set D containing text, image and trajectory three modalities is constructed; the modal data is preprocessed, feature extracted and time-space position coding is constructed to form a unified modal input unit Z. A preset time threshold and a space threshold are aggregated into a time-space cluster, and are unfolded according to the latest time in the cluster and the order within the cluster to generate a unified multi-modal time-space feature sequence U. Based on the sequence U, multi-head self-attention fusion is performed in parallel within the sliding window cluster, multi-layer LlaMATransformer encoder is used for cross-modal modeling to obtain sequence-level feature representation H of fused semantic, time-space and modal features, and a cross-modal time alignment loss function L is designed. align The model output is optimized to improve the semantic fusion and space-time consistency modeling capability of the model for multi-source heterogeneous data, and the specific method is:

[0092] Step 1.1.1: define the original multi-modal data set as D={d i |d i =(Text i ,Img i ,Tra i )}, wherein the text set Text i =<uid i ,content i ,ts i > represents the i-th traffic event description text, uid i is a unique sample identifier, content i is the text content, tsi is the corresponding time stamp (unit: millisecond), Img i =<uid i ,pixel i, ts i , gps i represent the same event, pixel i RGB color image pixel matrix, gps i = <lat i , lon i > represents the latitude and longitude coordinates of the shooting location; Traj i = <uid i , seq i > represents the same event d i time sequence arrangement of trajectory point set, seq i = {p i,k │p i,k = <lat i,k , lon i,k , ts i,k , v i,k , a i,k >}, (lat i,k , lon i,k ) represents the latitude and longitude coordinates of the trajectory point p i,k , ts i,k is the timestamp, v i,k is the instantaneous speed calculated based on the distance between adjacent points and the time difference; a i,k is the instantaneous acceleration calculated based on the difference of the speed sequence, i∈[1,N], N is the total number of text entries, k∈[1,K], K is the number of trajectory point sequences.

[0093] Step 1.1.2: Extract semantic and structural features for each of the three modalities and generate supervised labels: discrete class encoding where C is the total number of predefined event categories; path node sequence question and answer pair provides annotation information for subsequent multi-modal fusion.

[0094] Step 1.2: Preprocess, extract features and construct spatio-temporal position encoding for each modality data to form a unified multi-modal input unit set Z, the specific steps are as follows:

[0095] Step 1.2.1: Data cleaning, normalization and sampling preprocessing are performed on the text set Text, image set Img and trajectory set Traj respectively, and unified conversion is performed to form structured sub-sequences. Each text content i is segmented into indefinite length token sequence tok i ; each image pixel i is normalized and divided into fixed size patch sequence patch i ; each trajectory seq iSampling and outlier rejection are performed to obtain a resampled trajectory point sequence s i .

[0096] Step 1.2.2: For the text token sequence tok i , the image patch sequence patch i and the trajectory point sequence s i , call the modal-specific embedding model respectively to obtain the semantic feature representation text token embedding image patch embedding trajectory point embedding respectively embody the text semantics, image spatial structure and trajectory spatio-temporal topology information.

[0097] Step 1.2.3: For each sample d i , through a learnable position encoder PosEnc ts , perform a sine-cosine embedding on the timestamp ts to generate a time position encoding P time = PosEnc ts (ts); use a set of learnable multi-layer perceptron MLP geo to map the geographic coordinates to generate a spatial position vector P geo = MLP geo (gps), to represent the embedding features of its geographic position.

[0098] Step 1.2.4: Concatenate the text feature embedding image feature embedding trajectory feature embedding obtained in step 1.2.2 with the spatial position encoding vector P geo and the time position encoding vector P time generated in step 1.2.3 in the channel dimension to form a multi-modal input unit All the z i together constitute a unified multi-modal input unit set Z = {z1, z2,..., z N}.

[0099] Step 1.3: Based on the multi-modal input unit obtained in step 1.2, aggregate into a spatio-temporal cluster with a preset time threshold and a spatial threshold, and expand according to the latest time in the cluster and the order within the cluster, to generate a unified multi-modal spatio-temporal feature sequence U, perform multi-head self-attention fusion in the sliding window cluster based on the multi-modal spatio-temporal feature sequence U to obtain a locally fused multi-modal spatio-temporal feature sequence U', use a multi-layer Transformer encoder to perform cross-modal modeling on U' to obtain a fusion feature representation H that fuses semantic, spatio-temporal and modal features, the specific steps are as follows:

[0100] Step 1.3.1: For the multimodal input unit vector z obtained in step 1.2.4 i Based on the collection timestamp tsi and geographic coordinates gps i Setting the time window width Δt and the space radius Δd will satisfy |ts k -ts i |≤Δt∧dist(gps k ,gps i )≤Δd condition for all units z k Construct spatiotemporal neighborhood clusters:

[0101] C m ={z k ∈Z||ts k -ts i |≤Δt,dist(gps k ,gps i )≤Δd},m∈[1,M]

[0102] Among them, M is the total number of spatiotemporal neighborhood clusters, ts k 、ts j Indicates two sampling timestamps, gps k ,gps j Represents the geographic latitude and longitude coordinates corresponding to the two sampling timestamps, with each cluster C m The maximum timestamp of each cluster C is used as the cluster sorting basis. After all clusters are sorted in ascending order, m All vectors z in k Flatten the vectors in the original time order to generate a multimodal spatiotemporal feature sequence U={u1,u2,...,u N}, |U|=N is equal to the total number of original input units, that is, the multimodal input unit vector z in the modal input unit Z i The total number of .

[0103] Step 1.3.2: Substitute the multimodal spatiotemporal feature sequence U = {u1,u2,...,u N According to the spatiotemporal neighborhood clusters to which each unit belongs, the same Transformer multi-head self-attention sub-layer is input in batches and in parallel, and the cross-modal information fusion is initially realized by the local self-attention mechanism, and the multimodal spatiotemporal feature sequence U′={u1′,u2′,...,u′ N}.

[0104] Step 1.3.3: Input the locally fused multimodal spatiotemporal feature sequence U′ into the Transformer encoder to generate a multimodal feature fusion representation H. The Transformer encoder is composed of T layers of the same structure, each layer contains a multi-head self-attention mechanism (Multi-Head Self-Attention) and a feed-forward network (Feed-Forward Network), and each sub-layer is followed by a residual connection and layer normalization (LayerNorm), and finally the fusion representation is obtained at the T layer. At this time, each They are the final multimodal fusion features of the lth spatiotemporal input unit in the sequence, which already contain the interactive information and spatiotemporal context from each modality.

[0105] Step 1.4: Fusion feature representation obtained in step 1.3 On this basis, a cross-modal alignment loss function L is constructed for all fusion representation pairs based on the spatiotemporal neighborhood. align , and minimize the loss in self-supervised training, for any two different encoding feature vectors (h i ,h j ), i≠j, if |ts i -ts j |≤Δt and dist(gps i ,gps j )≤Δd, then (h i ,h j ) is considered as a positive pair; otherwise it is considered as a negative pair, and the cross-modal alignment loss is defined as:

[0106]

[0107] Among them, 1(·) is an indicator function, which only counts the loss for the pair (i, j) that meets the spatiotemporal neighborhood conditions. is a shared linear map, is the feature vector after projection alignment, is the square of the Euclidean norm, which is used to measure the square distance between eigenvectors.

[0108] like Figure 2 As shown in the multimodal data fusion diagram, through parallel BERT text encoding, ViT image encoding and HMM+GCN trajectory encoding, the text token, image patch and trajectory point sequence are mapped to the same vector space, and then encoded through sliding window self-attention and multi-layer Transformer, and finally the fused feature sequence H is output.

[0109] Step 2: Based on the multimodal feature fusion representation H, bidirectional graph reasoning technology is used to integrate structured traffic knowledge, see Figure 3Bidirectional graph reasoning structure schematic diagram, the bidirectional graph reasoning technology specifically comprises: constructing a traffic knowledge graph G comprising multi-granularity nodes of roads, intersections, regions and cities and multi-type relationships; adopting a graph neural network with a multi-scale gating propagation mechanism and a relationship attention aggregation strategy to perform forward reasoning on the graph to generate forward embedding representation E f ; and mapping the traffic decision scheme generated by the language model based on H and E f back to the knowledge graph to trigger reverse graph reasoning to generate reverse embedding representation E b ; finally, weighting and integrating E f and E b according to learnable fusion weights to obtain knowledge-enhanced embedding representation E, and the specific method is as follows:

[0110] Step 2.1: Construct a multi-granularity traffic knowledge graph based on four types of entity nodes of roads (r), intersections (j), regions (a) and cities (c), define the graph G=(V, E, R), wherein, The edge set E represents the connection relationship between entities, and the relationship type set R includes but is not limited to static relationships such as “connect”, “adjacent”, “belongs”, “covers”, and dynamic event relationships such as “restriction”, “incident” and “construction”.

[0111] Step 2.2: Adopt a graph neural network with a multi-scale gating propagation mechanism and a relationship attention aggregation strategy to perform forward reasoning on the graph, in this embodiment, a heterogeneous graph neural network (Heterogeneous GNN) is adopted to complete multi-scale gating propagation and relationship attention aggregation, and finally generate forward embedding sequence E f The specific steps are as follows:

[0112] Step 2.2.1: Traverse all nodes v in the knowledge graph G=(V, E, R), read the type (road r i , intersection j j , region a k or city node c) to which the node belongs, and set the initial propagation radius reference r v according to the level of the node, if v is a road node, r v =R road ; if v is an intersection node, r v =R junction ; if v is a region node, r v =R area ; if v is a city node, r v =R city ; wherein R road <Rjunction <R area <R city is a preset constant.

[0113] Step 2.2.2: Representation based on multimodal feature fusion and its corresponding index spatial position code P geo , each Align (map) to the traffic knowledge graph G and P geo The set of matching nodes And the initial embedding of these nodes Set to the corresponding

[0114] Step 2.2.3: Define a learnable gating function in is the embedding vector of node v in round t-1, P geo (v) is the spatial position encoding of v, σ(·) is the element-wise Sigmoid function, b g ∈R d is a learnable parameter with the same output dimension as the node embedding dimension. This gate vector determines the proportion of each node "absorbing new information" from its neighbors during the iteration process. Control node v's neighbor set where d hop (v,u) represents the shortest number of hops between nodes v and u in the graph.

[0115] Step 2.2.4: For each edge (u→v)∈E in the graph, read the edge type r uv ∈R, and maps it into a learnable relational embedding vector e(r uv )∈R d , the previous round of embedding and relation embedding e(r uv ) Mapping after splicing:

[0116]

[0117] Where W r ∈R 1×3d is a learnable linear mapping matrix, the candidate neighbor set of node v (i.e. satisfying d hop (v,u)≤r v SoftmaxNormalization is performed on all u) to calculate the relational attention weights:

[0118]

[0119] Step 2.2.5: For each node v, initialize the 0th round embedding For node features, for the tth iteration (t = 1,..., T), perform the following aggregation and update, neighbor information weighted aggregation:

[0120]

[0121] where W h ∈R d×d , b h ∈R d are learnable parameters, and the gated fusion update:

[0122]

[0123] where is element-wise multiplication, ensuring dynamic balance between new and old information for nodes.

[0124] Step 2.2.6: After T rounds of iteration, collect the final embedding of each node v ∈ V in the traffic knowledge graph G = (V, E, R), and organize all node final embedding vectors into a forward embedding set

[0125] Step 2.3: Based on H and E f generated by the large language model, map the traffic decision scheme back to the knowledge graph G, trigger backward graph reasoning, and generate the reverse embedding representation E b , the specific steps are as follows:

[0126] Step 2.3.1: Let the multi-modal feature fusion representation and the forward graph embedding be the input of the large language model LLM, and through the pre-designed Prompt instruction prompt, let LLM jointly generate a series of preliminary traffic decision schemes f denoted as: Natural language traffic decision instructions, t' ∈ [1, T'], T' ∈ N + , l ∈ [1, L], T represents the number of Transformer encoder layers.

[0127] Step 2.3.2: For each traffic decision instruction in use natural language parsing or rule template technology to parse into a structured triple mapping set where each (v, r, u) represents node v acting on node u through relation r, based on M t′ , extract the corresponding subgraph G t′ = (V​t′ ,E t′ ), where:

[0128] V t′ = {v | (v, r, u) G M t′} U {u | (v, r, u) G M t′},

[0129] E t′ = {(v, u) | (v, r, u) G M t′}

[0130] For each subgraph node v G V t′ , its initial embedding is taken from the forward embedding If v is not present in E f , then use the static attribute mapping to generate

[0131] Step 2.3.3: For each G t′ = (V t′ , E t′ ), perform T b rounds of backward propagation on the subgraph using a Heterogeneous GNN to obtain the backward embedding t′ for each node v G V t′ , where the backward propagation updates the formula in each round similarly to Step 2.2, but only iterates over the subset V t′ and E b : for t = 1, 2,..., T t′ , for each v G V G , first perform a gating computation as follows:

[0132]

[0133] Filtering candidate neighbors:

[0134]

[0135] where dist r (u, v) is the shortest hop distance between u and v in graph G, is the initial propagation radius for node v in Step 2.2.1; for each (u, v) G E d , read its relation type r and project the last round embedding concatenated with the relation embedding r:

[0136]

[0137] Normalize the list of relations using Softmax

[0138]

[0139] where a ∈ R d are learnable vectors, and T b is the number of iterations. t′ the backward embedding of each node v in G

[0140]

[0141] constitute the set of backward embeddings Each time a subgraph V t′ is generated, a set of

[0142] Step 2.4: fuse the backward embeddings with the forward embeddings to generate the final knowledge-enhanced embeddings E, the specific steps are as follows:

[0143] Step 2.4.1: define a consistency measure:

[0144]

[0145] where is the forward embedding, is the backward embedding. If δ (t′) < τ (τ is a pre-set threshold), it is determined that the preliminary scheme is consistent with the knowledge graph, otherwise it is discarded or rolled back.

[0146] Step 2.4.2: for all nodes v ∈ V t′ that pass the test, i.e. δ (t′) < τ, calculate the mean of the node in all sets of backward embeddings to obtain the denoised backward feature:

[0147]

[0148] Step 2.4.3: splice the forward feature and the denoised backward feature and input it into a single-layer linear network e v = W fuse x v + b fuse , where is a learnable parameter, and finally output the knowledge-enhanced embeddings e v of all nodes E = {e v | v ∈ V}.

[0149] Step 3: Based on the open-source large language model LLaMA-3-8B, the LoRA-multi-task joint fine-tuning technology is used to jointly model and optimize the multimodal feature fusion representation H and the knowledge-enhanced embedding set E. Through the multi-task learning framework, domain-specific model parameters adapted to the traffic domain are obtained and the domain-adapted traffic large model M is derived. The specific method is as follows:

[0150] Step 3.1: Based on fusion feature representation And the knowledge-enhanced embedding representation set E = {e v |v∈V}, for each feature vector h in the sequence l , according to its spatiotemporal position encoding, calculate the embedding e of each knowledge v The cosine similarity s l,v , retrieve the node v with the largest similarity in the embedding library * =argmax v s l,v And embed the corresponding knowledge As the fusion source, h l For query, The sequence H′={h1′,...,h L ′} is used as the multimodal fusion representation after knowledge enhancement.

[0151] Step 3.2: Construct the LoRA enhancement module and set the main weight set W of the LLaMA-3-8B pre-trained model base ={W Q ,W K ,W V ,W FFN ,…} is loaded into the fine-tuning framework, where each weight matrix d is the hidden layer dimension of the model. To achieve efficient parameter fine-tuning, the query projection matrix W Q Introducing a pair of trainable low-rank matrices r< <d,并定义增量矩阵 Similarly, for the key projection matrix W K , value projection matrix W V and the feedforward network weight W FFN Construct low-rank increments ΔW respectively K ,ΔW V ,ΔW FFN , so that the overall parameter set after fine-tuning is:

[0152]

[0153] Keep W during fine-tuning base The original weights in are frozen, and only the incremental matrix group {A *,B *}Perform gradient update.

[0154] Step 3.3: Construct a multi-task learning framework. In the downstream task set T = {T1, T2, T3} in the traffic field, define the traffic event recognition task T1, the classification head and cross entropy loss L1; define the travel path planning task T2, generate the head and sequence negative log-likelihood loss L2; ​​define traffic scene question answering task T3, generate head And the negative log-likelihood loss of the conditional language model; the loss of each task is calculated according to the preset weight λ i Combined into a unified multi-task objective function

[0155] Step 3.4: In the GPU training environment, perform multi-task joint optimization on the lightweight fine-tuning parameters and load the LoRA-enhanced model parameters in the distributed training framework on a single GPU with at least 24GB of video memory. Synchronously load the knowledge-enhanced fusion representation sequence H′={h1′,...,h L ′} and its corresponding supervision labels, divided into several training subsets according to the batch size 14; for each training subset, according to Perform forward reasoning on H′ to obtain the output of each task, calculate the event classification loss L1, path generation loss L2 and question answering loss L3 respectively, and aggregate them into the joint target L according to the preset weights {0.4, 0.3, 0.3)} joint , with a learning rate of η = 5 × 10 -5 The AdamW optimization algorithm performs gradient backpropagation on ΔW and evaluates the validation set every 1000 steps. If the indicator does not improve for three consecutive times, the learning rate is multiplied by 0.5. Early stopping is triggered when the validation loss stabilizes or reaches the maximum number of epochs of 10, such as Figure 4 The figure shows the change of the loss function of the model when it is trained for 20 rounds, 100 rounds and 150 rounds respectively; after the training, the final LoRA incremental parameter ΔW and the basic model weight W are output. base Save them together.

[0156] Step 3.5: Compare the fine-tuned LoRA incremental parameters ΔW with the base model weights W base Fusion, generate complete domain-specific model parameters W final , using ONNX's dynamic quantization tool to quantize W final Perform INT8 quantization on the entire model, use TensorRT Builder to convert the quantized model to TensorRT Engine, optimize GPU inference performance, generate .onnx and .trt files, and store them as a model file M that can be loaded online.

[0157] Step 4: Obtain the traffic data to be processed, and generate a multi-modal spatio-temporal feature sequence U using a spatio-temporal clustering algorithm real , initialize three task parallel inference channels of event classifier, path planning generator and scene question answer generator, and process dynamic data stream using time-effect perception attention mechanism to obtain dynamic time-effect constrained multi-modal fusion features H live , process multi-modal fusion features H live , generate knowledge enhanced sequence H' based on knowledge enhanced embedding representation set E live , and output traffic event identification, path planning and scene question answer three task results in parallel, when the results conflict, arbitrate based on knowledge graph calculation path reliability score, and construct decision evidence chain C to generate traffic decision output O. The specific method is:

[0158] Step 4.1: The input interface continuously receives multi-modal data P={p t |p t =(Text t ,Img t ,Traj t ),t∈[1,T];Reuse the preprocessing process of steps 1.3-1.6 to generate unified input units for each p t , obtain buffer queue Z buffer ={z1,z2,…,z T}, and apply the spatio-temporal clustering algorithm of step 1.7 to generate multi-modal spatio-temporal feature sequence U real ={u1,u2,…,u T}.

[0159] Step 4.2: Deploy the domain model M derived in step 3.5 and the knowledge graph embedding E generated in step 2, and initialize three task parallel inference channels of event classifier, path planning generator and scene question answer generator.

[0160] Step 4.3: For multi-modal spatio-temporal feature sequence U real , introduce time-effect perception attention mechanism, and construct dynamic decay mask matrix B live ∈R T×T :

[0161]

[0162] Wherein when |ts i -ts j |>τ live , the mask value is -∞, otherwise it is 0, τ live is a preset time-effect threshold, and the mask is injected into the attention calculation of each layer of the model M. Transformer to ensure that the weight of outdated data is automatically decayed, and the calculation formula of the time-effect perception attention head is defined as follows:

[0163]

[0164] The time-aware attention head explicitly constrains the time range of attention calculation through a dynamic attenuation mask matrix, which is different from the sliding window clustering based on a fixed space-time threshold in step 1.8 and the standard Transformer encoding in step 1.9 during the training phase, thereby achieving efficient time control in real-time inference. The features output by the time-aware attention head are spliced to obtain a dynamic time-constrained multi-modal fusion feature H live = {h live,1 ,…,u live,L′}, which has the same semantic hierarchy as the multi-modal fusion representation H in the training phase.

[0165] Step 4.4: Perform knowledge-enhanced decision-making on each feature unit h live,i Retrieving the most similar embedding in the knowledge graph:

[0166]

[0167] Using a cross-attention mechanism to implement feature deep fusion between each feature unit h live,i and the most similar embedding:

[0168]

[0169] Generating a knowledge-enhanced sequence H′ live = {h′ 1ive,1 ,…,u′ live,L′} to the three-task channel.

[0170] Step 4.5: Receiving parallel output pieces of labels from the three-task channel Path sequence Question and answer When a conflict between tasks is detected, trigger the path reliability arbitration rule based on the knowledge graph G: when the path planning result contains an accident node with event classification labels, retrieve the relationship subgraph G conflict of the conflict node in the knowledge graph G; calculate the path reliability score:

[0171]

[0172] where, is the edge sequence output by path planning, w vu is the edge weight, Deg(v,u) is the real-time congestion degree of edge (v,u), β is the scaling factor, and the recommended value is 2.5. If ρ route < 0.6, generate a candidate path set Y conflict that avoids G candidate ; for each candidate path yk ∈Y candidate Recalculate p k ; output Otherwise keep the original path

[0173] The threshold selection basis is shown in the following table 1:

[0174] Table 1 Threshold selection basis

[0175] Reliability level ρ range Action strategy Congestion level High reliability [0.8,1] Direct adoption Overall free flow Medium reliability [0.6,0.8) Monitoring and warning Local mild congestion Low reliability [0.4,0.6) Trigger replacement (τ = 0.6) Key road segment severe congestion Unreliable [0,0.4) Emergency escape + manual intervention Road network paralysis

[0176] Step 4.6: Construct a structured decision evidence chain C = {(v, r, u, score) | v, u ∈ V conflict , r ∈ R}, where v, u are graph node identifiers, r is a relationship type identifier, and score is a numerical confidence output by the model, V conflict is a subset of graph nodes involved in the conflict resolution process, and based on the nodes, relationships and confidence data in the decision evidence chain C, a natural language explanation of the traffic decision output O is generated by a pre-defined template mapping function f template : C → Σ * . For example, when C = {(A, accident, null, 0.92), (B, congestion, C, 0.73)}, the decision is generated: Recommend alternative path D→E (detour distance +12%, time consumption -18%) due to detection of node A accident (confidence 92%) and path B→C congestion index 73%.

[0177] The application also discloses a traffic large model construction and decision device based on multi-modal and bidirectional graph reasoning, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the computer program realizes the traffic large model construction and decision method based on multi-modal bidirectional graph reasoning when loaded into the processor.

[0178] The above embodiments are only for illustrating the technical concept and characteristics of the application, and the purpose is to enable those skilled in the art to understand the content of the application and implement it, and cannot limit the protection scope of the application. Any equivalent transformation or modification made according to the spirit and essence of the application should be covered within the protection scope of the application.

Claims

1. A traffic model construction and decision-making method based on multimodal bidirectional graph reasoning, characterized by: The steps include: Step 1: Based on traffic events, a multimodal dataset of text, images, and trajectories is constructed. The fused feature representation H is generated through spatiotemporal clustering and cross-modal Transformer encoding. Step 2: Based on the fused feature representation H from step 1, bidirectional graph reasoning is performed in conjunction with the traffic knowledge graph. An embedding representation is generated through a forward graph neural network. The decision solution generated by the language model is then reverse-mapped to the graph based on the fused feature representation H and the embedding representation to verify consistency, and the knowledge-enhanced embedding representation set E is output. Step 3: Using LoRA multi-task joint fine-tuning technology, the fused feature representation H and the knowledge-enhanced embedding representation set E are jointly modeled and optimized. Through the multi-task learning framework, domain-specific model parameters adapted to the traffic domain are obtained and the domain-adapted traffic model M is derived. Step 4: Obtain the traffic data to be processed and use the spatiotemporal clustering algorithm to generate a multimodal spatiotemporal feature sequence U real Initialize the parallel reasoning channel of the three tasks of event classifier, path planning generator and scene question answering generator, and use the time-aware attention mechanism to process the dynamic data stream to obtain the multimodal fusion feature H with dynamic time constraints. live , for the multimodal fusion feature H live Generate knowledge enhancement sequence H′ based on knowledge enhancement decision making based on knowledge enhancement embedding representation set E live , and output the results of three tasks: traffic event recognition, path planning and scenario question answering in parallel.

2. The method for constructing a large traffic model and making decisions based on multimodal bidirectional graph reasoning according to claim 1 is characterized in that: The specific steps in step 1 are as follows: Step 1.1: For each traffic event, construct a multimodal dataset D consisting of three modalities: text, image, and trajectory. The text includes the traffic event description, sample unique identifier, text content, and corresponding timestamp; the image includes the RGB color image pixel matrix and the latitude and longitude coordinates of the shooting location; the trajectory includes the set of trajectory points arranged in time sequence for the same event; Step 1.2: Preprocess the data of each modality, extract features, and construct spatiotemporal position encoding to form a unified multimodal input unit set Z; The feature extraction calls the BERT-base pre-trained model, the ViT-base model, and the road node embedding model Embed based on graph neural network training. geo , obtain semantic feature representation text feature embedding, image feature embedding, trajectory feature embedding; the spatiotemporal position encoding includes spatial position encoding P geo and temporal position code P time , embedding text features, image features, trajectory features and spatial position encoding vector P geo , time position encoding vector P time A multimodal input unit is formed by splicing according to the channel dimension, and multiple multimodal input units constitute a multimodal input unit set Z; Step 1.3: Based on the multimodal input units obtained in step 1.2, they are aggregated into spatiotemporal clusters using preset time and space thresholds. These clusters are then expanded according to the latest time in the cluster and the order within the cluster to generate a unified multimodal spatiotemporal feature sequence U. Based on the multimodal spatiotemporal feature sequence U, multi-head self-attention fusion is performed in parallel within the sliding window cluster to obtain a locally fused multimodal spatiotemporal feature sequence U′. A multi-layer Transformer encoder is used to perform cross-modal modeling on U′ to obtain a fused feature representation H that integrates semantic, spatiotemporal, and modal features. Step 1.4: Design a cross-modal temporal alignment loss function L based on the fused feature representation H align , to minimize the loss in self-supervised training.

3. The method for constructing a large traffic model and making decisions based on multimodal bidirectional graph reasoning according to claim 2 is characterized in that: In step 1.3, the multimodal input unit vectors obtained in step 1.2 are aggregated into spatiotemporal clusters based on preset time and space thresholds, and expanded according to the latest time in the cluster and the order within the cluster to generate a unified multimodal spatiotemporal feature sequence U, as follows: For the multimodal input unit vector z i Based on the collection timestamp tsi and geographic coordinates gps i Setting the time window width Δt and the space radius Δd will satisfy |ts k -ts i |≤Δt∧dist(gps k ,gps i )≤Δd condition for all units z k Construct spatiotemporal neighborhood clusters: C m ={z k ∈Z∣|ts k -ts i |≤Δt,dist(gps k ,gps i )≤Δd},m∈[1,M] Among them, M is the total number of spatiotemporal neighborhood clusters, ts k 、ts j Indicates two sampling timestamps, gps k ,gps j Represents the geographic latitude and longitude coordinates corresponding to the two sampling timestamps, with each cluster C m The maximum timestamp of each cluster C is used as the cluster sorting basis. After all clusters are sorted in ascending order, m All vectors z in k Flatten the vectors in the original time order to generate a multimodal spatiotemporal feature sequence U={u1,u2,...,u N }, |U|=N is equal to the total number of original input units, that is, the multimodal input unit vector z in the modal input unit Z i the total number of; The cross-modal alignment loss function L align Specifically: For any two different encoding feature vectors (h i ,h j ), i≠j, if |ts i -ts j |≤Δt and dist(gps i ,gps j )≤Δd, then (h i ,h j ) is considered as a positive pair; otherwise it is considered as a negative pair, and the cross-modal alignment loss is defined as: Among them, 1(·) is the indicator function, and only the pairs (i, j) that meet the spatiotemporal neighborhood conditions are counted as losses, ts i 、ts j and GPS i 、GPS j Corresponding to h i With h j The original acquisition timestamp and geographic coordinates, Δt represents the time window threshold, Δd represents the spatial distance threshold, dist(gps i ,gps j ) is the distance function between the geographic coordinates of two points, Proj: is a shared linear map, is the feature vector after projection alignment, is the square of the Euclidean norm, which is used to measure the square distance between eigenvectors.

4. The method for constructing a large-scale traffic model and making decisions based on multimodal bidirectional graph reasoning according to claim 1 is characterized in that: The specific process of outputting the knowledge enhanced embedding representation E in step 2 is as follows: Step 2.1: Construct a multi-granularity traffic knowledge graph G = (V, E, R), where the node set V contains four entity nodes: roads, intersections, regions, and cities. The edge set E represents the connection between entities. The relationship type set R includes but is not limited to "connection", "adjacency", "attribution", "coverage", "restriction", "accident", and "construction". Step 2.2: Based on the spatial position encoding of the multimodal feature fusion representation H and its corresponding index, align each fusion feature representation in the multimodal feature fusion representation H to a node set that matches the multi-granularity traffic knowledge graph and the spatial position encoding. Use a graph neural network with a multi-scale gated propagation mechanism and a relational attention aggregation strategy to perform forward inference on the graph and generate a forward embedding representation E. f ; Step 2.3: Use the large language model LLM to compare H and E f Processing generates a traffic decision plan, and maps the traffic decision plan back to the knowledge graph G, triggering reverse graph reasoning to generate a reverse embedding representation E b ; Step 2.4: Fusion of backward embedding and forward embedding generates the final knowledge-enhanced embedding E, E = {e v ∣v∈V}.

5. The method for constructing a large-scale traffic model and making decisions based on multimodal bidirectional graph reasoning according to claim 4 is characterized in that: The graph neural network with multi-scale gated propagation mechanism and relational attention aggregation strategy in step 2.2 is a heterogeneous graph neural network for the graph spectrum, specifically as follows: Step 2.2.1: Traverse all nodes v in the knowledge graph G = (V, E, R); Step 2.2.2: Representation based on multimodal feature fusion and its corresponding index spatial position code P geo , each Aligned to the traffic knowledge graph G and P geo The set of matching nodes And the initial embedding of these nodes Set to the corresponding T represents the number of layers of the Transformer encoder; Step 2.2.3: Define a learnable gating function in is the embedding vector of node v in round t-1, P geo (v) is the spatial position encoding of node v, σ(·) is the element-wise Sigmoid function, b g ∈R d is a learnable parameter, and the output dimension is the same as the node embedding dimension; Control node v's neighbor set where d hop (v,u) represents the shortest number of hops between nodes v and u in the graph; Step 2.2.4: For each edge (u→v)∈E in the graph, read the edge type r uv ∈R, and maps it into a learnable relational embedding vector e(r uv )∈R d , the previous round of embedding and relation embedding e(r uv ) Mapping after splicing: Among them, W r ∈R 1×3d is a learnable linear mapping matrix, the candidate neighbor set of node v Perform softmax normalization and calculate the relation attention weights: Step 2.2.5: For each node v, initialize the 0th round embedding For node features, for the tth iteration, t = 1, ..., T, perform the following aggregation and update, weighted aggregation of neighbor information: Among them, W h ∈R d×d 、b h ∈R d For learnable parameters, gated fusion updates: Among them, ⊙ is the element-wise multiplication, which ensures that the node is dynamically balanced between new and old information; Step 2.2.6: After T rounds of iterations, collect the final embeddings of each node v∈V in the traffic knowledge graph G=(V,E,R) and organize the final embedding vectors of all nodes into a forward embedding set 6. The method for constructing a large traffic model and making decisions based on multimodal bidirectional graph reasoning according to claim 4 is characterized in that: The specific operations of step 2.3 are: Step 2.3.1: Let multimodal feature fusion representation and forward graph embedding As the input of the large language model LLM, the pre-designed Prompt instruction prompts LLM to combine H and E f Generate a series of preliminary transportation decision options Denoted as: ∈ natural language traffic decision instruction, t′∈[1,T′],T′∈N + , N + represents a set of positive integers, l∈[1,L], and T represents the number of layers of the Transformer encoder; Step 2.3.2: For Each traffic decision instruction in Using natural language parsing or rule template technology, Parsed into a structured triple mapping set Among them, each (v, r, u) represents that node v acts on node u through relationship r, based on M t′ , extract the corresponding subgraph G in the multi-granularity traffic knowledge graph G = (V, E, R) t′ =(V t′ ,E t′ ),in: V t′ ={v∣(v,r,u)∈M t′ }∪{u∣(v,r,u)∈M t′ }, E t′ ={(v,u)∣(v,r,u)∈M t′ } For each subgraph node v∈V t′ , whose initial embedding is taken from the forward embedding If v is in E f If it does not exist, it will be generated using static property mapping. Step 2.3.3: For each G t′ =(V t′ ,E t′ ), execute T on the subgraph b Round heterogeneous graph neural network back propagation, get each node v∈V t′ Backward embedding Only for the subset V t′ and E t′ Iterate: for t=1,2,…,T b , each v∈V t′ , first perform gate calculation according to the following formula: Filter candidate neighbors: Among them, dist G (u,v) is the shortest hop count between u and v in graph G, is the initial propagation radius corresponding to node v; for each (u,v)∈E r , read its relation type r, and embed the previous round Concatenate with the relation embedding r and project: Use Softmax to normalize the relationship list: Where a∈R d is a learnable vector, after T b After rounds of iteration, we get subgraph G t′ The backward embedding of each node v is: Constructing a reverse embedding set Each time subgraph V t′ Generate a set 7. The method for constructing a large-scale traffic model and making decisions based on multimodal bidirectional graph reasoning according to claim 1 is characterized in that: The specific process in step 3 is: Step 3.1: Based on fusion feature representation And the knowledge-enhanced embedding representation set E = {e v |v∈V}, for each feature vector h in the sequence l , according to its spatiotemporal position encoding, calculate the embedding e of each knowledge v The cosine similarity s l,v , retrieve the node v with the largest similarity in the embedding library * =argmax v s l,v And embed the corresponding knowledge As the fusion source, h l For query, The sequence H′={h1′,…,h L ′} as the multimodal fusion representation after knowledge enhancement; Step 3.2: Construct the LoRA enhancement module and load the set of main body weights \(W\) of the LLaMA-3-8B pre-trained model base =\(\{W\) Q ,W K ,W V ,W FFN ,...\} into the fine-tuning framework, where each weight matrix d is the dimension of the model's hidden layer. For the query projection matrix \(W\) Q , introduce a pair of trainable low-rank matrices r << d, and define the incremental matrix . Similarly, for the key projection matrix \(W\) K , the value projection matrix \(W\) V and the feed-forward network weight \(W\) FFN , construct low-rank increments \(\Delta W\) K , \(\Delta W\) V , \(\Delta W\) FFN respectively. After fine-tuning, the overall parameter set is as follows: Keep W during fine-tuning base The original weights in are frozen, and only the incremental matrix group {A * ,B * } Perform gradient update; Step 3.3: Construct a multi-task learning framework. In the downstream task set T = {T1, T2, T3} in the traffic field, define the traffic event recognition task T1 and the classification head f1: And cross entropy loss L1; define the travel path planning task T2 and generate head f2: And the sequence negative log-likelihood loss L2; ​​define the traffic scene question answering task T3 and generate the head f3: And the negative log-likelihood loss of the conditional language model; the loss of each task is calculated according to the preset weight λ i Combined into a unified multi-task objective function Step 3.4: In the GPU training environment, perform multi-task joint optimization on the lightweight fine-tuning parameters and load the LoRA-enhanced model parameters Synchronously load the knowledge-enhanced fusion representation sequence H′={h1′,...,h L ′} and its corresponding supervision labels, divided into several training subsets according to the batch size 14; for each training subset, according to Perform forward reasoning on H′ to obtain the output of each task. After the training is completed, the final LoRA incremental parameter ΔW and the basic model weight W are output. base Save together; Step 3.5: Compare the fine-tuned LoRA incremental parameters ΔW with the base model weights W base Fusion, generate complete domain-specific model parameters W final , and get the final traffic model M.

8. The method for constructing a large-scale traffic model and making decisions based on multimodal bidirectional graph reasoning according to claim 1 is characterized in that: In step 4, the time-sensitive attention mechanism is used to process the dynamic data stream to obtain the dynamic time-constrained multimodal fusion feature H live , as follows: For the multimodal spatiotemporal feature sequence U real , introduce the time-sensitive attention mechanism and construct the dynamic attenuation mask matrix B live ∈R T×T : Among them, when |ts i -ts j |>τ live The mask value is -∞ when ts is set, otherwise it is 0. i 、ts j Represents two sampling timestamps, τ live To preset the timeliness threshold, the mask is injected into the attention calculation of each layer of the Transformer of the traffic model M. The calculation formula of the timeliness-aware attention head is defined as follows: The features output by the time-sensitive attention head are concatenated to obtain the dynamic time-constrained multimodal fusion feature H live ={h live , 1, ...,u live,L′ }, whose semantic level is consistent with the multimodal fusion representation H in the training stage.

9. The method for constructing a large-scale traffic model and making decisions based on multimodal bidirectional graph reasoning according to claim 1 is characterized in that: In step 4, the multimodal fusion feature H live Generate knowledge enhancement sequence H′ based on knowledge enhancement decision making based on knowledge enhancement embedding representation set E live , as follows: Perform knowledge enhancement decision-making for each feature unit h in the sequence live,i The knowledge-enhanced embedding of the retrieved knowledge graph represents the embedding with the highest similarity in the set E: Use the cross-attention mechanism to each feature unit h live,i And the embedding with the highest similarity achieves deep feature fusion: Generate knowledge-enhanced sequence H′ live ={h′ live,1 ,...,u′ live,L′ } to the three-task channel.

10. A traffic model construction and decision-making device based on multimodal and bidirectional graph reasoning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the method for constructing and making a large traffic model based on multimodal bidirectional graph reasoning according to any one of claims 1 to 9 is implemented.

Images