Traffic multimodal large model construction method based on graph embedding and multimodal alignment

Abstract

The application discloses a traffic multi-modal large model construction method based on graph embedding and multi-modal alignment, and belongs to the technical field of intelligent traffic. In order to solve the technical problems that a general large model is difficult to adapt to traffic graph structure data, a domain mechanism is disjointed from general intelligence, and multi-source heterogeneous traffic information cannot be uniformly fused, the application obtains corresponding graph embedding by respectively processing a traffic static road network, a rule knowledge graph and a dynamic space-time graph through a graph neural network, projects the graph embedding to a large model original text semantic space through a multi-modal adapter, completes semantic alignment by using a contrast loss, and finally obtains a target model through low-rank adaptive fine-tuning. The application can be widely applied to traffic planning, traffic flow prediction, intelligent decision and the like, and significantly improves model reasoning precision, generalization and interpretability.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a method for constructing a large multimodal transportation model based on graph embedding and multimodal alignment. Background Technology

[0002] In recent years, large-scale language models and vision-language multimodal models have rapidly developed in the field of general artificial intelligence, achieving widespread application in various scenarios thanks to their powerful semantic understanding and generation capabilities. As a highly complex structured system, the transportation system's core data includes three types of graph structure data: static road network topology maps, traffic rule knowledge graphs, and dynamic traffic spatiotemporal graphs. These contain rich topological, spatiotemporal, and semantic association information, necessitating deep integration of general-purpose large-scale models with the transportation domain to achieve intelligent upgrades for complex tasks such as traffic planning, prediction, and decision-making. Among existing technologies, graph neural networks can achieve feature learning of traffic graph structures. Some solutions attempt to combine knowledge graphs with spatiotemporal graph convolutions for traffic flow prediction, while other technologies achieve visual and textual multimodal feature alignment through language perception, providing a foundational approach for constructing large-scale transportation models.

[0003] However, existing technologies have significant drawbacks: First, general-purpose large models can only process serialized and gridded data, and cannot effectively model traffic graph structure data, making it difficult to integrate prior knowledge such as traffic physical constraints and rule mechanisms. Second, there is a modal gap between graph embedding in graph neural networks and general-purpose large models, which can only complete feature learning for specific tasks and cannot connect with the general reasoning capabilities of large models, resulting in insufficient generalization in scenarios with few or no samples. Third, existing multimodal alignment methods are only for text and visual modalities and are not adapted to proprietary graph modalities such as traffic network topology and spatiotemporal states. Static road networks, traffic rules, dynamic states, and other multi-source data are fragmented and lack a unified representation and collaborative modeling mechanism, making it difficult for models to understand the dynamic behavior of traffic systems from a mechanistic perspective and support intelligent traffic decision-making across time, space, and scenarios. Summary of the Invention

[0004] To address the technical problems of general multimodal large models being difficult to adapt to traffic map structure data, the disconnect between traffic domain mechanisms and general intelligence, and the inability to uniformly represent and fuse multi-source heterogeneous traffic information, this invention proposes a method for constructing a traffic multimodal large model based on graph embedding and multimodal alignment, including:

[0005] S1. Model the traffic network as a static traffic network map, and learn the node embedding vector of the static traffic network map through graph attention network;

[0006] S2. Based on the node embedding vector of the static traffic network map, the traffic rule knowledge graph is associated and mapped to the static traffic network map to obtain the graph embedding vector of the rule-enhanced road network graph.

[0007] S3. Model the traffic status data into a traffic dynamic spatiotemporal map, and use graph neural networks and time series models to extract the spatiotemporal feature embedding vector of the traffic dynamic spatiotemporal map;

[0008] S4. Using the graph embedding vector of the rule-enhanced road network map and the spatiotemporal feature embedding vector of the traffic dynamic spatiotemporal map as input, the graph embedding and text embedding are projected to the native text semantic space of the general multimodal large model through a multimodal adapter, and the graph embedding and text embedding are aligned using a contrastive loss function to obtain the multimodal alignment result.

[0009] S5. Based on the multimodal alignment results, the general multimodal large model is fine-tuned using a low-rank adaptive method to obtain a traffic multimodal large model that integrates traffic domain mechanisms.

[0010] Furthermore, the node embedding update formula for the graph attention network described in S1 is:

[0011]

[0012]

[0013] in: for The set of neighboring nodes; For attention weights, For the first The weight matrix of the layer, For activation function, This represents vector concatenation, after... After layer iteration, nodes The final embedding vector is .

[0014] Furthermore, the association mapping described in S2 specifically involves: using geographic information coordinates and semantic description parsing to associate the rule nodes of the traffic rule knowledge graph with the corresponding nodes or road segments of the static traffic road network map, adding a set of rule attributes to the road network nodes, and forming a rule-enhanced road network graph.

[0015] Furthermore, the fusion formula for the spatiotemporal feature embedding vectors described in S3 is as follows:

[0016]

[0017]

[0018] in, Let be the spatial feature matrix of node i at time t, and GNN be the graph neural network for feature extraction. Let be the node state matrix of the traffic dynamic spatiotemporal graph at time t, and let A be the adjacency matrix of the traffic dynamic spatiotemporal graph. Let be the spatiotemporal feature embedding vector of node i at time t, Transformer be the temporal feature extraction network, and T be the time step.

[0019] Furthermore, the projection formula for the multimodal adapter described in S4 is:

[0020]

[0021] in, This is the embedding vector of the projected graph. The embedding vector is the original graph. , These are adapter parameters.

[0022] Furthermore, the formula for the contrastive loss function described in S4 is:

[0023]

[0024] Where sim is the cosine similarity. For temperature coefficient, For graph embedding vectors, Let K be the positive sample text embedding vector, and K be the number of negative samples. This is achieved by minimizing... Training a multimodal adapter aligns the graph structure with the text semantics.

[0025] Furthermore, the low-rank adaptive method fine-tuning described in S5 specifically involves: freezing the original parameters of the general multimodal large model, and fine-tuning the model by optimizing the parameters of the newly added low-rank adaptation matrix to adapt to the image-text fusion input.

[0026] The beneficial effects of this invention are:

[0027] (1) Deep integration of traffic mechanism and general intelligence: This method breaks through the bottleneck that general large models cannot directly process graph structure data. It uses graph neural networks to efficiently represent road network topology, traffic rules, and spatiotemporal states, and then uses multimodal alignment technology to semantically inject them into the large model. This enables the model not only to perform language understanding and generation, but also to inherently "understand" the physical structure, spatiotemporal evolution law and rule constraint logic of the traffic network, significantly improving its cognitive depth and reasoning accuracy in traffic scenarios.

[0028] (2) Constructing a unified and scalable traffic knowledge representation and fusion framework: This method creatively associates and enhances static road network maps, dynamic spatiotemporal maps, and traffic rule knowledge graphs to form a "rule-enhanced spatiotemporal road network map". This provides a unified and structured semantic representation for multi-source heterogeneous traffic data, enabling the model to jointly learn and reason about physical connections, dynamic changes, and logical constraints in a shared semantic space, which greatly enhances the efficiency of collaborative use of information.

[0029] (3) Improve the generalization, interpretability and decision reliability of the model: By explicitly linking traffic rules to road network nodes / segments and aligning graph embedding with text semantics, the model can reason based on explicit domain knowledge (such as traffic regulations and physical constraints), making its decision-making process more interpretable. At the same time, the model, which integrates deep domain mechanisms, shows stronger generalization ability and more reliable decision quality when facing unseen road network structures, new traffic state patterns or complex and sudden scenarios. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method described in this invention. Detailed Implementation

[0031] The technical solution of the present invention will be further described below with reference to embodiments, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention. In the following embodiments, process equipment or devices not specifically specified are all conventional equipment or devices in the art. Unless specifically specified, the technical means used in the embodiments of the present invention are all conventional means well known to those skilled in the art.

[0032] Example 1, combined with Figure 1 This embodiment describes a method for constructing a large multimodal traffic model based on graph embedding and multimodal alignment, including:

[0033] S1. Model the traffic network as a static traffic network map, and learn the node embedding vector of the static traffic network map through graph attention network;

[0034] S2. Based on the node embedding vector of the static traffic network map, the traffic rule knowledge graph is associated and mapped to the static traffic network map to obtain the graph embedding vector of the rule-enhanced road network graph.

[0035] S3. Model the traffic status data into a traffic dynamic spatiotemporal map, and use graph neural networks and time series models to extract the spatiotemporal feature embedding vector of the traffic dynamic spatiotemporal map;

[0036] S4. Using the graph embedding vector of the rule-enhanced road network map and the spatiotemporal feature embedding vector of the traffic dynamic spatiotemporal map as input, the graph embedding and text embedding are projected to the native text semantic space of the general multimodal large model through a multimodal adapter, and the graph embedding and text embedding are aligned using a contrastive loss function to obtain the multimodal alignment result.

[0037] S5. Based on the multimodal alignment results, the general multimodal large model is fine-tuned using a low-rank adaptive method to obtain a traffic multimodal large model that integrates traffic domain mechanisms.

[0038] Furthermore, the node embedding update formula for the graph attention network described in S1 is:

[0039]

[0040]

[0041] in: for The set of neighboring nodes; For attention weights, For the first The weight matrix of the layer, For activation function, This represents vector concatenation, after... After layer iteration, nodes The final embedding vector is .

[0042] Specifically, the transportation network can be represented as a static graph structure. ,in For a set of nodes (intersections, stations, etc.). This represents the set of edges (connections). The goal of road network embedding is to learn the low-dimensional vector representations of nodes. This enables vectors to capture the structural and attribute features of nodes.

[0043] This step encodes entities (such as intersections and stations) and their connections (road segments) in the traffic system into a unified input for a graph neural network. It adaptively aggregates neighbor information and generates node embeddings through a graph attention mechanism. Its core purpose is to extract standardized feature representations that combine structural and semantic aspects from the original road network topology, addressing the problem that general-purpose large models cannot natively process graph-structured data. This provides a foundational structure for subsequent traffic rule knowledge injection and feature fusion.

[0044] Furthermore, the association mapping described in S2 specifically involves: using geographic information coordinates and semantic description parsing to associate the rule nodes of the traffic rule knowledge graph with the corresponding nodes or road segments of the static traffic road network map, adding a set of rule attributes to the road network nodes, and forming a rule-enhanced road network graph.

[0045] Specifically, traffic rules can be represented as a traffic rule knowledge graph through information extraction and causal analysis. In this system, nodes represent specific rule entries, and edges represent logical, causal, or conflicting relationships between rules. Traffic network maps and traffic rule maps can be parsed using semantic descriptions such as geographic coordinates and signs to map traffic rules to the road network. Traffic rules displayed in the system are related to A specific constrained node forms an enhanced road network map. Each road network node / segment is associated with a set of rule attributes. .

[0046] This step constructs a traffic rule knowledge graph by extracting information and performing causal analysis on traffic rules such as speed limits, traffic restrictions, traffic light control, and lane management. Based on geospatial consistency and semantic consistency, rule entries are explicitly associated with constrained road network nodes or road segments, so that each road network node carries a unique set of rule attributes. The core purpose is to deeply bind prior rules in the traffic domain with the road network topology, solve the problem that pure structural embedding lacks traffic mechanism constraints, and allow graph embedding to carry both road network structure information and traffic rule semantic information, thereby improving the domain relevance of the embedding vector.

[0047] Furthermore, the fusion formula for the spatiotemporal feature embedding vectors described in S3 is as follows:

[0048]

[0049]

[0050] in, Let be the spatial feature matrix of node i at time t, and GNN be the graph neural network for feature extraction. Let be the node state matrix of the traffic dynamic spatiotemporal graph at time t, and let A be the adjacency matrix of the traffic dynamic spatiotemporal graph. Let be the spatiotemporal feature embedding vector of node i at time t, Transformer be the temporal feature extraction network, and T be the time step.

[0051] Specifically, a traffic spatiotemporal diagram can be represented as a dynamic graph structure. ,in for The node state matrix at any given time (e.g., flow rate, speed). For the state dimension. The spacetime diagram is in... The node embedding vectors at each time step need to capture both spatial and temporal dependencies. Combining a spatiotemporal graph neural network model, a GNN is used to extract spatial features, and then the spatial features from multiple time steps are input into a Transformer to capture temporal dependencies.

[0052] This step models real-time traffic status data such as traffic flow, driving speed, and road segment occupancy into a dynamic spatiotemporal graph based on time series. First, spatial propagation dependency features between road segments are extracted using a graph neural network. Then, the spatial features from multiple time steps are input into the Transformer model to capture long-term evolution dependencies. Finally, the spatiotemporal features are jointly fused. The core purpose is to simultaneously model the spatial topological relationships and temporal dynamic changes of traffic data, adapt to the characteristics of real-time traffic status evolution, and provide a feature carrier of dynamic traffic status for multimodal alignment.

[0053] Furthermore, the projection formula for the multimodal adapter described in S4 is:

[0054]

[0055] in, This is the embedding vector of the projected graph. The embedding vector is the original graph. , These are adapter parameters.

[0056] Specifically, the multimodal adapter maps heterogeneous traffic map embedding vectors to the native text semantic space of the general multimodal large model, ensuring that the dimensions of the graph embedding and the text embedding are consistent and the semantic space is unified. The core purpose is to build a bridge between the traffic map modality and the large model text modality, eliminate the dimensional gap and semantic barrier between the two types of modalities, and provide a foundation for subsequent semantic alignment.

[0057] Furthermore, the formula for the contrastive loss function described in S4 is:

[0058]

[0059] Where sim is the cosine similarity. For temperature coefficient, For graph embedding vectors, Let K be the positive sample text embedding vector, and K be the number of negative samples. This is achieved by minimizing... Training a multimodal adapter aligns the graph structure with the text semantics.

[0060] Specifically, this step employs a contrastive learning strategy to maximize the similarity between graph embeddings and text embeddings of the same traffic entity, while minimizing the similarity between embeddings of different entities. This forces graph structure information and text semantics to achieve accurate matching within a unified space. The core objective is to fundamentally address the problem of general-purpose large models failing to understand the semantics of traffic graph structures, enabling large models to accurately identify the traffic entities, rules, and state information corresponding to graph embeddings, eliminating the reasoning illusion of large models, and improving the accuracy of reasoning in traffic scenarios.

[0061] Furthermore, the low-rank adaptive method fine-tuning described in S5 specifically involves: freezing the original parameters of the general multimodal large model, and fine-tuning the model by optimizing the parameters of the newly added low-rank adaptation matrix to adapt to the image-text fusion input.

[0062] Specifically, this step adopts a low-rank adaptive LoRA lightweight fine-tuning strategy. The core purpose is to enable the model to quickly adapt to the multimodal input of graphs and text in the transportation domain without destroying the original general reasoning and generalization capabilities of the general large model, and to efficiently complete the adaptation and learning of transportation domain knowledge.

Claims

1. A method for constructing a traffic multi-modal large model based on graph embedding and multi-modal alignment, characterized in that, include: S1. Model the traffic network as a static traffic network map, and learn the node embedding vector of the static traffic network map through graph attention network; S2. Based on the node embedding vector of the static traffic network map, the traffic rule knowledge graph is associated and mapped to the static traffic network map to obtain the graph embedding vector of the rule-enhanced road network graph. S3. Model the traffic status data into a traffic dynamic spatiotemporal map, and use graph neural networks and time series models to extract the spatiotemporal feature embedding vector of the traffic dynamic spatiotemporal map; S4. Using the graph embedding vector of the rule-enhanced road network map and the spatiotemporal feature embedding vector of the traffic dynamic spatiotemporal map as input, the graph embedding and text embedding are projected to the native text semantic space of the general multimodal large model through a multimodal adapter, and the graph embedding and text embedding are aligned using a contrastive loss function to obtain the multimodal alignment result. S5. Based on the multimodal alignment results, the general multimodal large model is fine-tuned using a low-rank adaptive method to obtain a traffic multimodal large model that integrates traffic domain mechanisms.

2. The traffic multi-modal large model construction method based on graph embedding and multi-modal alignment according to claim 1, characterized in that, The node embedding update formula for the graph attention network described in S1 is: in: for The set of neighboring nodes; For attention weights, For the first The weight matrix of the layer, For activation function, This represents vector concatenation, after... After layer iteration, nodes The final embedding vector is .

3. The traffic multi-modal large model construction method based on graph embedding and multi-modal alignment according to claim 1, characterized in that, The association mapping described in S2 specifically involves: using geographic information coordinates and semantic description parsing to associate the rule nodes of the traffic rule knowledge graph with the corresponding nodes or road segments of the static traffic road network map, adding a set of rule attributes to the road network nodes, and forming a rule-enhanced road network graph.

4. The method for constructing a large multimodal traffic model based on graph embedding and multimodal alignment according to claim 1, characterized in that, The fusion formula for the spatiotemporal feature embedding vectors described in S3 is as follows: in, Let be the spatial feature matrix of node i at time t, and GNN be the graph neural network for feature extraction. Let be the node state matrix of the traffic dynamic spatiotemporal graph at time t, and let A be the adjacency matrix of the traffic dynamic spatiotemporal graph. Let be the spatiotemporal feature embedding vector of node i at time t, Transformer be the temporal feature extraction network, and T be the time step.

5. The method for constructing a large multimodal traffic model based on graph embedding and multimodal alignment according to claim 1, characterized in that, The projection formula for the multimodal adapter described in S4 is: in, This is the embedding vector of the projected graph. The embedding vector of the original graph. , These are adapter parameters.

6. The method for constructing a large multimodal traffic model based on graph embedding and multimodal alignment according to claim 1, characterized in that, The formula for the contrastive loss function described in S4 is: Where sim is the cosine similarity. For temperature coefficient, For graph embedding vectors, Let K be the positive sample text embedding vector, and K be the number of negative samples. This is achieved by minimizing... Training a multimodal adapter aligns the graph structure with the text semantics.

7. The method for constructing a large multimodal traffic model based on graph embedding and multimodal alignment according to claim 1, characterized in that, The low-rank adaptive method fine-tuning described in S5 specifically involves freezing the original parameters of the general multimodal large model and fine-tuning the model by optimizing the parameters of the newly added low-rank adaptation matrix to adapt to the image-text fusion input.

Images