A text-enhanced multi-modal road network representation learning method

Abstract

The application discloses a kind of text enhanced multi-modal road network representation learning method, belong to city computing field.The method includes: first, extract target area road vector data, road section static attribute and the panorama street view image of each road section center point;Multi-modal large language model is generated natural language description by street view image, combined with static attribute and prompt word is output structured text description by first large language model, further obtain the directed road network graph containing structured text description;Topology enhanced semantic representation is output based on structured text description by graph neural network, further by word element random masking, the graph neural network and topology enhanced semantic representation of training are completed;Second large language model can output specified downstream task result according to topology enhanced semantic representation.The application effectively solves the problems that the existing road network representation method is difficult to model the correlation of modes, the reasoning ability of data missing is weak, and the high-order semantic modeling is insufficient.

Description

Technical Field

[0001] This invention belongs to the field of urban computing, specifically relating to a text-enhanced multimodal road network representation learning method. Background Technology

[0002] Urban road network representation learning, as a core foundation of geographic information systems, urban computing, and intelligent transportation systems, aims to map complex road network topologies and heterogeneous attributes to a low-dimensional continuous vector space, providing unified feature support for downstream tasks such as urban functional area identification, traffic flow prediction, travel time estimation, and route recommendation.

[0003] Current solutions mainly fall into two categories: one is a topology modeling method based on graph neural networks, which captures spatial correlation by aggregating the features of neighboring nodes; the other is a representation method based on trajectory data synchronization, which uses the sequentiality of vehicle driving trajectories to enhance node features.

[0004] However, the aforementioned existing technologies still have the following significant drawbacks in practical applications: (1) Heterogeneity gap in representation space: Traditional methods often use simple feature splicing or shallow projection to fuse numerical and visual attributes, which makes it difficult to achieve essential alignment in deep semantic space, resulting in serious modal representation bias.

[0005] (2) Sparsity and imbalance of multidimensional sensing data: In real-world scenarios, due to limitations in collection costs and privacy protection, road network data often exhibits significant spatiotemporal polarization characteristics, making it difficult to extract effective features from edge road segments due to sparse samples. At the same time, existing methods lack the ability to semantically complete missing modalities, and simple numerical filling is prone to introducing noise, which seriously interferes with the robustness of the model's representation under incomplete data.

[0006] (3) Generalization bottleneck caused by topological coupling: Existing models are highly dependent on the global topological structure of a specific city. However, the significant differences in road network morphology and functional layout between different cities limit the model's ability to transfer knowledge in cross-city tasks.

[0007] Therefore, how to overcome the representation gap of multi-source heterogeneous data, under the constraint of sparse perceptual data, utilize the powerful logical induction and common sense completion capabilities of multimodal large language models, and combine them with cascade encoder architecture to achieve deep decoupling and collaborative fusion of cross-temporal semantics and topology, so as to construct a general road network representation model with high generalization performance, remains a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a text-enhanced multimodal road network representation learning method. This invention enables deep semantic deconstruction of heterogeneous road segment-related data using a large multimodal model, constructing a cascaded language model-graph neural network architecture, effectively bridging the representation gap between unstructured visual semantics and structured topological space. Furthermore, by fine-tuning graph instructions, the zero-shot inference potential of the large model is fully unleashed, significantly improving the cross-domain generalization ability of road representations in unseen urban areas.

[0009] The specific technical solution is as follows: S1, divide the roads within the target area into road segments according to intersections; collect multimodal static attribute information of each road segment, and further obtain panoramic street view images of the center point of each road segment; S2, the multimodal large language model generates natural language descriptions of corresponding road segments based on street view images, and inputs them into the first large language model along with static attribute information and preset prompt words to obtain structured text descriptions; S3. Construct a text-enhanced road network graph with road segments as nodes, the topological relationships between road segments as directed edges, and the structured text descriptions corresponding to the nodes as node attributes, and extract the context subgraph corresponding to each node. S4, taking each context subgraph as a unit, transforms the structured text description of each road segment node into a text sequence, and the graph neural network obtains the topology-enhanced semantic representation based on the text sequence of all nodes in the context subgraph. S5. After performing word-level random masking on the text sequence, it is concatenated with the topologically enhanced semantic representation to obtain a fused representation. Based on the fused representation and the text sequence, the first cross-entropy loss is calculated and the parameters of the graph neural network are updated until convergence, resulting in a trained graph neural network. S6, consisting of a text-enhanced road network graph composed of roads to be represented, is input into a trained graph neural network to obtain semantic representations of the roads after topological enhancement.

[0010] Furthermore, in S1, the multimodal static attribute information includes: the geometry of the road segment, road length, number of lanes, single / double lane attributes, traffic permission restrictions, and special intersection structure identifiers.

[0011] Furthermore, in step S3, the process of extracting the context subgraph corresponding to each node is specifically as follows: S301, calculate the correlation score between the current node and other nodes in the text-enhanced road network graph in turn, and further obtain the candidate nodes with the top K correlation scores, and form a subgraph node set with the node. S302 retains only directed edges whose starting and ending nodes are both located in the subgraph node set, and forms a context subgraph with the subgraph node set.

[0012] Furthermore, in S4, the process of obtaining the topologically enhanced semantic representation is specifically as follows: First, the structured text description of each road segment node is transformed into a text sequence containing start and end markers using a tokenizer; Secondly, the text sequence is input into the language model to obtain semantic representation, and then the lexical units of the starting marker are extracted from the semantic representation; Finally, the initial tokens of all road segments are stacked and then input into the graph neural network along with the adjacency matrix corresponding to the context subgraph to obtain the topology-enhanced semantic representation.

[0013] Furthermore, the adjacency matrix is ​​a matrix with road segments as rows and columns, and the topological connection relationship between road segments as element values. An element value of 1 indicates that the road segment corresponding to the row of the element and the road segment corresponding to the column of the element are connected by an intersection, and an element value of 0 indicates that there is no connected topological relationship between the road segment corresponding to the row of the element and the road segment corresponding to the column of the element.

[0014] Furthermore, in S5, the calculation formula for the fusion representation is specifically as follows: ; in, Represents a column vector consisting entirely of 1s, with the top right corner... Indicates transpose. Indicates the outer product; Indicates feature splicing; Indicates a linear layer; Represents a non-linear activation function. This represents the fusion representation of the i-th road segment. This represents the number of tokens (excluding start and end markers) in the text sequence of the i-th road segment. This represents the semantic representation of the i-th road segment after random masking. This represents the topologically enhanced semantic representation of the i-th road segment.

[0015] Furthermore, in S5, the specific formula for calculating the first cross-entropy loss is as follows: ; in, Indicates the total number of road segments. This represents the mask position indicator function. A value of 1 indicates that the i-th road segment in the masked text sequence is the first... The ith bit contains a mask; a value of 0 indicates that the ith bit in the text sequence after the ith road segment is masked is the ith bit. No mask exists for this bit; This indicates that the j-th word in the text sequence is... The text sequence after masking the i-th road segment The predicted probability of the bit. Represents cross-entropy loss, Indicates a set of road segments. This represents the i-th road segment. This represents the number of tokens in the text sequence of the i-th road segment, excluding the start and end markers.

[0016] Furthermore, it also includes: The semantic representation of the road topology to be represented, after enhancement, and the instruction template of the specified downstream task are input into the second language model to obtain the downstream task results of the road segment; the downstream tasks include road segment function identification, traffic speed prediction, and travel time estimation.

[0017] Furthermore, the second large language model includes an adapter, and the fine-tuning process of the second large language model is as follows: S601, an adapter is added to the attention mechanism layer of the original second-largest language model to obtain a second-largest language model containing the adapter; S602, the topologically enhanced semantic representation is mapped to the input embedding space of the second largest language model to obtain the mapped semantic representation; S603, input the mapped semantic representation and the instruction template of the specified downstream task into the second language model, and calculate the second cross-entropy loss based on the output results; S604, update the adapter parameters according to the second cross-entropy loss and repeat S603 until convergence, to obtain the second large language model with fine-tuning.

[0018] Furthermore, in S603, the specific formula for calculating the second cross-entropy loss is as follows: ; in, denoted as the predicted cross-entropy loss, T represents the total length of the sequence, the first L words of the sequence are the instruction template and the semantic representation after mapping, and L+1 to T are the results of the downstream task generated by the prediction of the second largest language model; This indicates the probability that the second largest language model correctly predicts a word. Represents the text history sequence, i.e., position. All previous morphemes; This represents the h-th word in the sequence.

[0019] The beneficial effects of this invention are: (1) This invention overcomes the shortcomings of independent encoding and loose association of modal features in traditional road network representation learning. By utilizing the powerful semantic alignment capability of large language models, the physical structural attributes of road nodes (such as length, number of lanes, speed limit, etc.) and unstructured visual information (such as street scene image semantics) are uniformly mapped to the natural language space. In this space, physical attributes and visual environment are transformed from isolated vectors to associated semantics through the logical chain of natural language, completing the deep deconstruction and high-order fusion of multi-source heterogeneous data.

[0020] (2) This invention fully utilizes the vast semantic prior knowledge pre-trained within the large language model, enabling implicit reasoning and automatic completion of missing attributes through latent semantic associations between modalities. Even when actual urban road network data is incomplete, this invention can still infer missing features through the semantics of the surrounding environment, improving the reliability of the road network representation model in real and complex scenarios.

[0021] (3) This invention uses a cascaded masking autoencoder framework to deeply link text feature extraction and graph feature aggregation, and adopts a masking-prediction self-supervised learning paradigm to mask key words in the text description. This requires the model not only to understand the internal context of the text, but also to obtain auxiliary information from the road network topology through cascaded paths. In this way, the generated node representations are forcibly injected with topological consistency, achieving dual alignment of semantic space and physical structure space.

[0022] (4) This invention introduces graph instruction fine-tuning technology, which uses a linear projection layer to map the road features extracted by the cascaded mask autoencoder to the embedding space of the large language model. By designing natural language instruction templates, the originally isolated graph structure data is transformed into LLM-understandable sequence input, thereby achieving accurate representation of the road network in unknown urban areas. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a model architecture diagram of the method of the present invention. Detailed Implementation

[0024] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0025] like Figure 1 and Figure 2 As shown, this embodiment provides a text-enhanced multimodal road network representation learning method, including a pre-training stage for graph neural networks and a fine-tuning stage for instructions on large language models.

[0026] Specifically: I. Extract raw road network data from the geographic information system of the target area and construct a basic road network map structure that includes a set of road segments and the topological connections between road segments.

[0027] In the preprocessing stage, the vehicle road vector data is cropped according to the preset latitude and longitude bounding boxes to construct a directed original map containing intersections and road segments.

[0028] Next, to more effectively represent the connections between road segments, this method constructs a line graph structure, converting the "edges" in the original graph into "nodes" in the line graph, thereby constructing a directed road network graph with road segments as nodes. , where the set of nodes Represents road segments, edge sets This represents the topological connections between road segments; simultaneously, it generates the corresponding adjacency matrix. In the matrix, an element with a value of 1 indicates that the corresponding road segments are connected by an intersection, while an element with a value of 0 indicates that the corresponding road segments are not connected by a topological relationship.

[0029] By treating road segments as nodes in a graph, the representation learning of complex road networks can be modeled as the representation learning of graph nodes. This method effectively transforms the topology of the road network into nodes and edges in graph theory, thereby enabling deep feature extraction using advanced machine learning techniques such as graph neural networks.

[0030] 2. Extract multimodal data of road segments from the original road network data and perform data preprocessing.

[0031] First, obtain the static attribute information of each road segment, which includes at least the geometric shape (obtained by collecting the latitude and longitude coordinates of the starting point, ending point and multiple sampling points in the segment), road length, number of lanes, single / double lane attributes, traffic permission restrictions and special intersection structure markings (such as roundabouts and overpasses).

[0032] After obtaining the attribute information, the preprocessing process includes extracting the latitude and longitude coordinates from the geometry and converting them into geohash codes to build a text spatial index suitable for understanding by large language models.

[0033] In addition, in order to further analyze the visual characteristics of road segments, the center point coordinates of all sampling points in each road segment are determined based on the collected latitude and longitude coordinates, and the corresponding 360° panoramic street view image is obtained based on these coordinates. If the number of sampling points is even, the average coordinates of the two middle sampling points are taken as the center point coordinates.

[0034] Third, a large language model is used to perform semantic parsing on multimodal data, and fine-grained structured text is generated for each road segment to construct a road text set. To achieve fine-grained semantic modeling of roads, this method introduces a multimodal large language model.

[0035] First, a pre-trained multimodal large language model (Qwen3-VL in this specific embodiment) is used to perform deep semantic parsing on the street view images associated with each road segment to obtain a natural language description of each road segment. This process aims to identify and extract the environmental features around the road, including architectural style, traffic flow density, and the distribution of roadside infrastructure (such as traffic lights, signs, and parking spaces).

[0036] Subsequently, these natural language descriptions derived from the images, the static road attributes obtained in step two (such as the number of lanes and length), and the preset prompt word templates are combined and input into the large language model. Through the logical reasoning and semantic induction capabilities of the large language model, a structured text description of the road segment with a unified format and rigorous logic is output.

[0037] This process can be formally represented as: Where P is a preset prompt word template, and Representing road nodes The street view images and static attributes, LLM is a large language model, and MLLM is a multimodal large language model.

[0038] This process transforms fragmented multimodal information into high-dimensional textual semantics, resulting in structured road segment text descriptions. ,in This is the text description corresponding to the Nth node, where N is the total number of road segments.

[0039] By using this as an attribute of the corresponding node in the road network map, a text-enhanced road network map is obtained. .

[0040] IV. Using a personalized PageRank algorithm to sample subgraphs from text-enhanced road network maps When faced with massive road data, directly feeding the entire road network map into the model for training is very costly and detrimental to training due to the large size of the original road network map. Therefore, in order to enable this system to scale to large-scale road networks, this method uses a personalized PageRank algorithm to sample subgraphs and feeds a context subgraph as a batch into the model for training.

[0041] Regarding the diagram any node on This invention utilizes the top-k personalized PageRank (PPR) algorithm to sample its context subgraph.

[0042] The specific steps are as follows: The PPR algorithm calculates the relationship between each node in graph G and the node. The relevance score is used to identify the relationship with the current node. The most relevant node. The specific formula for the relevance score is as follows: in, It is the identity matrix. For the normalized adjacency matrix, For a predefined transmission factor, For nodes The corresponding one-hot vector.

[0043] The top-k nodes by relevance score and the edges connecting them form a context subgraph. In form, let's set If it is the set of the top-k nodes based on the PPR score, then ,in , This indicates that only the two related nodes on the edge in the original road network diagram are retained. The edges in; and These are two connected nodes in the subgraph. It is the text description corresponding to the subgraph node.

[0044] Compared to other local sampling strategies, PageRank (PPR) can identify key nodes and structures that are important in a broader context and have greater general transferability, significantly improving the system's generalization ability across different cities. By employing a personalized PageRank sampling strategy, this method can construct structurally most relevant and information-rich anchor node local subgraphs, thereby enabling the model to scale to large-scale urban road networks.

[0045] Fifth, perform word-level random masking on the road segment text, extract the semantic representation of the road segment after masking based on the language model, and extract the tokens with global semantic information from the masked semantic representation. Input the tokens with global semantic information into the graph neural network to aggregate neighborhood information based on the topological relationship of the road network graph to obtain the topologically enhanced semantic representation. Traditional road network representation learning often focuses on single topological structure modeling, making it difficult to capture the complex socio-economic attributes behind road segments. Therefore, this method proposes a cascaded mask autoencoder framework of "language model + graph neural network" to achieve deep decoupling and collaborative modeling of text semantics and topological structure.

[0046] During the model training and encoding phase, the system first performs token-level random masking on the generated road node structured text.

[0047] Specifically, for any road segment's text description The word segmenter transforms it into a text sequence containing a start marker [CLS] and a stop marker [SEP]. : in, For nodes The word sequence of the text, For nodes Sub-words in the lexical sequence of the text, excluding special markers.

[0048] According to a preset masking ratio (in this specific embodiment, the masking ratio is set to 15%), word positions in the sequence are randomly selected and replaced with special [MASK] markers to generate a masked text sequence. : in, For nodes The word sequence hidden in the text.

[0049] Furthermore, a dual-path joint coding structure is employed to extract features from the masked text sequence.

[0050] First, using a pre-trained language model Semantic extraction is performed (in this specific embodiment, a BERT pre-trained language model is used). For each node... Language models for masked text sequences Encode the masked semantic representation: in, For the masked semantic representation, including Output embeddings of individual sub-words and special words [CLS] and [SEP] ( and ); The dimension of the semantic representation after masking.

[0051] Secondly, from Extracting lexical units with global semantic information As the initial representation of the nodes, it is used for subsequent structural aggregation, where Let N be the real number space, N be the number of nodes, and d be the dimension of the initial representation.

[0052] To capture the spatial context topology of road segments, this invention first uses a pre-trained language model. Extract each text sequence semantic representation Further, lexical units with global semantic information are extracted from them, and then processed through a graph convolutional neural network. In the adjacency matrix Information dissemination is carried out online. in, This is a topology-enhanced semantic representation that includes all road segments; These are lexical units that possess global semantic information, i.e., lexical units with start tags. It contains the starting marker words corresponding to all road segments.

[0053] To enable the prediction head to have structure-aware capabilities, further... Output topologically enhanced semantic representation semantic representation after masking To merge: in, It is a column vector of all 1s, with the top right corner... Indicates transpose. To represent the outer product, in order to achieve the same result as... Dimensional alignment; The features are combined to form a shape. The fusion representation; This is a linear layer used to project the fusion matrix back into the original embedding space; Represents a non-linear activation function. This represents the fused characterization.

[0054] Finally, a multilayer perceptron is used to map the fused representations to the prediction space corresponding to the vocabulary of the language model, generating word-level prediction probabilities. : in, This is a language modeling head composed of a multilayer perceptron.

[0055] VI. The semantic representation after masking and the semantic representation after topological enhancement are fused. The fused representation is mapped to the prediction space corresponding to the vocabulary of the language model using a multilayer perceptron to predict the real word at the masked position. The model is then trained and optimized based on the cross-entropy loss between the prediction result and the real word. A multilayer perceptron is used to map the fused representations onto the lexical space of a language model, predict the true words at masked positions, and calculate the cross-entropy loss between the predicted probability distribution and the true word distribution. : in, It is the number of nodes. This is a mask position indicator function; a value of 1 indicates a node. The first in the text sequence A bit exists as a mask; a value of 0 indicates a node. The first in the text sequence No mask exists. This indicates that the real tag will be displayed. Assigned to nodes The first in the text sequence The predicted probability of a position.

[0056] The graph neural network model is trained and optimized by calculating the cross-entropy loss between the predicted probability distribution and the true word distribution until the loss function converges or the maximum number of iterations is reached. At this point, the trained graph neural network model and the semantic representation with topology enhancement are obtained. This is the final road network representation. .

[0057] Through this mask prediction task, the model is forced to simultaneously understand the textual semantic attributes of roads and the topological connectivity of the road network, thereby learning a highly discriminative road network representation.

[0058] VII. Representing the road network Input a large language model, fine-tune the strategy through instructions, and perform downstream task prediction.

[0059] To achieve accurate representation of road networks in unseen urban areas, this invention constructs an instruction fine-tuning strategy. A linear projection layer is used to map the road network representation to the input embedding space of a large language model (in this specific embodiment, the vicuna-7B-v1.5 large language model is used), establishing a unified representation dimension for structured topological data and unstructured text semantics. in, For learnable parameters, This is the mapped road semantic representation. It is a linear projection layer.

[0060] Mapped road representation The prompt is concatenated with a natural language instruction template U containing a task description to generate a multimodal compound prompt word. The task description covers multiple downstream tasks such as road segment function recognition, traffic speed prediction, and travel time estimation, which are used to constrain the inference boundary of the model. The large language model outputs the prediction result of the corresponding task based on the prompt word.

[0061] When the downstream task is road segment function classification, the large language model predicts and outputs the category to which the road belongs (such as main road or auxiliary road); when the downstream task is traffic speed prediction, the large language model predicts and outputs the average speed of vehicles running on the road; when the downstream task is travel time estimation, the large language model outputs the travel time of a specified route from a specified origin to a specified destination.

[0062] This invention uses cross-entropy loss, commonly used in generative tasks, as the loss function for fine-tuning large language models. It only calculates the loss of the predicted target part, while the instruction part is not included in the loss calculation. in, The cross-entropy loss is used, where the starting point L+1 is the first word of the answer and the ending point T is the last word of the answer. Indicates the first in the text sequence The true words at each location, i.e., the labels, indicate the category to which the road segment belongs when the downstream task is road segment function classification; when the downstream task is traffic speed prediction, the labels indicate the average speed of vehicles running on the road; when the downstream task is travel time estimation, the labels indicate the estimated travel time from a specified origin to a specified destination. Represents the text history sequence, i.e., position. All previous lexical units (including road representations) and natural language commands (U). This represents the probability that the large language model predicts the correct lexical.

[0063] While keeping the basic word embedding weights of the large language model frozen, the parameters are updated only by deploying a low-rank adapter (LoRA) for the attention mechanism layer. This ensures that the model's prior knowledge and road-specific map features are deeply integrated under limited computing resources, thereby improving the model's zero-shot generalization performance for unknown urban areas.

[0064] Repeat this process until... Convergence yields the fully fine-tuned large language model. Because the large language model has generalization capabilities, in practical applications, it can be fine-tuned for only one downstream task and then directly used for other downstream tasks, or it can be fine-tuned separately for each downstream task.

[0065] After training, during the actual operation phase, multimodal road data of the specified target area is acquired, and a pre-trained large language model is used to obtain the text description of each road. This description is then input into the trained graph neural network model to obtain the road representation. These road representations are combined with natural language instruction templates and then input into the fine-tuned large language model to obtain the prediction results of the downstream task.

[0066] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A text-enhanced multimodal road network representation learning method, characterized in that, include: S1, divide the roads within the target area into road segments according to intersections; collect multimodal static attribute information of each road segment, and further obtain panoramic street view images of the center point of each road segment; S2, the multimodal large language model generates natural language descriptions of corresponding road segments based on street view images, and inputs them into the first large language model along with static attribute information and preset prompt words to obtain structured text descriptions; S3. Construct a text-enhanced road network graph with road segments as nodes, the topological relationships between road segments as directed edges, and the structured text descriptions corresponding to the nodes as node attributes, and extract the context subgraph corresponding to each node. S4, taking each context subgraph as a unit, transforms the structured text description of each road segment node into a text sequence, and the graph neural network obtains the topology-enhanced semantic representation based on the text sequence of all nodes in the context subgraph. S5. After performing random masking at the word level on the text sequence, it is concatenated with the topologically enhanced semantic representation to obtain the fused representation. Based on the fused representation and text sequence, the first cross-entropy loss is calculated and the parameters of the graph neural network are updated until convergence, resulting in a trained graph neural network. S6, consisting of a text-enhanced road network graph composed of roads to be represented, is input into a trained graph neural network to obtain semantic representations of the roads after topological enhancement.

2. The text-enhanced multimodal road network representation learning method according to claim 1, characterized in that, In S1, the multimodal static attribute information includes: the geometry of the road segment, road length, number of lanes, single / double lane attributes, traffic permission restrictions, and special intersection structure identifiers.

3. The text-enhanced multimodal road network representation learning method according to claim 1, characterized in that, In step S3, the process of extracting the context subgraph corresponding to each node is as follows: S301, calculate the correlation score between the current node and other nodes in the text-enhanced road network graph in turn, and further obtain the candidate nodes with the top K correlation scores, and form a subgraph node set with the node. S302 retains only directed edges whose starting and ending nodes are both located in the subgraph node set, and forms a context subgraph with the subgraph node set.

4. The text-enhanced multimodal road network representation learning method according to claim 1, characterized in that, In S4, the process of obtaining the topologically enhanced semantic representation is specifically as follows: First, the structured text description of each road segment node is transformed into a text sequence containing start and end markers using a tokenizer; Secondly, the text sequence is input into the language model to obtain semantic representation, and then the lexical units of the starting marker are extracted from the semantic representation; Finally, the initial tokens of all road segments are stacked and then input into the graph neural network along with the adjacency matrix corresponding to the context subgraph to obtain the topology-enhanced semantic representation.

5. The text-enhanced multimodal road network representation learning method according to claim 4, characterized in that, The adjacency matrix is ​​a matrix with road segments as rows and columns, and the topological connection relationship between road segments as element values. An element value of 1 indicates that the road segment corresponding to the row of the element and the road segment corresponding to the column of the element are connected by an intersection, and an element value of 0 indicates that there is no topological connection between the road segment corresponding to the row of the element and the road segment corresponding to the column of the element.

6. The text-enhanced multimodal road network representation learning method according to claim 1, characterized in that, In S5, the calculation formula for the fusion representation is as follows: ； in, Represents a column vector consisting entirely of 1s, with the top right corner... Indicates transpose. Indicates the outer product; Indicates feature splicing; Indicates a linear layer; Represents a non-linear activation function. This represents the fusion representation of the i-th road segment. This represents the number of tokens (excluding start and end markers) in the text sequence of the i-th road segment. This represents the semantic representation of the i-th road segment after random masking. This represents the topologically enhanced semantic representation of the i-th road segment.

7. The text-enhanced multimodal road network representation learning method according to claim 1, characterized in that, In S5, the specific formula for calculating the first cross-entropy loss is as follows: ； in, Indicates the total number of road segments. This represents the mask position indicator function. A value of 1 indicates that the i-th road segment in the masked text sequence is the first... The ith bit contains a mask; a value of 0 indicates that the ith bit in the text sequence after the ith road segment is masked is the ith bit. No mask exists for this bit; This indicates that the j-th word in the text sequence is... The text sequence after masking the i-th road segment The predicted probability of the bit. Represents cross-entropy loss, Indicates a set of road segments. This represents the i-th road segment. This represents the number of tokens in the text sequence of the i-th road segment, excluding the start and end markers.

8. The text-enhanced multimodal road network representation learning method according to claim 1, characterized in that, Also includes: The semantic representation of the road topology to be represented, after enhancement, and the instruction template of the specified downstream task are input into the second language model to obtain the downstream task results of the road segment. The downstream tasks include road segment function identification, traffic speed prediction, and travel time estimation.

9. The text-enhanced multimodal road network representation learning method according to claim 8, characterized in that, The second major language model includes an adapter, and the fine-tuning process of the second major language model is as follows: S601, an adapter is added to the attention mechanism layer of the original second-largest language model to obtain a second-largest language model containing the adapter; S602, the topologically enhanced semantic representation is mapped to the input embedding space of the second largest language model to obtain the mapped semantic representation; S603, input the mapped semantic representation and the instruction template of the specified downstream task into the second language model, and calculate the second cross-entropy loss based on the output results; S604, update the adapter parameters according to the second cross-entropy loss and repeat S603 until convergence, to obtain the second large language model with fine-tuning.

10. A text-enhanced multimodal road network representation learning method according to claim 8, characterized in that, In S603, the specific formula for calculating the second cross-entropy loss is as follows: ； in, denoted as the predicted cross-entropy loss, T represents the total length of the sequence, the first L words of the sequence are the instruction template and the semantic representation after mapping, and L+1 to T are the results of the downstream task generated by the prediction of the second largest language model; This indicates the probability that the second largest language model correctly predicts a word. Represents the text history sequence, i.e., position. All previous morphemes; This represents the h-th word in the sequence.

Images