Travel time prediction method based on multi-modal semantic encoding and large language model

Abstract

The present application belongs to the field of traffic control system, and particularly relates to a travel time prediction method based on multi-modal semantic coding and large language model. The method is subjected to a multi-modal semantic coding module to jointly code road structure features, POI semantic features and remote sensing visual features, and finally obtain node semantic vectors, sub-path semantic vectors and global path semantic representation, so as to realize hierarchical modeling of various factors affecting travel time. Subsequently, a text description generation module is used to convert structured path semantic information into natural language description rich in semantics. Finally, the semantic understanding and reasoning ability of the large language model is used to input the generated path semantic description into the LLM, and the zero-shot or few-shot prompt optimization strategy is combined to realize the reasoning prediction and explainable analysis of the travel time, so as to improve the accuracy, generalization ability and result explainability of the travel time prediction, and further improve the precision of the travel time prediction.

Description

Technical Field

[0001] This invention pertains to traffic control systems, and particularly relates to a method for travel time prediction based on multimodal semantic coding and large language models. Background Technology

[0002] With the improvement of urban sensing capabilities and the popularization of mobile internet, massive amounts of mobile data such as vehicle GPS and mobile phone signals are continuously recorded, providing a foundation for travel time prediction. The core task of travel time prediction is to predict the time required to traverse a given path, departure time, and external environmental conditions. It is essentially a complex modeling problem that is high-level, dynamic, and nonlinear.

[0003] Early methods primarily relied on historical statistics, predicting traffic by calculating the historical average travel time for specific road segments during the same time period. While simple, these methods struggled to handle real-time traffic fluctuations. Subsequently, traditional machine learning methods such as support vector machines and random forests were applied, improving predictive performance by incorporating features such as time and road type. However, these methods still heavily depended on manual feature engineering and were insufficient for characterizing the dynamic relationships within complex traffic systems.

[0004] In recent years, deep learning methods have become mainstream. Recurrent neural network-based methods can model the temporal features of trajectories, but they typically treat the entire path as a single sequence, making it difficult to effectively capture complex spatial dependencies. Subsequently, the road network was modeled as a graph structure, and graph convolutional networks, graph attention networks, and spatiotemporal graph neural networks were used to jointly model the spatial and temporal features of roads, improving prediction accuracy.

[0005] Despite the good results achieved by existing methods, the following problems remain: First, insufficient utilization of multimodal contextual information. Second, weak generalization ability to rare scenarios and complex traffic conditions. Third, lack of interpretability; most models directly map high-dimensional features to temporal results, failing to explain the basis of predictions. Fourth, lack of intermediate semantic representations, making the model's reasoning process lack interpretability.

[0006] In recent years, Large Language Models (LLM) and Multimodal Large Language Models (MLLM) have demonstrated strong capabilities in complex semantic understanding, multi-source information association, and logical reasoning, providing new research ideas for travel time prediction. Chinese patent document CN109741597A discloses a method for predicting bus route travel time based on an improved deep forest, including: collecting bus route information, GPS information, and route information; normalizing the collected data; inputting the processed data into an improved deep forest for training; the improved deep forest consists of two parts: convolutional multi-granularity scanning and a restricted cascaded forest; and selecting seven variables as training samples. The eight-dimensional input vector The current road segment's travel time is selected as the output vector. The method selects 70% of the sample data as training samples and inputs them into the improved deep forest, while the remaining 30% is input into the improved deep forest detection training results. However, it mainly relies on structured numerical features such as bus routes, weather, and road segment travel time for prediction, making it difficult to comprehensively represent the complex factors affecting travel time. Furthermore, it only learns from single road segment features, failing to effectively capture spatial dependencies within the route and cross-regional traffic status changes, resulting in weak generalization ability. In addition, this method is essentially still a traditional regression prediction framework, directly mapping features to time results, lacking intermediate semantic representation and reasoning processes, and unable to provide interpretable analysis of the prediction results.

[0007] Chinese patent document CN120032530A discloses a deep learning-based method for predicting bus arrival times based on spatiotemporal features, implemented by a bus data collection system. The method includes the following steps: S1: Using bus stops and inter-stop sections as nodes in the operating area to obtain an undirected graph; S2: Collecting GPS data and calculating travel time and stop time; S3: Calculating temporal features using statistical methods and obtaining spatial features from map information; S4: Concatenating the temporal and spatial features to obtain complete feature data; S5: Sorting the complete feature data and combining it with the undirected graph to form a set as input; S6: Establishing and training a regional bus arrival time prediction model; S7: Calculating the predicted arrival times of regional buses. However, the introduction of POI features, which are simply concatenated as static node attributes, leads to a lack of effective semantic interaction between multimodal information, easily resulting in decreased generalization ability and large prediction errors. Furthermore, this model is essentially still a black-box prediction method of "feature input - neural network regression - time output," unable to provide an interpretable reasoning basis for the prediction results.

[0008] To address the problems of insufficient utilization of environmental semantic information, uninterpretable models, and weak generalization ability in traditional methods, this invention proposes a travel time prediction method based on multimodal semantic coding and a large language model. Summary of the Invention

[0009] To address the aforementioned technical challenges, this invention proposes a travel time prediction method based on multimodal semantic coding and a large language model. The core task of travel time prediction is to predict the travel time of a given path based on trajectory data and its corresponding multimodal environmental information. The method first performs map matching processing on the original trajectory data, mapping discrete GPS points to a road network, thereby transforming the trajectory into a natural language description of the path. All subsequent modeling and prediction use the path as the basic input unit. By processing the path node sequence, the distribution information of POIs around the path, and remote sensing image features, the final travel time prediction result and inference explanation for the path are obtained. It mainly includes five stages: trajectory data preprocessing, multimodal data construction, multimodal semantic coding, path text generation, and travel time inference prediction based on a large language model. This method uses the structural, visual, and contextual information of the path at different spatial granularities for unified modeling, and transforms the path semantics into a natural language description, thereby utilizing a large language model to complete the travel time prediction.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: The method for travel time prediction based on multimodal semantic coding and large language models includes the following steps: S1. Preprocess the raw GPS trajectory data of the target path to generate an ordered sequence of road nodes for the target path; S2. Construct multimodal data for each road node in the ordered road node sequence of the target path. The multimodal data includes path structure data, POI data, and remote sensing image data. S3. Based on the constructed multimodal data, perform multi-granularity modeling; the multi-granularity includes node granularity, sub-path granularity, and global path granularity; S4. Construct a multimodal semantic coding module, which includes a node semantic coding submodule, a sub-path semantic coding submodule, and a global path semantic coding submodule; the node semantic coding submodule includes a graph attention network; the sub-path semantic coding submodule includes a Transformer encoder based on a multi-head self-attention mechanism; and the global path semantic coding submodule includes a multi-layer Transformer coding structure. Input the node granularity, sub-path granularity, and global path granularity constructed in step S3 into the pre-trained multimodal semantic encoding module, and output the node semantic vector, sub-path semantic vector, and global path semantic representation respectively; S5. Construct a text description generation module, which includes a semantic mapping submodule and a regression branch submodule; Input the node semantic vector, sub-path semantic vector, and global path semantic representation generated in step S4 into the pre-trained text description generation module, and output the corresponding label probabilities and continuous semantic attributes; generate the path natural language description text based on the label probabilities and continuous semantic attributes. The semantic mapping submodule includes a fully connected layer and a Softmax classification layer; S6. Construct a question template for travel time prediction; S7. Input the path natural language description text generated in step S5, the question template for trip time prediction constructed in step S6, and the trip start time into the large language model to predict the trip time.

[0011] Preferably, in step S1, the original GPS trajectory data is cleaned to remove abnormal positioning points and drift point samples, and a map matching algorithm is used to map the discrete GPS point sequence to a standard road network graph structure to generate a corresponding ordered road node sequence.

[0012] Preferably, step S2 includes: S21. Constructing Path Structure Data: Based on the OpenStreetMap open-source geodatabase, extract the road structure attributes corresponding to each road node in the ordered sequence of road nodes of the target path, and encode the road structure attributes into a road attribute vector. This is used to construct the path structure data; The road attribute vector Each dimension represents one or more of the following: road length, number of lanes, and road grade. All attribute features have been normalized and mapped to a unified dimensional space. The path structure data is represented as follows: (1) in, This represents the path structure data of the target path P. This represents the road attribute vector corresponding to the i-th road node; N represents the number of road nodes in the target path; S22. Constructing POI Data: Based on the OpenStreetMap open-source geodatabase, using road nodes as the center, retrieve POI information around the nodes within a preset spatial radius, obtain the corresponding POI identifiers and their functional categories, and form a POI set. , represented as: (2) in, For road nodes This is a collection of all Points of Interest (POIs) retrieved within a radius r centered at a given point. Each element contains the POI's coordinates and function category. Represents the latitude and longitude coordinates of the j-th POI, used to identify the spatial geographic location of the POI; The j-th POI's functional category label is represented by: dis() represents the spatial distance function, usually Euclidean distance or spherical distance, used to calculate the straight-line distance between the POI coordinates and the road node coordinates; r is the preset spatial buffer radius, used to define the spatial neighborhood range centered on the road node. This represents the spatial location of the i-th road node.

[0013] S23. Constructing remote sensing image data: Based on the latitude and longitude coordinates of all road nodes in the ordered road node sequence, calculate the minimum bounding rectangle area covering the entire target path, and crop the target path remote sensing image area from the original remote sensing image based on the minimum bounding rectangle area; and form node remote sensing image data from the target path remote sensing image area after cropping.

[0014] In a further preferred embodiment, step S23 involves cropping a fixed-size local image patch within the target path remote sensing image area, centered on each road node, according to a set spatial scale s, to obtain the node remote sensing image patch corresponding to each road node. Each node remote sensing image patch is then resampled and normalized to form a node remote sensing image sequence. , (3) Where N represents the number of road nodes, I represents the remote sensing image corresponding to the target path, and s represents the cropping scale centered on a single road node. Road nodes The corresponding image block.

[0015] Preferably, step S3 specifically includes: S31. Using the spatial coordinates of road nodes as a unified index key, spatially align the path structure data, POI data, and remote sensing image data of road nodes to form a node granularity that includes multimodal data. S32. Based on the semantic changes of node granularity, divide the node granularity into several semantically consistent sub-path units to form sub-path granularity. S33. The sequence structure formed by the sub-path granularities divided in step S32 according to the passage order is taken as the global path granularity.

[0016] In a further preferred embodiment, step S32 involves dividing the multimodal data at the node granularity into several semantically consistent sub-path units based on the changes in road structure attributes, POI functional category distribution, and traffic status of the spatially aligned road nodes in the ordered road node sequence.

[0017] More preferably, in step S32, when an adjacent road node meets the corresponding preset threshold condition in any dimension of road structure attribute, POI functional category distribution change, and historical traffic status, the adjacent road node is taken as the sub-path boundary, the current node is the previous sub-path boundary, and the next node is the current sub-path boundary, thereby constructing a sub-path sequence with consistent structure and semantics; if three or more consecutive road nodes meet the corresponding preset threshold condition in any dimension, the optimal node is selected from the road nodes, and this node is taken as the previous sub-path boundary; its next node is taken as the current sub-path boundary. The corresponding preset threshold conditions include: 1) Judgment based on changes in road structure attributes: The road structure difference degree is calculated based on the road attribute vectors corresponding to adjacent road nodes. When the road structure difference degree between adjacent road nodes is greater than a preset road structure difference threshold, such as greater than 0.5, it is determined that the road structure between adjacent road nodes has changed significantly, and the position of the adjacent road node is used as the sub-path division boundary. The road structure difference degree can be obtained by calculating the Euclidean distance between the normalized road attribute vectors of adjacent road nodes.

[0018] 2) Based on the distribution change of POI functional categories, when the cosine similarity of the node vectors of adjacent road nodes is less than 0.6, the position of the adjacent road node is used as the boundary of the sub-path division; the cosine similarity is calculated based on the vector dot product and norm. 3) Determine based on historical traffic status, that is, calculate the average historical traffic speed of each road node based on the ordered road node sequence and the timestamp information corresponding to the road nodes, and then calculate the relative rate of change of historical traffic speed of adjacent road nodes. If the value is greater than 0.3, then the position of the adjacent road node is used as the boundary for sub-path division.

[0019] The optimal node refers to the road node with the highest score selected by normalizing and weighting the road structure difference, 1-cosine similarity, and relative change rate of historical traffic speed.

[0020] Preferably, in step S4, the node granularity, sub-path granularity, and global path granularity constructed in step S3 are input into the pre-trained multimodal semantic encoding module, which outputs node semantic vectors, sub-path semantic vectors, and global path semantic representations, respectively. Specifically: a. Independently encode the multimodal data at the node granularity in step S31 to obtain multimodal features. Input the multimodal features into the node semantic encoding submodule and output the node semantic vector. The node semantic encoding submodule includes a multimodal node feature fusion structure and a graph attention network. b. Based on the sub-path granularity constructed in step S32, each sub-path is represented as an ordered sequence of nodes consisting of multiple node semantic vectors in the actual driving order; the sub-path passes through the sub-path semantic encoding sub-module to generate sub-path semantic vectors. c. Based on the global path granularity constructed in step S33, the global path is represented as an ordered sequence of multiple sub-path semantic vectors according to the actual passage order; the global path is then encoded using global path semantics to output the global path semantic representation.

[0021] Preferably, step a specifically includes: a1. Independently encode the multimodal data at the node granularity to generate road structure features for road nodes. POI semantic features of road nodes Remote sensing visual features of road nodes ; a2. Introduce adaptive weighted fusion to analyze the road structure features of road nodes. POI semantic features of road nodes Remote sensing visual features of road nodes The fusion process is performed to generate a multimodal fusion representation of the road nodes: The obtained road structure features of the road nodes POI semantic features of road nodes Remote sensing visual features of road nodes Weight coefficients for the three types of features are generated using a learnable weight mapping function. , and The sum of the weights is then ensured to be 1 through normalization; subsequently, the three types of modal features are weighted and combined to obtain the node multimodal fusion representation: (4) in, Let represent the multimodal fusion representation of the i-th road node. , , These represent the road structure features, POI semantic features, and remote sensing visual features of the i-th road node, respectively. a3. Input the generated multimodal fusion representation of the road nodes into the graph attention network, and generate node semantic vectors that fuse spatial structure information and environmental semantic information by adaptively learning the influence weights between neighboring nodes: (5) in, Represents the semantic vector of a node.

[0022] Preferably, step b specifically comprises: b1. Introduce location encoding information into the node semantic vector to represent the relative order relationship between road nodes, and generate the corresponding location encoding vector. .

[0023] b2. Add the position encoding vector to the node semantic vector element by element to obtain the node representation with fused order information: (6) b3. Based on the sub-path granularity obtained in step S32, each sub-path is represented as an ordered sequence of nodes composed of multiple nodes with fused order information, arranged according to the actual driving order. The sub-path is then input into the Transformer encoder layer, where a multi-head attention mechanism is used to characterize the interaction relationships between different nodes, resulting in the context-enhanced representation of the sub-path, i.e., the sub-path semantic vector. (7) Where j represents the index number of the j-th sub-path, and its value ranges from 1 to the total number of sub-paths J; This represents the number of road nodes contained within the j-th sub-path; (j,1) to (j, ) represent the first to the last sub-path in the j-th sub-path. The index of the node; represents the semantic vector of each node in the j-th sub-path; Transformer() represents a Transformer encoder based on a multi-head self-attention mechanism, used to perform context modeling on the node sequence within the sub-path, capturing the temporal dependencies and spatial associations between nodes.

[0024] Preferably, step c specifically includes: c1. Generate a corresponding position encoding vector for each sub-path semantic vector. ; c2. Add the position encoding vector to the sub-path semantic vector element-wise to obtain the sub-path representation with fused order information: (8) c3. The sub-path representations with fused sequence information are transformed using a Transformer encoder to obtain a context-enhanced sub-path representation sequence: (9) in, H represents the semantic representation vector of the j-th subpath, where j is the total number of subpaths. ) indicates a sequence of encoded context-enhanced representations.

[0025] c4. Assess the importance of the encoded context-enhanced representation sequence and calculate the attention score for each sub-path: (10) in, Let W and b represent the context-enhanced semantic representation of the j-th subpath, where W and b are learnable parameters. This represents the attention score for the corresponding sub-path.

[0026] c5. Normalize the attention scores of the sub-paths to obtain the attention weights of the sub-paths: (11) in, This represents the attention weight of the j-th sub-path, reflecting its relative importance in the overall path.

[0027] c6. Perform weighted fusion of the sub-path representations according to the weights to obtain the global path semantic representation: (12) in, This represents the global semantic representation of the entire path.

[0028] Preferably, in step S5, the node semantic vector, sub-path semantic vector, and global path semantic representation generated in step S4 are input to the corresponding semantic mapping module, and the label probabilities are output according to the preset node semantic labels, sub-path semantic labels, and global path semantic labels, respectively. At the same time, a continuous attribute regression branch is used to predict the continuous semantic attributes of nodes, sub-paths, and global paths, and generate continuous attribute vectors. The path natural language description text is generated together based on the label probabilities and continuous attribute vectors.

[0029] More preferably, the node semantic tags include one or more of the following: densely populated commercial areas, school areas, residential areas, office areas, transportation facility areas, and mixed-use areas; The semantic mapping of a node semantic vector is represented as follows: (13) in, This represents the semantic label probability distribution of node i. and These are learnable parameters; The sub-path semantic tags include one or more of the following: commercial area road segments, school road segments, and mixed-use area road segments; The sub-path semantic vector is obtained through mapping: (14) in, This represents the semantic tag probability distribution corresponding to the j-th sub-path. This represents the semantic vector of the j-th sub-path. , These represent the learnable weight matrix and bias term in the sub-path semantic mapping, respectively.

[0030] The global path semantic tags include one or more of the following: high proportion of commercial areas, high proportion of school areas, and high proportion of mixed functional areas. Global path semantic representation After mapping, we get: (15) in, This represents the global path semantic label probability distribution corresponding to the target path. , These represent the learnable weight matrix and bias term in the global path semantic mapping, respectively.

[0031] The node-level regression is represented as: (16) in, Let represent the continuous attribute vector corresponding to the i-th road node, where each dimension represents the node's continuous attributes such as intersection complexity, congestion risk, road traffic flow, and POI mixing degree. , These represent the learnable weight matrix and bias term in the continuous attribute regression path of a node, respectively.

[0032] Similarly, continuous attribute vectors are obtained by performing continuous attribute regression branch prediction on both the sub-path and the global path. and ;in, Let represent the continuous attribute vector corresponding to the j-th sub-path, where each dimension represents the sub-path's continuous attributes: average congestion risk, degree of road structure change, traffic stability, and average smoothness. This represents a continuous attribute vector corresponding to the entire target path, where each dimension represents a global continuous attribute such as the proportion of high-risk sub-paths, the overall congestion risk, and the overall road complexity. After semantic mapping is completed, the path natural language description is generated according to preset rules based on the label probability distribution of nodes, sub-paths and global paths and continuous attribute vectors output by the semantic mapping module.

[0033] More preferably, the preset rule is specifically: At the node granularity level, the label with the highest probability in the node semantic label probability distribution is selected as the node dominant label, and a natural language description of the node is generated based on the node dominant label and the node continuous attributes. At the sub-path granularity level, the label with the highest probability in the sub-path semantic label probability distribution is selected as the dominant label of the sub-path, and the sub-path natural language description is generated by combining the continuous attributes of the sub-path and its attention weight in the global path. At the global granularity level, the label with the highest probability in the global path semantic label probability distribution is selected as the global dominant label, and combined with global continuous attributes to generate a global path natural language description; The natural language descriptions generated at the node granularity, sub-path granularity, and global path granularity are concatenated according to the actual passage order of the target path to obtain a complete path natural language description.

[0034] Preferably, the question template for trip time prediction in step S6 includes the reasoning role, reasoning target, output format, and reasoning process of the language model.

[0035] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention employs a multimodal, multi-granularity path semantic representation mechanism, integrating road structure information, remote sensing images, and semantic information of the surrounding POI environment. It also performs hierarchical modeling of the path at three levels: node, sub-path, and global path, achieving a comprehensive semantic representation of the path's traffic status. Compared to existing methods that rely solely on historical statistical information, single path features, or shallow spatiotemporal relationships for modeling, this invention effectively reduces prediction errors caused by complex road structure changes, functional area changes, and sudden traffic events. It maintains stable prediction performance even in complex traffic environments and cross-regional scenarios, thus significantly enhancing the model's robustness and prediction accuracy.

[0036] (2) This invention employs a multimodal, multi-granularity path semantic representation mechanism, breaking through the limitations of traditional travel time prediction that relies solely on historical speed or single structural features. It also incorporates path structure information, remote sensing images, and semantic information of the surrounding POI environment. Furthermore, it models the path hierarchically at three granularities, achieving a hierarchical expression from local to global. This approach can comprehensively describe the characteristics of the path from multiple dimensions, enhancing the model's expressive power and robustness.

[0037] (3) This invention transforms path semantic vectors into text descriptions, avoiding the problem of insufficient interpretability caused by traditional deep models directly mapping high-dimensional features into time values. By describing path semantics as natural language, the prediction results not only provide a prediction of travel time, but also explain why it takes so long, thereby enhancing the interpretability and credibility of the prediction results and enabling users to understand the basis of the prediction.

[0038] (4) The present invention constructs a question template for travel time prediction, which clearly constrains the reasoning goal of the large language model, avoids generating redundant content that is unrelated to the prediction goal, and improves the stability of the prediction results. This method gives full play to the advantages of the large language model in semantic understanding and reasoning.

[0039] (5) In the training phase, the present invention uses the large language model as a frozen external inference module to avoid repeated training of the large-scale language model, significantly reduce training costs, and improve the generalization ability of the model in different city and path scenarios. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall framework of the travel time prediction method in Embodiment 1 of the present invention; Figure 2 This is a multimodal spatial alignment and multi-granularity path construction diagram of the travel time prediction method in Embodiment 1 of the present invention; Figure 3 This is a multimodal three-granularity coding diagram of the travel time prediction method in Embodiment 1 of the present invention; Figure 4 This is a multimodal node feature fusion structure diagram of the travel time prediction method in Embodiment 1 of the present invention; Figure 5 This is a flowchart of the travel time prediction method model in Embodiment 1 of the present invention. Detailed Implementation

[0041] In this invention, a trajectory refers to a sequence of GPS points continuously sampled by a vehicle over time, typically represented as a set of timestamped spatial points. A path refers to an ordered sequence of road nodes in a road network corresponding to the trajectory data after map matching.

[0042] The travel time prediction method based on multimodal semantic coding and large language model (LLM) first uses a multimodal semantic coding module to jointly encode road structure features, POI semantic features, and remote sensing visual features, ultimately obtaining node semantic vectors, sub-path semantic vectors, and a global path semantic representation, thus achieving hierarchical modeling of multiple factors affecting travel time. Subsequently, a text description generation module transforms the structured path semantic information into semantically rich natural language descriptions. Finally, leveraging the semantic understanding and reasoning capabilities of the large language model (LLM), the generated path semantic descriptions are input into the LLM, and combined with zero-shot or few-shot suggestion optimization strategies, inferential prediction and interpretable analysis of travel time are achieved. This improves the accuracy, generalization ability, and interpretability of travel time prediction, thereby enhancing the precision of travel time prediction.

[0043] During the model training phase, this invention employs a phased collaborative training strategy to ensure the stability and consistency of path multimodal semantic modeling, textual description, and travel time inference. In this invention, the multimodal path encoder and text generation module are trainable modules; the large language model serves as an external semantic inference module, with its parameters remaining fixed during training and not participating in backpropagation. Training data originates from real-world urban traffic trajectory datasets. The raw data records the continuous driving trajectories of vehicles in the urban road network and their corresponding time information. Through map matching and path reconstruction operations, the raw trajectories are mapped to candidate paths in the road network, and training samples are constructed using paths as the basic unit. For each path sample, following data preprocessing and multi-granularity construction methods, path structure data, surrounding POI data, and corresponding remote sensing image data are generated, and multimodal representations of nodes, sub-paths, and global paths are further constructed.

[0044] Example 1 Travel time prediction methods based on multimodal semantic coding and large language models, such as Figure 1 As shown, it includes the following steps: S1. Preprocess the raw GPS trajectory data of the target path to generate an ordered sequence of road nodes for the target path; The original data used in this invention takes Chengdu as an example, and the trajectory data comes from the Chengdu taxi trajectory dataset of November 2016. This dataset records the continuous driving trajectory information of Chengdu taxis in the urban road network. The trajectory data includes vehicle identification information, timestamp information, location information, and vehicle operating status information. The trajectory data records the continuous GPS sampling points of the vehicle in chronological order, and each trajectory sample consists of a series of latitude and longitude coordinates.

[0045] Specifically, the raw GPS trajectory data of the target path is cleaned to remove abnormal location points and drift points. A map matching algorithm (e.g., a map matching algorithm based on a Hidden Markov Model) is then used to map the discrete GPS point sequence onto the standard road network map structure of Chengdu, generating a corresponding ordered sequence of road nodes and determining their corresponding road nodes and spatial coordinates. After map matching is completed, based on the continuous passenger-carrying intervals of the same vehicle, the corresponding road node sequence is divided into independent path samples, and the actual travel time is calculated based on the path start and end times.

[0046] S2. Construct multimodal data for each road node in the ordered road node sequence of the target path. The multimodal data includes path structure data, POI data, and remote sensing image data. To comprehensively describe the characteristics of the route, this method uses three types of core data: route structure data, POI data, and remote sensing image data.

[0047] S21. Constructing Path Structure Data: Based on the OpenStreetMap open-source geodatabase, extract the road structure attributes corresponding to each road node in the ordered sequence of road nodes of the target path, and encode the road structure attributes into a road attribute vector. This is used to construct the path structure data; The road attribute vector Each dimension represents one or more of the following: road length, number of lanes, and road grade. All attribute features have been normalized and mapped to a unified dimensional space. The path structure data is represented as follows: (1) in, This represents the path structure data of the target path P. The path structure data represents the road attribute vector corresponding to the i-th road node; N represents the number of road nodes in the target path; the path structure data is used to characterize the local road structure features and road traffic attributes of the path.

[0048] S22. Constructing POI Data: Based on the OpenStreetMap open-source geodatabase, using road nodes as the center, retrieve POI information around the nodes within a preset spatial radius, obtain POI identifiers around the path nodes, and count the functional categories of the POI identifiers to form a POI set. The POI data originates from the OpenStreetMap open-source geographic database for the Chengdu area. This data contains geographic entity information for different functional areas within Chengdu, with key fields including POI identifiers, latitude and longitude coordinates, and functional category labels. These functional categories include commercial facilities, educational institutions, medical institutions, transportation facilities, office areas, and residential service facilities.

[0049] This invention uses the spatial coordinates of road nodes as the center and performs spatial neighborhood retrieval on POI data within a preset spatial radius r (e.g., 200 meters) to obtain information on surrounding facilities such as hospitals, shopping malls, and office buildings. Specifically, by constructing a POI spatial index structure, the invention performs neighborhood queries on the POI database based on the latitude and longitude coordinates of nodes, extracts POI plans within a specified radius, and forms a POI set. : (2) in, For road nodes This is a collection of all Points of Interest (POIs) retrieved within a radius r centered at a given point. Each element contains the POI's coordinates and function category. Represents the latitude and longitude coordinates of the j-th point of interest (POI), used to identify the spatial geographic location of the POI; This represents the functional category label of the j-th POI, such as commercial facilities, educational institutions, medical institutions, transportation facilities, office areas, and life services; dis() represents the spatial distance function, usually using Euclidean distance or spherical distance (such as the Haversine formula), used to calculate the straight-line distance between the POI coordinates and the road node coordinates; r is the preset spatial buffer radius, in meters, used to define the spatial neighborhood range centered on the road node, and POIs within this radius are considered to be related to that node; This represents the spatial location of the i-th road node.

[0050] In the multimodal semantic encoder part, for the set Different functional categories of POIs are statistically analyzed and indexed to form semantic feature representations of POIs corresponding to nodes, which are used to describe the category distribution and semantic attributes of the functional environment surrounding the node.

[0051] S23. Constructing remote sensing image data: Based on the latitude and longitude coordinates of all road nodes in the ordered road node sequence, calculate the minimum bounding rectangle area covering the entire target path, and crop the target path remote sensing image area from the original remote sensing image based on the minimum bounding rectangle area; remote sensing image data is formed by cropping the target path remote sensing image area.

[0052] The original remote sensing image can be obtained by replacing it with a map tile image. The map tile image is obtained by the OpenStreetMap map tile service, and local image blocks corresponding to each road node are cropped according to the location of the road node or the coverage of the path. If necessary, the target path is rendered and superimposed on the map tile image in the form of a polyline, thereby reducing the impact of imaging factors such as cloud cover on the extraction of visual modal features.

[0053] The remote sensing image data used covers the target area, and images from the same time period as the trajectory data are selected for processing. To ensure the spatial correspondence between the remote sensing images and path nodes, the minimum bounding rectangle covering the entire path is first calculated based on the latitude and longitude coordinates of the path nodes after map matching. Specifically, the left and right boundaries of the rectangle are determined by finding the minimum and maximum longitude values ​​of all nodes in the path, and the upper and lower boundaries are determined by finding the minimum and maximum latitude values ​​of the nodes, thus forming the minimum bounding rectangle of the path. Subsequently, image regions corresponding to the geographical area are cropped from the original remote sensing images for subsequent node-level cropping and feature extraction.

[0054] Specifically, at the level of each road node, a fixed-size local image patch is cropped within the remote sensing image area of ​​the target path, centered on each road node, within a spatial scale s (e.g., 200m × 200m). This yields the node remote sensing image patch corresponding to each road node. Each node remote sensing image patch is then resampled and normalized to form a node remote sensing image sequence. All nodal image patches are stored using node IDs as indexes to ensure spatial alignment and one-to-one correspondence between remote sensing visual information, path structure data, and POI data at the node level, represented as follows: , (3) Where N represents the number of road nodes, I represents the remote sensing image corresponding to the target path, and s represents the cropping scale centered on a single road node. Road nodes The corresponding image patch. Subsequently, the remote sensing image patch corresponding to the node will be... The input is fed into a pre-trained ResNet-50 for multi-layer feature encoding to extract its multi-scale spatial semantic information and generate node remote sensing visual embedding representations. This feature vector reflects the road morphology and building density around the node, providing visual support for path environmental semantic modeling. Remote sensing image data specifically includes road surface and building density; this type of data can effectively enhance the model's ability to understand environmental semantics. By fusing these three types of data and uniformly encoding road structure, functional semantics, and visual spatial information, a multimodal foundation for path semantic understanding will be provided.

[0055] S3. Based on the constructed multimodal data, perform multi-granularity modeling; the multi-granularity includes node granularity, sub-path granularity, and global path granularity; Multi-granularity modeling represents paths at three levels: nodes, sub-paths, and global paths, achieving a semantic characterization from local to global. Node granularity describes local traffic characteristics, sub-path granularity reflects the structural and semantic changes of continuous road segments, and global granularity characterizes overall traffic trends and cross-regional influence relationships. This multi-layered structure effectively captures complex dependencies within paths and provides a foundation for subsequent text generation and large language model inference, thereby enhancing the accuracy and interpretability of travel time prediction.

[0056] S31. Using the spatial coordinates of road nodes as a unified index key, spatially align the path structure data, POI data, and remote sensing image data corresponding to the road nodes to form a node granularity containing multimodal data. Specifically, after obtaining the road node sequence corresponding to the target path, the spatial coordinates of each road node are used as a unified index key to spatially align the path structure data, POI data, and remote sensing image data constructed by S2. This allows each node to be associated with its road attributes, the distribution of surrounding POIs, and the corresponding remote sensing image patch, forming a multimodal feature representation at the node granularity. This ensures that the node features are consistent with the corresponding POI information and remote sensing images, the sub-path features are consistent with the semantic distribution of their covered areas, and the global path features match the overall path range, thereby guaranteeing the semantic consistency of multimodal information at different granularity levels.

[0057] By combining the adjacency relationships, connection edge relationships, and travel order in the target path among the road nodes in the road network, node structural features are constructed. This results in a multimodal data representation at the node level that includes road structural features, POI semantic features, and remote sensing visual information. Its spatial alignment and multi-granularity path construction are as follows: Figure 2 As shown.

[0058] S32. Based on the semantic changes of node granularity, divide the node granularity into several semantically consistent sub-path units to form sub-path granularity. Based on the changes in road structure attributes, POI functional category distribution, and traffic status of spatially aligned road nodes in an ordered road node sequence, the multimodal data at the node granularity is divided into several semantically consistent sub-path units. Specifically, when an adjacent road node meets the corresponding preset threshold conditions in any dimension of road structure attributes, POI functional category distribution changes, and historical traffic status, the adjacent road node is taken as the sub-path boundary, the current node is the previous sub-path boundary, and the next node is the current sub-path boundary, thereby constructing a sub-path sequence with consistent structure and semantics; if three or more consecutive road nodes meet the corresponding preset threshold conditions in any dimension, the optimal node is selected from the road nodes and taken as the previous sub-path boundary; its next node is taken as the current sub-path boundary. The corresponding preset threshold conditions include: 1) Judgment based on changes in road structure attributes: The road structure difference degree is calculated based on the road attribute vectors corresponding to adjacent road nodes. When the road structure difference degree between adjacent road nodes is greater than a preset road structure difference threshold, such as greater than 0.5, it is determined that the road structure between adjacent road nodes has changed significantly, and the position of the adjacent road node is used as the sub-path division boundary. The road structure difference degree can be obtained by calculating the Euclidean distance between the normalized road attribute vectors of adjacent road nodes.

[0059] 2) Based on the distribution change of POI functional categories, when the cosine similarity of the node vectors of adjacent road nodes is less than 0.6, the position of the adjacent road node is used as the boundary of the sub-path division; the cosine similarity is calculated based on the vector dot product and norm. 3) Determine based on historical traffic status, that is, calculate the average historical traffic speed of each road node based on the ordered road node sequence and the timestamp information corresponding to the road nodes, and then calculate the relative rate of change of historical traffic speed of adjacent road nodes. If the value is greater than 0.3, then the position of the adjacent road node is used as the boundary for sub-path division.

[0060] The optimal node refers to the road node with the highest score selected by normalizing and weighting the road structure difference, 1-cosine similarity, and relative change rate of historical traffic speed.

[0061] Using the above rules, the original ordered road node sequence is divided into several sub-path units with consistent internal semantics and clear boundaries.

[0062] S33. The sequence structure formed by the sub-path granularities divided in step S32 according to the passage order is taken as the global path granularity.

[0063] At the global path granularity level, the entire path is represented as a sequence structure composed of multiple sub-path units arranged in the order of travel, which is used to describe the overall structural characteristics, environmental distribution, and travel trends of the path.

[0064] By constructing the above multi-granularity structure, the path is organized hierarchically from nodes to sub-paths and then to the global path, so that the path semantics can be expressed in a structured form, providing a unified data foundation for subsequent path semantic encoding and feature modeling.

[0065] S4. Construct a multimodal semantic coding module, which includes a node semantic coding submodule, a sub-path semantic coding submodule, and a global path semantic coding submodule; the node semantic coding submodule includes a graph attention network; the sub-path semantic coding submodule includes a Transformer encoder based on a multi-head self-attention mechanism; and the global path semantic coding submodule includes a multi-layer Transformer coding structure. Input the node granularity, sub-path granularity, and global path granularity constructed in step S3 into the pre-trained multimodal semantic encoding module, and output the node semantic vector, sub-path semantic vector, and global path semantic representation respectively; Specifically, such as Figure 3As shown, the method proposed in this invention generally includes the following stages: multimodal path data construction, multimodal path semantic encoding, path semantic structured representation, path semantic text generation, and travel time prediction processing based on a large language model. These stages are sequentially connected according to the data flow direction, forming a complete process from raw data to travel time output. First, the path's structural data, remote sensing image data, and surrounding context data are preprocessed and spatially aligned. Then, path features are encoded using multimodal encoding and multi-granularity semantics to obtain a path semantic representation. Finally, in the text generation stage, the path semantic representation is converted into a text description. Ultimately, the large language model performs semantic understanding and reasoning based on the generated path text description, outputting the corresponding travel time prediction result.

[0066] The step of inputting the node granularity, sub-path granularity, and global path granularity constructed in step S3 into the pre-trained multimodal semantic encoding module, and outputting the node semantic vector, sub-path semantic vector, and global path semantic representation respectively, is as follows: a. Independently encode the multimodal data at the node granularity in step S31 to obtain multimodal features. Input the multimodal features into the node semantic encoding submodule and output the node semantic vector. The node semantic encoding submodule includes a multimodal node feature fusion structure and a graph attention network. At the node level, the road network is modeled as a graph structure, where nodes represent road units and edges represent connections between adjacent road units. Each node corresponds to multimodal information that integrates road structural features, POI semantic features, and remote sensing visual features.

[0067] a1. Independently encode the multimodal data at the node granularity to generate road structure features for road nodes. POI semantic features of road nodes Remote sensing visual features of road nodes ; In this process, road structure data is encoded through an embedding mapping network to obtain the road structure features of road nodes. The embedded mapping network consists of two fully connected layers, with the ReLU activation function used in the middle for nonlinear mapping; POI data is vectorized using a semantic embedding module. This involves encoding the POI categories retrieved within a predefined spatial radius of a node using category indices, and mapping these indices to low-dimensional continuous vectors through a learnable fully connected embedding layer. When a node corresponds to multiple POI categories, the embedding vectors for each category are averaged and pooled to obtain the semantic feature representation of the node's POIs. ; Remote sensing visual features are extracted using a convolutional neural network. Specifically, a ResNet-50 model pre-trained on a publicly available image dataset is used to encode features of the remote sensing images corresponding to the nodes. Spatial features are extracted layer by layer through multiple convolutional layers, residual connection structures, and pooling operations. A fixed-dimensional visual feature vector, i.e., the remote sensing visual features of the road nodes, is obtained through global average pooling. It is used to characterize the spatial structure and environmental semantic information of the area surrounding the node.

[0068] a2. Introduce adaptive weighted fusion to analyze the road structure features of road nodes. POI semantic features of road nodes Remote sensing visual features of road nodes The paths are fused to generate a multimodal fusion representation of the road nodes: After completing the multimodal feature encoding, adaptive weighted fusion is introduced to fuse features from different modalities.

[0069] like Figure 4 As shown, the fusion structure includes weight calculation and weighted summation to achieve adaptive fusion of features from different modalities. This is used to obtain the road structure features of road nodes. POI semantic features of road nodes Remote sensing visual features of road nodes Then, weight coefficients for the three types of features are generated first using a learnable weight mapping function. , and The sum of the weights is then ensured to be 1 through normalization. Subsequently, the three types of modal features are weighted and combined to obtain the node's multimodal fusion representation: (4) in, Let represent the multimodal fusion representation of the i-th road node. , , These represent the road structure features, POI semantic features, and remote sensing visual features of the i-th road node, respectively. Through an adaptive weighted fusion mechanism, the model automatically adjusts the importance of each modality information according to different path environments, thereby improving the path semantic representation capability.

[0070] a3. Input the multimodal fusion representation of the generated road nodes into the graph attention network, and generate node semantic vectors that fuse spatial structure information and environmental semantic information by adaptively learning the influence weights between neighboring nodes. Specifically, to further characterize the influence of road topology on traffic conditions, the fused node features are input into a Graph Attention Network (GAT) for structural modeling. By adaptively learning the influence weights between neighboring nodes, node semantic vectors that fuse spatial structural information and environmental semantic information are generated. (5) in, Represents the semantic vector of a node.

[0071] b. Based on the sub-path granularity constructed in step S32, each sub-path is represented as an ordered sequence of nodes consisting of multiple node semantic vectors in the actual driving order; the sub-path passes through the sub-path semantic encoding sub-module to generate sub-path semantic vectors. At the sub-path granularity level, each sub-path is represented as an ordered sequence of multiple node semantic vectors arranged according to the actual driving order. It should be noted that the node semantic representation has already integrated road structure features, POI semantic features, and remote sensing visual features at the node level. Therefore, the sub-path semantic representation models the node sequence based on multimodal fusion, thereby preserving and further organizing multimodal semantic information. Considering the complex temporal dependencies and contextual relationships among nodes within a sub-path, this invention employs a Transformer encoder based on a multi-head self-attention mechanism to model the sub-path sequence.

[0072] b1. Introduce location encoding information into the node semantic vector to represent the relative order relationship between road nodes, and generate the corresponding location encoding vector. .

[0073] The positional encoding uses a sine function, and its dimension is consistent with the dimension of the node vector.

[0074] The positional encoding adopts the sine-cosine positional encoding method in Transformer, and its dimension is consistent with the dimension of the node semantic vector.

[0075] b2. Add the position encoding vector to the node semantic vector element by element to obtain the node representation with fused order information: (6) b3. Based on the sub-path granularity obtained in step S32, each sub-path is represented as an ordered sequence of nodes composed of multiple nodes with fused order information, arranged according to the actual driving order. The sub-path is then input into the Transformer encoder layer, where a multi-head attention mechanism is used to characterize the interaction relationships between different nodes, resulting in the context-enhanced representation of the sub-path, i.e., the sub-path semantic vector. (7) Where j represents the index number of the j-th sub-path, and its value ranges from 1 to the total number of sub-paths J; This represents the number of road nodes contained within the j-th sub-path; (j,1) to (j, ) represent the first to the last sub-path in the j-th sub-path. The index of the node; This represents the semantic vector of each node within the j-th sub-path. This semantic representation has incorporated multimodal information during the node-level encoding stage and has also been augmented with sequence information between nodes through positional encoding. `Transformer()` represents a Transformer encoder based on a multi-head self-attention mechanism used to model the context of the node sequence within the sub-path, capturing temporal dependencies and spatial relationships between nodes.

[0076] Through the above encoding process, the semantic representation of sub-paths can comprehensively reflect road changes, differences in functional area distribution, and potential traffic bottlenecks.

[0077] c. Based on the global path granularity constructed in step S33, the global path is represented as an ordered sequence of multiple sub-path semantic vectors according to the actual passage order; the global path is then semantically encoded to output the global path semantic representation. The global path semantic representation is composed of sub-path semantic sequences, thus transmitting and integrating multimodal semantic information of nodes and sub-paths at a hierarchical level to achieve semantic expression on a global scale. To characterize the cross-regional dependencies between different sub-paths and their comprehensive impact on overall travel time, a Transformer encoder based on a multi-head self-attention mechanism is introduced to model the global context of the sub-path sequences. The sub-path semantic representation sequence is input into the global path encoder, and positional encoding is introduced to represent the relative order of sub-paths within the entire path.

[0078] c1. Generate a corresponding position encoding vector for each sub-path semantic vector. ; The position encoding adopts the same sine-cosine position encoding method as the node position encoding, and its dimension is consistent with the dimension of the sub-path semantic vector.

[0079] c2. Add the position encoding vector to the sub-path semantic vector element-wise to obtain the sub-path representation with fused order information: (8) c3. The sub-path representations with fused sequence information are transformed using a Transformer encoder to obtain a context-enhanced sub-path representation sequence: (9) in, H represents the semantic representation vector of the j-th subpath, where j is the total number of subpaths. ) indicates a sequence of encoded context-enhanced representations.

[0080] The sequence is feature-transformed using a multi-layer Transformer encoding structure. In each layer, a multi-head self-attention mechanism is used to model the interaction relationships between different sub-paths, thereby capturing the traffic pattern associations, structural change dependencies, and potential congestion propagation effects of different sub-paths within the same path, resulting in a context-enhanced sub-path representation sequence. After completing the global context modeling, to characterize the different contributions of different sub-paths to the overall travel time, a semantic aggregation method based on an attention mechanism is introduced, as follows: c4. Assess the importance of the encoded context-enhanced representation sequence and calculate the attention score for each sub-path: (10) in, Let W and b represent the context-enhanced semantic representation of the j-th subpath, where W and b are learnable parameters. This represents the attention score of the corresponding sub-path, used to measure its contribution to the overall path semantics.

[0081] c5. Normalize the attention scores of the sub-paths to obtain the attention weights of the sub-paths: (11) in, This represents the attention weight of the j-th sub-path, reflecting its relative importance in the overall path.

[0082] c6. Perform weighted fusion of the sub-path representations according to the weights to obtain the global path semantic representation: (12) in, This represents the global semantic representation of the entire path. Through the above encoding process, a further abstraction from sub-path semantics to global path semantics is achieved, enabling the model to comprehensively model the overall structural features, environmental distribution, and travel state evolution of the path, providing a unified high-level semantic representation for subsequent text generation and travel time reasoning.

[0083] Multimodal semantic coding module training: By setting a time prediction module after the multimodal semantic coding module, the global path semantic representation output by the multimodal semantic coding module is input into the time prediction module; and using the actual travel time as the supervision signal, the parameters of the multimodal semantic coding module are optimized by minimizing the mean square error loss, resulting in a pre-trained multimodal semantic coding module; the time prediction module includes at least one fully connected layer and a regression output layer, used to map the global path semantic representation to the predicted travel time; Specifically, the trajectory dataset described in step S1 is divided into a training set and a validation set in a 7:3 ratio, and processed according to steps S1-S3. The processed training set data is used as training samples to train the multimodal semantic coding module. In the initial training phase, the model focuses on optimizing the multimodal semantic coding module (MSC) to ensure it can fully extract discriminative path semantic representations. During this phase, the MSC is not trained in isolation but is supervised and optimized by an auxiliary travel time prediction module. Specifically, the global path semantic representation output by the MSC is input to the travel time prediction module to obtain the corresponding travel time prediction results. The actual travel time is used as the supervision signal, and the parameters of the MSC are optimized using mean squared error loss. The mean squared error loss is: (13) Where D is the training path sample and P is the target path. It is the actual travel time corresponding to the target path P. It is the predicted travel time output by the time prediction module based on the global path semantic representation.

[0084] Calculate the actual travel time based on the start and end times of the route.

[0085] The training termination condition is as follows: the maximum number of training rounds is preset to 200 rounds, and an early stopping strategy is adopted. If the value of the mean squared error loss of the validation set is within the fluctuation range of ±0.5% of the mean squared error loss of those 10 rounds for 10 consecutive rounds, the training will be terminated early.

[0086] S5. Construct a text description generation module, which includes a semantic mapping submodule and a regression branch submodule; Input the node semantic vector, sub-path semantic vector, and global path semantic representation generated in step S4 into the pre-trained text description generation module, and output the corresponding label probabilities and continuous semantic attributes; generate the path natural language description text based on the label probabilities and continuous semantic attributes. Specifically, the node semantic vector, sub-path semantic vector, and global path semantic representation generated in step S4 are input into the corresponding semantic mapping module. The label probabilities are output according to the preset node semantic labels, sub-path semantic labels, and global path semantic labels, respectively. At the same time, a continuous attribute regression branch is used to predict the continuous semantic attributes of nodes, sub-paths, and global paths, and generate continuous attribute vectors. The path natural language description text is generated together based on the label probabilities and continuous semantic attributes.

[0087] Specifically as follows: After completing the multi-granularity semantic encoding of nodes, sub-paths, and global paths, semantic mapping is first performed using the multi-granularity semantic representation output by the aforementioned encoder. This ensures that the text description inherits the fused multimodal information from the path encoder. Specifically, the node semantic vectors are... Sub-path semantic vector and global path semantic representation The inputs are fed into the corresponding semantic mapping module. The semantic mapping module consists of a fully connected layer and a Softmax classification layer, and is used to map semantic vectors into semantic labels with traffic meaning.

[0088] The semantic mapping of a node semantic vector is represented as follows: (14) in, This represents the probability distribution of the semantic label of node i. and These are learnable parameters. The preset node semantic labels include one or more of the following: densely populated commercial areas, school areas, residential areas, office areas, transportation facility areas, and mixed-use areas.

[0089] At the sub-path level, sub-path semantics This indicates that the result obtained through mapping is: (15) in, This represents the probability distribution of the semantic labels of the sub-path corresponding to the j-th sub-path. This represents the semantic vector of the j-th sub-path. , These represent the learnable weight matrix and bias term in the sub-path semantic mapping, respectively. The preset sub-path semantic labels include one or more of the following: commercial area segments, school segments, and mixed-use area segments.

[0090] Global path semantic representation After mapping, we get: (16) in, This represents the global path semantic label probability distribution corresponding to the target path. , These represent the learnable weight matrix and bias term in the global path semantic mapping, respectively. The preset global path semantic labels include one or more of the following: high proportion of commercial areas, high proportion of school areas, and high proportion of mixed-use areas.

[0091] To avoid information loss during the mapping of high-dimensional semantic vectors to discrete labels, this invention further sets up a continuous attribute regression branch to predict the continuous semantic attributes of nodes, sub-paths, and global paths. The node-level regression representation is as follows: (17) in, This represents the continuous attribute vector corresponding to the node, where each dimension represents the node's continuous attributes such as intersection complexity, congestion risk, road accessibility, and POI mixing degree. , These represent the learnable weight matrix and bias term in the continuous attribute regression path of a node, respectively.

[0092] Similarly, continuous attribute regression branches are set for the sub-path semantic vector and the global path semantic representation respectively to obtain the sub-path continuous attribute vector. and global path continuous attribute vector .in, Let represent the continuous attribute vector corresponding to the j-th sub-path, where each dimension represents the sub-path's continuous attributes: average congestion risk, degree of road structure change, traffic stability, and average smoothness. This represents a continuous attribute vector corresponding to the entire target path, where each dimension represents the global continuous attributes of the proportion of high-risk sub-paths, the overall congestion risk, and the overall road complexity.

[0093] After semantic mapping is completed, the path natural language description is generated according to preset rules based on the label probability distribution of nodes, sub-paths and global paths and continuous attribute vectors output by the semantic mapping module.

[0094] The preset rules are as follows: At the node granularity level, the label with the highest probability in the node semantic label probability distribution is selected as the node dominant label, and a natural language description of the node is generated together with the node dominant label and the node continuous attributes.

[0095] Specifically, when the probability of a node's dominant label is greater than or equal to a preset node label threshold, a preset node natural language template corresponding to that dominant label is invoked to generate a basic description of the node. The preset node natural language template represents the correspondence between node semantic labels and descriptive statements. For example, "dense commercial area" corresponds to "commercial facilities are relatively concentrated around this road segment," "school area" corresponds to "this road segment is located in the area surrounding a school," "residential area" corresponds to "this road segment is located in a residential area," and "mixed-function area" corresponds to "the functional types around this road segment are relatively mixed." Other node labels are preset with corresponding natural language templates in the same manner.

[0096] When the probability of a node being the dominant label is less than a preset node label threshold, a basic description of the node "normal road conditions" is generated.

[0097] After generating the basic node description, the basic node description is supplemented based on the significance threshold of the node's continuous attributes.

[0098] Specifically, saliency thresholds are set for the continuous attributes of each node, including intersection complexity threshold, congestion risk threshold, road traffic flow threshold, and POI mixing threshold; relevant descriptions of the continuous attributes of nodes that are greater than or equal to the corresponding thresholds are added to the basic node description to generate a natural language description of the node. The relevant description of the node's continuous attributes is as follows: When the intersection complexity is greater than or equal to the intersection complexity threshold, the relevant description is "there are many nearby intersections"; When the congestion risk is greater than or equal to the congestion risk threshold, the relevant description is "there is some traffic pressure". When the road traffic flow is greater than or equal to the road traffic flow threshold, the relevant description is "traffic is relatively smooth"; When the degree of POI mixing is greater than or equal to the threshold of POI mixing, the relevant description is "the surrounding functional types are relatively mixed".

[0099] Specifically, when both congestion risk and road traffic flow in the continuous attributes of a node reach their corresponding significance thresholds, a description of the node's continuous attributes is generated according to a preset state combination template. This preset state combination template is used to simultaneously express the degree of risk and the current traffic status, for example, generating a description such as "the risk of localized congestion is relatively high, but the current traffic flow is relatively smooth."

[0100] To avoid redundancy caused by generating text node by node, consecutive road nodes with the same dominant label and continuous node attributes in the target path are merged into path segments. These path segments are then described as "front segment," "middle segment," or "rear segment" according to their relative position in the entire path, forming the final natural language description of the nodes. The relative position of a path segment in the entire path can be determined based on the position of the road nodes it contains in the ordered sequence of road nodes in the target path; for example, segments located in the first 1 / 3 are described as "front segment," segments located in the middle 1 / 3 are described as "middle segment," and segments located in the last 1 / 3 are described as "rear segment."

[0101] At the sub-path granularity level, the label with the highest probability in the sub-path semantic label probability distribution is selected as the dominant label of the sub-path. This dominant label is then combined with the continuous attributes of the sub-path and its attention weight in the global path to generate a natural language description of the sub-path. Specifically, when the probability of the dominant label of a sub-path is greater than or equal to a preset sub-path label threshold, a preset sub-path natural language template corresponding to that dominant label is invoked to generate a basic description of the sub-path. For example, "commercial area segment" corresponds to "this sub-path passes through a commercial area," and "mixed-function area segment" corresponds to "the functional types along this sub-path are relatively mixed." Other sub-path labels are preset with corresponding natural language templates in the same way.

[0102] When the probability of the dominant sub-path label is less than the preset sub-path label threshold, a basic sub-path description of "general sub-path" is generated.

[0103] After generating the basic description of the sub-path, the basic description of the sub-path is further supplemented based on the significance threshold of the continuous attributes of the sub-path and the attention weight.

[0104] Specifically, significance thresholds are set for the continuous attributes and attention weights of each sub-path, including the average congestion risk threshold, the average smoothness threshold, the road structure change threshold, the traffic stability threshold, and the attention weight threshold for the sub-path. The descriptions of the continuous attributes and attention weights of the sub-paths that are greater than or equal to the corresponding thresholds are added to the basic description of the sub-path to form a natural language description of the sub-path. The descriptions of the continuous attributes of the sub-paths and the attention weights are as follows: When the average congestion risk of a sub-path is greater than or equal to the average congestion risk threshold of the sub-path, the relevant description is "there is a certain amount of traffic pressure on the sub-path" or "the local congestion risk of the sub-path is relatively high". When the degree of change in road structure is greater than or equal to the threshold for road structure change, the relevant description is "the road structure of this sub-path has changed significantly". When the sub-path traffic stability is greater than or equal to the sub-path traffic stability threshold, the relevant description is "the traffic status of the sub-path is relatively stable". When the average smoothness of a sub-path is greater than or equal to the average smoothness threshold of the sub-path, the relevant description is "the current traffic status of this sub-path is relatively smooth"; When the attention weight is greater than or equal to the attention weight threshold, the relevant description is "this sub-path has a significant impact on the overall travel time".

[0105] Specifically, when both the average congestion risk and the average traffic flow of a sub-path are greater than or equal to the corresponding thresholds in the continuous attributes of the sub-path, a description of the continuous attributes of the sub-path is generated according to a preset state combination template. The preset state combination template is used to express both the degree of risk and the traffic flow simultaneously, for example, generating a description such as "the local congestion risk of this sub-path is relatively high, but the current traffic flow is relatively smooth."

[0106] At the global granularity level, the label with the highest probability in the global path semantic label probability distribution is selected as the global dominant label, and combined with global continuous attributes to generate a global path natural language description. Specifically, when the probability of the global dominant label is greater than or equal to a preset global label threshold, a preset global natural language template corresponding to that global dominant label is invoked to generate a global basic description. The preset global natural language template represents the correspondence between global path semantic labels and descriptive statements. For example, "high proportion of commercial areas" corresponds to "overall, the path passes through a relatively high proportion of commercial areas," and "high proportion of mixed functional areas" corresponds to "overall, the functional types along the path are relatively mixed." Other global path labels are preset with corresponding natural language templates in the same way.

[0107] When the probability of the global dominant label is less than the preset global label threshold, a global basic description of the "general path status" is generated. After generating the global basic description, the global basic description is supplemented based on the significance threshold of the global continuous attributes.

[0108] Specifically, a significance threshold is set for each global continuous attribute, including the overall congestion risk threshold, the high-risk sub-path proportion threshold, and the overall road complexity threshold; the descriptions of global continuous attributes that are greater than or equal to the corresponding thresholds are added to the global basic description to form a global path natural language description. The description of the global continuous attribute is as follows: When the overall congestion risk is greater than or equal to the overall congestion risk threshold, the relevant description is "the overall congestion risk is relatively high" or "the overall traffic pressure is relatively high". When the proportion of high-risk sub-paths is greater than or equal to the threshold for the proportion of high-risk sub-paths, the relevant description is "the proportion of high-risk road segments is relatively large"; When the overall road complexity is greater than or equal to the overall road complexity threshold, the relevant description is "the overall road structure is relatively complex".

[0109] Finally, the text generated by the node layer, sub-path layer, and global layer is concatenated according to the actual passage order to obtain a complete natural language description of the path. During the concatenation, the main semantic features of the first, middle, and last sections of the path are described first, then the key sub-paths are described, and finally the overall passage status of the entire path is given. Adjacent descriptions are connected by semicolons, and the end is marked with a period.

[0110] Finally, the natural language descriptions generated at the node granularity, sub-path granularity, and global path granularity are concatenated according to the actual travel order of the target path to obtain a complete path natural language description. During concatenation, the main semantic features of the beginning, middle, and end of the path are described first, then key sub-path descriptions are added at the corresponding positions, and finally, an overall travel status description of the entire path is given; adjacent descriptions are connected by semicolons, and a period is used at the end.

[0111] For example, when the dominant label of a node in the first segment of a path is "dense commercial area," and the intersection complexity and congestion risk in the node's continuous attributes both exceed the corresponding thresholds; the dominant label of a node in the middle segment of the path is "school area"; the dominant label of a sub-path is "mixed-use zone section," and the congestion risk in the sub-path's continuous attributes and the attention weight of the sub-path in the global path both exceed a threshold; and the dominant label of the global path is "high proportion of commercial areas," and the overall congestion risk and the proportion of high-risk sub-paths in the overall continuous attributes both exceed the corresponding thresholds, the generated natural language description of the complete path is as follows: "The initial part of the route passes through an area with a high concentration of commercial facilities and many nearby intersections, which puts some pressure on traffic. The middle part enters the area around schools. One section of the route passes through a mixed-use area, which also puts some pressure on traffic and has a significant impact on the overall travel time. Overall, the route passes through a high proportion of commercial areas, so the overall risk of congestion is high and the proportion of high-risk sections is relatively large." The pre-trained text description generation module: After completing the training of the multimodal semantic coding module, the parameters of the multimodal semantic coding module are fixed, and the semantic mapping submodule and regression branch submodule included in the text description generation module are further trained so that the semantic representations of nodes, sub-paths and global paths can be mapped into classification labels and continuous attribute values ​​with clear traffic semantics, providing a semantic foundation for subsequent text description generation.

[0112] The training samples used in the training phases of the semantic mapping submodule and the regression branch submodule are consistent with those used in the training phase of the multimodal semantic coding module. They are all constructed based on path structure data, POI data, remote sensing image data, and corresponding real travel time. The text description generation module is trained by processing the training set data as training samples according to steps S1-S4.

[0113] During training, the classification branch of the semantic mapping submodule uses cross-entropy loss. The regression branch uses mean squared error loss. The overall loss function for this stage is: (18) in, and These are the corresponding weight coefficients. Through the above training, the semantic mapping submodule and the regression branch submodule are able to accurately map multi-granular semantic representations into interpretable semantic labels and continuous attribute values.

[0114] The training termination condition is as follows: the maximum number of training rounds is preset to 200 rounds, and an early stopping strategy is adopted. If the value of the validation set loss is within the fluctuation range of ±0.5% of the average loss of those 10 rounds for 10 consecutive rounds, the training will be terminated early.

[0115] Wherein, the cross-entropy loss This is used to supervise the prediction results of the classification branch of the semantic mapping module for the classification semantic labels. The classification semantic labels include at least the following three types of supervision signals: (1) Functional semantic tags automatically generated based on POI data; (2) Semantic tags for travel status generated based on the ratio of actual travel time to theoretical travel time of sub-paths; (3) Semantic labels for global risk level determined based on the ratio of actual travel time to theoretical travel time for the entire route.

[0116] The semantic mapping module outputs the corresponding probability distributions for the three types of labels mentioned above, and calculates the cross-entropy loss with the automatically generated supervision labels. Cross-entropy loss The weighted summation of the three types of cross-entropy losses is as follows: + (19) in, These are the functional semantic labels, the traffic status semantic labels, and the cross-entropy loss corresponding to the traffic status semantic labels, respectively. The weighting coefficient is used. The set of categories for the semantic tags is a preset or configurable set, and can be expanded and set according to the application scenario.

[0117] The mean square error loss Continuous attribute values ​​are used to monitor the output of regression branch paths. These continuous attributes include one or more of intersection complexity, congestion risk, and the proportion of high-risk sub-paths. The intersection complexity monitoring value can be calculated and normalized based on the number of adjacent roads to a road node, the number of intersections, changes in road grade, or changes in the number of lanes. The congestion risk monitoring value can be normalized based on the ratio of actual travel time to theoretical travel time. The high-risk sub-path proportion monitoring value can be obtained based on the proportion of high-congestion-risk sub-paths to the total number of sub-paths. The mean squared error loss is calculated between the predicted continuous attribute values ​​of the regression branch path output and the automatically generated continuous attribute monitoring values.

[0118] The theoretical travel time, used to characterize the reference time required for a vehicle to traverse a corresponding sub-path or the entire route under free-flow conditions, can be calculated from road structure data. Specifically, it depends on the target path segment length *d* and the free-flow velocity. Calculate the theoretical travel time: (20) If the road speed limit information is included in the road data, then that road speed limit will be used as the free-flow speed; if the road speed limit information is missing, then a preset default speed will be selected as the free-flow speed based on the road level.

[0119] Through the above training strategy, the path semantic representation is made to have the ability to distinguish strongly correlated with the travel time, thereby ensuring that effective time information is still retained after being expressed by text in the reasoning stage, so that the large language model can make reasonable inferences based on this semantics.

[0120] It should be noted that the auxiliary time prediction module is used to constrain the temporal relevance during the training phase of the multimodal semantic encoding module, thereby enhancing the correlation between semantic representation and travel time. During the training of the semantic mapping module in this phase, the parameters of the multimodal semantic encoding module remain fixed, and the auxiliary time prediction module does not participate in the final inference output. In the inference phase, the auxiliary time prediction module no longer participates, and the final travel time is obtained by the large language model based on the path description text generated in step S5, thus improving the interpretability of the results while ensuring prediction accuracy.

[0121] S6. Construct a question template for travel time prediction; To ensure that the large language model can effectively reason about the generated text in relation to the travel time prediction task, this invention constructs a prompt template oriented towards travel time prediction during the prediction phase, clearly constraining the language model's role and reasoning objectives. Because the large language model possesses strong general language understanding and generation capabilities, without limiting its reasoning task, it may easily produce output results unrelated to travel time prediction. Therefore, by using a question template, the language model is explicitly instructed to analyze and reason about the path from the perspective of an expert in the field of travel time prediction.

[0122] The question template should clearly define the following information: define the reasoning role of the language model, ensuring the model focuses on travel time analysis; define the reasoning objective, which is to output the overall travel time required for the path based on path feature descriptions; and constrain the output format, meaning the generated reasoning result should be given in a clear time prediction format, with the option to provide the reasoning process. The question template can be represented as follows: Trip time prediction prompt template construction: Input: Character setting text, task description text, output format constraint text; Output: Trip time prediction question template; step: Step 1: Use the predefined role setting text as a template prefix. The role setting text is used to limit the large language model to perform analysis as an expert in the field of travel time prediction. Step 2: Add the task description text to the template. The task description text is used to instruct the large language model to predict the trip time based on the subsequent input path natural language description and trip start time. Step 3: Use the output format constraint text as a template suffix. The output format constraint text is used to limit the predicted travel time output by the large language model and can be selected to output the reasoning basis. Step 4: Combine the texts obtained from Steps 1 to 3 to form a travel time prediction question template.

[0123] For example, the travel time prediction question template can be: "You are an expert in the field of travel time prediction. Based on the input natural language description of the path and the travel start time, analyze the traffic conditions of the target path and predict the travel time of the path. The output should include the predicted travel time and a brief explanation of the reasoning." By constructing the above question template, the reasoning task of the large language model is limited to the scope of travel time prediction, thereby avoiding the generation of content unrelated to travel time prediction and improving the stability and reproducibility of the language model's prediction results.

[0124] S7. Input the path natural language description text generated in step S5, the question template for trip time prediction constructed in step S6, and the trip start time into the large language model to predict the trip time.

[0125] After receiving the above input, the large language model first performs semantic parsing on the natural language description of the route, extracting key information related to travel time, including road structure distribution, types of functional zones along the route, traffic complexity, and potential congestion risks. Then, combining the trip start time and prior knowledge in the traffic domain, it performs reasoning analysis on the overall traffic status of the route and generates corresponding travel time prediction results. This invention further constrains the output format of the language model, ensuring that the language model returns results according to a preset format. The output results include at least a travel time field and may also include a trip end time field and a reasoning basis field. When the output results do not meet the preset format, the results can be corrected by re-calling the language model or based on rules to ensure the standardization and consistency of the output results. For example: Trip end time: XX-XX-XX XX:XX:XX, Trip time: XX:XX:XX, Reasoning basis The prediction method differs from traditional direct numerical mapping methods based on regression models. Instead, it utilizes a language model to understand and logically infer path semantic information, achieving a semantic inference process from multimodal features to travel time. This enhances generalization ability, robustness, and interpretability in complex traffic environments. A complete prediction process is as follows: Figure 5 As shown.

[0126] The large language model is a pre-trained language model with natural language understanding, contextual reasoning, and text generation capabilities, including closed-source and open-source large language models.

Claims

1. A method for travel time prediction based on multimodal semantic coding and large language models, characterized in that, Includes the following steps: S1. Preprocess the raw GPS trajectory data of the target path to generate an ordered sequence of road nodes for the target path; S2. Construct multimodal data for each road node in the ordered road node sequence of the target path. The multimodal data includes path structure data, POI data, and remote sensing image data. S3. Based on the constructed multimodal data, perform multi-granularity modeling; the multi-granularity includes node granularity, sub-path granularity, and global path granularity; S4. Construct a multimodal semantic coding module, which includes a node semantic coding submodule, a sub-path semantic coding submodule, and a global path semantic coding submodule; the node semantic coding submodule includes a graph attention network; the sub-path semantic coding submodule includes a Transformer encoder based on a multi-head self-attention mechanism; and the global path semantic coding submodule includes a multi-layer Transformer coding structure. Input the node granularity, sub-path granularity, and global path granularity constructed in step S3 into the pre-trained multimodal semantic encoding module, and output the node semantic vector, sub-path semantic vector, and global path semantic representation respectively; S5. Construct a text description generation module, which includes a semantic mapping submodule and a regression branch submodule; Input the node semantic vector, sub-path semantic vector, and global path semantic representation generated in step S4 into the pre-trained text description generation module, and output the corresponding label probabilities and continuous semantic attributes; generate the path natural language description text based on the label probabilities and continuous semantic attributes. The semantic mapping submodule includes a fully connected layer and a Softmax classification layer; S6. Construct a question template for travel time prediction; S7. Input the path natural language description text generated in step S5, the question template for trip time prediction constructed in step S6, and the trip start time into the large language model to predict the trip time.

2. The travel time prediction method based on multimodal semantic coding and large language model according to claim 1, characterized in that, Step S2 includes: S21. Constructing Path Structure Data: Based on the OpenStreetMap open-source geodatabase, extract the road structure attributes corresponding to each road node in the ordered sequence of road nodes of the target path, and encode the road structure attributes into a road attribute vector. This is used to construct the path structure data; The road attribute vector Each dimension represents one or more of the following: road length, number of lanes, and road grade. All attribute features have been normalized and mapped to a unified dimensional space. The path structure data is represented as follows: （1） in, This represents the path structure data of the target path P. This represents the road attribute vector corresponding to the i-th road node; N represents the number of road nodes in the target path; S22. Constructing POI Data: Based on the OpenStreetMap open-source geodatabase, using road nodes as the center, retrieve POI information around the nodes within a preset spatial radius, obtain the corresponding POI identifiers and their functional categories, and form a POI set. , is represented as: （2） in, For road nodes This is a collection of all Points of Interest (POIs) retrieved within a radius r centered at a given point. Each element contains the POI's coordinates and function category. Represents the latitude and longitude coordinates of the j-th POI, used to identify the spatial geographic location of the POI; The label represents the functional category of the j-th POI; dis() represents the spatial distance function; r is the preset spatial buffer radius, used to define the spatial neighborhood range centered on the road node; This represents the spatial location of the i-th road node; S23. Constructing remote sensing image data: Based on the latitude and longitude coordinates of all road nodes in the ordered road node sequence, calculate the minimum bounding rectangle area covering the entire target path, and crop the target path remote sensing image area from the original remote sensing image based on the minimum bounding rectangle area; and form node remote sensing image data from the target path remote sensing image area after cropping.

3. The travel time prediction method based on multimodal semantic coding and large language model according to claim 1, characterized in that, Step S3 is as follows: S31. Using the spatial coordinates of road nodes as a unified index key, spatially align the path structure data, POI data, and remote sensing image data of road nodes to form a node granularity that includes multimodal data. S32. Based on the semantic changes of node granularity, divide the node granularity into several semantically consistent sub-path units to form sub-path granularity. S33. The sequence structure formed by the sub-path granularities divided in step S32 according to the passage order is taken as the global path granularity.

4. The travel time prediction method based on multimodal semantic coding and large language model according to claim 3, characterized in that, In step S32, based on the changes in road structure attributes, POI functional category distribution, and traffic status of the spatially aligned road nodes in the ordered road node sequence, the multimodal data at the node granularity is divided to form several semantically consistent sub-path units.

5. The method for travel time prediction based on multimodal semantic coding and a large language model according to claim 4, characterized in that, In step S32, if an adjacent road node meets the corresponding preset threshold condition in any dimension of road structure attribute, POI functional category distribution change, and historical traffic status, then the adjacent road node is taken as the sub-path boundary, the current node is the previous sub-path boundary, and the next node is the current sub-path boundary, thereby constructing a sub-path sequence with consistent structure and semantics; if three or more consecutive road nodes meet the corresponding preset threshold condition in any dimension, then the optimal node is selected from the road nodes and this node is taken as the previous sub-path boundary. Use its next node as the boundary of the current sub-path; The corresponding preset threshold conditions include: 1) Judgment based on changes in road structure attributes: Calculate the road structure difference degree based on the road attribute vectors corresponding to adjacent road nodes. When the road structure difference degree between adjacent road nodes is greater than the preset road structure difference threshold, it is determined that the road structure between adjacent road nodes has changed significantly, and the position of the adjacent road node is used as the sub-path division boundary. The road structure difference degree can be obtained by calculating the Euclidean distance between the normalized road attribute vectors of adjacent road nodes. 2) Based on the distribution change of POI functional categories, when the cosine similarity of the node vectors of adjacent road nodes is less than 0.6, the position of the adjacent road node is used as the boundary of the sub-path division; the cosine similarity is calculated based on the vector dot product and norm. 3) Determine based on historical traffic status, that is, calculate the average historical traffic speed of each road node based on the ordered road node sequence and the timestamp information corresponding to the road node, and then calculate the relative rate of change of historical traffic speed of adjacent road nodes; if the value is greater than 0.3, then the position of the adjacent road node is used as the sub-path division boundary; The optimal node refers to the road node with the highest score selected by normalizing and weighting the road structure difference, 1-cosine similarity, and relative change rate of historical traffic speed.

6. The method for travel time prediction based on multimodal semantic coding and a large language model according to claim 1, characterized in that, Step S4 involves inputting the node granularity, sub-path granularity, and global path granularity constructed in step S3 into the pre-trained multimodal semantic encoding module, which outputs node semantic vectors, sub-path semantic vectors, and global path semantic representations, respectively. a. Independently encode the multimodal data at the node granularity in step S31 to obtain multimodal features. Input the multimodal features into the node semantic encoding submodule and output the node semantic vector. The node semantic encoding submodule includes a multimodal node feature fusion structure and a graph attention network. b. Based on the sub-path granularity constructed in step S32, each sub-path is represented as an ordered sequence of nodes consisting of multiple node semantic vectors in the actual driving order; the sub-path passes through the sub-path semantic encoding sub-module to generate sub-path semantic vectors. c. Based on the global path granularity constructed in step S33, the global path is represented as an ordered sequence of multiple sub-path semantic vectors according to the actual passage order; the global path is then encoded using global path semantics to output the global path semantic representation.

7. The method for travel time prediction based on multimodal semantic coding and a large language model according to claim 6, characterized in that, Step a specifically includes: a1. Independently encode the multimodal data at the node granularity to generate road structure features for road nodes. POI semantic features of road nodes Remote sensing visual features of road nodes ; a2. Introduce adaptive weighted fusion to analyze the road structure features of road nodes. POI semantic features of road nodes Remote sensing visual features of road nodes The fusion process is performed to generate a multimodal fusion representation of the road nodes: The obtained road structure features of the road nodes POI semantic features of road nodes Remote sensing visual features of road nodes Weight coefficients for the three types of features are generated using a learnable weight mapping function. , and The sum of the weights is then ensured to be 1 through normalization; subsequently, the three types of modal features are weighted and combined to obtain the node multimodal fusion representation: （4） in, Let represent the multimodal fusion representation of the i-th road node. , , These represent the road structure features, POI semantic features, and remote sensing visual features of the i-th road node, respectively. a3. Input the generated multimodal fusion representation of the road nodes into the graph attention network, and generate node semantic vectors that fuse spatial structure information and environmental semantic information by adaptively learning the influence weights between neighboring nodes: （5） in, Represents the semantic vector of a node; Step b specifically involves: b1. Introduce location encoding information into the node semantic vector to represent the relative order relationship between road nodes, and generate the corresponding location encoding vector. ; b2. Add the position encoding vector to the node semantic vector element by element to obtain the node representation with fused order information: （6） b3. Based on the sub-path granularity obtained in step S32, each sub-path is represented as an ordered sequence of nodes composed of multiple nodes with fused order information, arranged according to the actual driving order. The sub-path is then input into the Transformer encoder layer, where a multi-head attention mechanism is used to characterize the interaction relationships between different nodes, resulting in the context-enhanced representation of the sub-path, i.e., the sub-path semantic vector. （7） Where j represents the index number of the j-th sub-path, and its value ranges from 1 to the total number of sub-paths J; This represents the number of road nodes contained within the j-th sub-path; (j,1) to (j, ) represent the first to the last sub-path in the j-th sub-path. The index of the node; This represents the semantic vector of each node within the j-th sub-path; Transformer() represents a Transformer encoder based on a multi-head self-attention mechanism, used to perform context modeling on the node sequence within the sub-path, capturing the temporal dependencies and spatial associations between nodes; Step c specifically includes: c1. Generate a corresponding position encoding vector for each sub-path semantic vector. ; c2. Add the position encoding vector to the sub-path semantic vector element-wise to obtain the sub-path representation with fused order information: （8） c3. The sub-path representations with fused sequence information are transformed using a Transformer encoder to obtain a context-enhanced sub-path representation sequence: （9） in, H represents the semantic representation vector of the j-th subpath, where j is the total number of subpaths. ) represents the encoded context-enhanced representation sequence; c4. Assess the importance of the encoded context-enhanced representation sequence and calculate the attention score for each sub-path: （10） in, Let W and b represent the context-enhanced semantic representation of the j-th subpath, where W and b are learnable parameters. This represents the attention score for the corresponding sub-path; c5. Normalize the attention scores of the sub-paths to obtain the attention weights of the sub-paths: （11） in, This represents the attention weight of the j-th sub-path, reflecting its relative importance in the overall path; c6. Perform weighted fusion of the sub-path representations according to the weights to obtain the global path semantic representation: （12） in, This represents the global semantic representation of the entire path.

8. The method for travel time prediction based on multimodal semantic coding and large language model according to claim 1, characterized in that, In step S5, the node semantic vector, sub-path semantic vector, and global path semantic representation generated in step S4 are input to the corresponding semantic mapping module. The label probabilities are output according to the preset node semantic labels, sub-path semantic labels, and global path semantic labels, respectively. At the same time, a continuous attribute regression branch is used to predict the continuous semantic attributes of nodes, sub-paths, and global paths, and generate continuous attribute vectors. The path natural language description text is generated together with the label probabilities and continuous attribute vectors.

9. The method for travel time prediction based on multimodal semantic coding and a large language model according to claim 8, characterized in that, The node semantic tags include one or more of the following: densely populated commercial areas, school areas, residential areas, office areas, transportation facility areas, and mixed-use areas; The semantic mapping of a node semantic vector is represented as follows: （13） in, This represents the semantic label probability distribution of node i. and These are learnable parameters; The sub-path semantic tags include one or more of the following: commercial area road segments, school road segments, and mixed-use area road segments; The sub-path semantic vector is obtained through mapping: （14） in, This represents the semantic tag probability distribution corresponding to the j-th sub-path. This represents the semantic vector of the j-th sub-path. , These represent the learnable weight matrix and bias term in the sub-path semantic mapping, respectively; The global path semantic tags include one or more of the following: high proportion of commercial areas, high proportion of school areas, and high proportion of mixed functional areas. Global path semantic representation After mapping, we get: （15） in, This represents the global path semantic label probability distribution corresponding to the target path. , These represent the learnable weight matrix and bias term in the global path semantic mapping, respectively; The node-level regression is represented as: （16） in, Let represent the continuous attribute vector corresponding to the i-th road node, where each dimension represents the node's continuous attributes such as intersection complexity, congestion risk, road traffic flow, and POI mixing degree. , These represent the learnable weight matrix and bias term in the continuous attribute regression path of a node, respectively. Continuous attribute vectors are obtained by performing continuous attribute regression branch prediction on both the sub-path and the global path. and ;in, Let represent the continuous attribute vector corresponding to the j-th sub-path, where each dimension represents the sub-path's continuous attributes: average congestion risk, degree of road structure change, traffic stability, and average smoothness. This represents a continuous attribute vector corresponding to the entire target path, where each dimension represents a global continuous attribute such as the proportion of high-risk sub-paths, the overall congestion risk, and the overall road complexity. After semantic mapping is completed, the path natural language description is generated according to preset rules based on the label probability distribution and continuous attribute vector of nodes, sub-paths and global paths output by the semantic mapping module. The preset rules are as follows: At the node granularity level, the label with the highest probability in the node semantic label probability distribution is selected as the node dominant label, and a natural language description of the node is generated based on the node dominant label and the node continuous attributes. At the sub-path granularity level, the label with the highest probability in the sub-path semantic label probability distribution is selected as the dominant label of the sub-path, and the sub-path natural language description is generated by combining the continuous attributes of the sub-path and its attention weight in the global path. At the global granularity level, the label with the highest probability in the global path semantic label probability distribution is selected as the global dominant label, and combined with global continuous attributes to generate a global path natural language description; The natural language descriptions generated at the node granularity, sub-path granularity, and global path granularity are concatenated according to the actual passage order of the target path to obtain a complete path natural language description.

10. The method for travel time prediction based on multimodal semantic coding and a large language model according to claim 1, characterized in that, The question template for trip time prediction described in step S6 includes the reasoning role, reasoning goal, output format, and reasoning process of the language model.

Images