Passing time estimation method, system and equipment combined with large model, and medium

Abstract

The invention discloses a passing time estimation method, system and device combined with a large model and a medium, and the method comprises the steps: carrying out the embedding processing of the current latitude and longitude, the current pose, the front real-time image and the current precise time of a vehicle, and obtaining first embedded data; inputting the first embedded data into a cross attention encoder for processing to obtain optimized feature data; performing embedding processing on the current longitude and latitude, the current pose, the front real-time picture and the current accurate time through a hard cue word method to obtain second embedded data; splicing the second embedded data and the optimized feature data, and inputting the spliced feature data into a large language model to obtain a passage time predicted value; an interpretable cause of the transit time predicted value is generated through a large language model based on the front real-time picture and the transit time predicted value. According to the method, the problems that a traditional mode depends on static information and lacks multi-modal fusion and interpretability can be solved, and the passing time estimation precision and the result credibility are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a passing time estimation method, system, device and medium combined with a large model. BACKGROUND

[0002] In the field scene, accurate estimation of passing time is crucial for off-road navigation, unmanned equipment scheduling and field autonomous driving systems. Traditional passing time estimation relies on static information such as terrain type and route length, making it difficult to incorporate real-time environmental changes such as vehicle pose, real-time image, weather and lighting conditions, and current accurate time. With the development of field robots, intelligent inspection equipment and unmanned vehicles, there is an urgent need for more accurate and dynamic passing time estimation methods to improve driving efficiency and safety in complex natural environments. At the same time, existing methods lack sufficient explanation ability when giving predictions, limiting the credibility and usability of the prediction results.

[0003] In the prior art, the invention patent with publication number CN115019507A discloses a real-time estimation method for urban road network travel time reliability, which collects real-time traffic detection data, calculates road segment delay time and travel time, and then establishes a probability distribution model of delay travel time ratio to realize real-time estimation of road network travel time reliability. The invention patent with publication number CN119783036A discloses a passing time estimation method based on deep learning in complex environments, which constructs a deep learning model based on Transformer for highland and mountainous environments, and uses GPS coordinates, slope, turning angle and weather category as feature data for passing time prediction.

[0004] However, the existing technology still has obvious deficiencies. The solution with publication number CN115019507A mainly relies on traditional traffic flow models, the data source is relatively single, and it is unable to effectively fuse multi-modal real-time information such as vehicle pose and real-time vision, and it does not provide explainability of the prediction results. The solution with publication number CN119783036A introduces an advanced model, but it is mainly designed for specific complex environments and has not been optimized for the particularity of field environments. At the same time, it does not fully utilize the powerful semantic understanding and reasoning ability of large language models, and lacks an explanation mechanism for the prediction results. The existing technology methods have not been able to realize deep feature fusion and correlation analysis of multi-source heterogeneous data, and cannot provide passing time estimation with high precision and explainability.

[0005] Therefore, there is an urgent need to provide a technical solution to solve the above problems. SUMMARY

[0006] To solve the above technical problems, the present application provides a passing time estimation method, system, device and medium combined with a large model.

[0007] In a first aspect, the present application provides a traffic time estimation method combined with a large model, and the technical scheme of the method is as follows: Embedding the current latitude and longitude information, the current pose information, the front real-time picture and the current accurate time of the target vehicle to obtain first embedding data; Inputting the first embedding data into a cross attention encoder for capturing the correlation between multiple data to obtain optimized feature data; Embedding the current latitude and longitude information, the current pose information, the front real-time picture and the current accurate time by a hard prompt word method to obtain second embedding data; Splicing the second embedding data and the optimized feature data to obtain spliced feature data, and inputting the spliced feature data into a pre-trained large language model for processing to obtain a traffic time prediction value; Based on the front real-time picture and the traffic time prediction value, generating an interpretable reason for the traffic time prediction value by the large language model.

[0008] The traffic time estimation method combined with a large model has the following beneficial effects: The method of the present application embeds the latitude and longitude, pose, front picture and accurate time of the vehicle, captures the correlation features of multiple source data by using a cross attention encoder, and combines a hard prompt word embedding to generate a prediction value and an interpretable reason by a pre-trained large language model, thereby solving the problems of relying on static information, lacking multi-modal fusion and interpretability in traditional methods, and improving the traffic time estimation accuracy and result reliability.

[0009] On the basis of the above-mentioned scheme, the traffic time estimation method combined with a large model of the present application can also be improved as follows.

[0010] In an optional manner, the step of embedding the current latitude and longitude information, the current pose information, the front real-time picture and the current accurate time of the target vehicle to obtain first embedding data comprises: Inputting the current latitude and longitude information into a first embedding layer for processing to obtain a latitude and longitude embedding vector; Inputting the current accurate time into a second embedding layer for processing to obtain a time embedding vector; Inputting the current pose information into a third embedding layer for processing to obtain a pose embedding vector; Inputting the front real-time picture into a visual embedding layer for processing to obtain a picture embedding vector; The latitude and longitude embedding vector, the time embedding vector, the pose embedding vector and the picture embedding vector are spliced in a feature dimension to obtain the first embedding data.

[0011] The beneficial effect of the above optional mode is that: by further inputting latitude and longitude, time, pose and picture into the special embedding layer respectively for processing, specific embedding vectors are obtained, fine feature extraction of multi-source heterogeneous data is realized, and the feature expression capability of subsequent cross-attention encoding is improved.

[0012] In an optional mode, the step of inputting the first embedding data into a cross-attention encoder for capturing the association between multiple data to obtain optimized feature data includes: The first embedding data is subjected to attention calculation through the multi-head attention mechanism in the cross-attention encoder to generate attention weighted features; The attention weighted features are subjected to linear transformation through the linear layer in the cross-attention encoder to obtain the optimized feature data.

[0013] The beneficial effect of the above optional mode is that: by further inputting latitude and longitude, time, pose and picture into the special embedding layer respectively for processing, specific embedding vectors are obtained, fine feature extraction of multi-source heterogeneous data is realized, and the feature expression capability of subsequent cross-attention encoding is improved.

[0014] In an optional mode, the step of embedding the current latitude and longitude information, the current pose information, the front real-time picture and the current accurate time through the hard prompt word method to obtain second embedding data includes: A hard prompt word template containing the current latitude and longitude information, the current pose information, the front real-time picture and the current accurate time is constructed; The hard prompt word template is input into a prompt word embedding layer for processing to obtain the second embedding data.

[0015] The beneficial effect of the above optional mode is that: by further constructing a hard prompt word template and inputting it into a prompt word embedding layer for processing, structured text information is converted into semantic embedding, effective alignment of original data and large language model understanding form is realized, and the semantic integrity of model input is enhanced.

[0016] In an optional mode, the step of splicing the second embedding data and the optimized feature data to obtain spliced feature data, and inputting the spliced feature data into a pre-trained large language model for processing to obtain a traffic time prediction value includes: The second embedding data is spliced with the optimization feature data in a spatial dimension to obtain the spliced feature data; The spliced feature data is subjected to context semantic modeling and feature conversion through a Transformer network layer in the pre-trained large language model to obtain intermediate feature data; The intermediate feature data is sequentially processed through an output aggregation layer and a projection layer in the pre-trained large language model to obtain the travel time prediction value.

[0017] The beneficial effects of the above optional mode are that the second embedding data is further spliced with the optimization feature data in the spatial dimension, and then subjected to context modeling through the Transformer network layer of the large language model, and then processed through the output aggregation layer and the projection layer, so as to realize feature deep fusion and accurate time prediction.

[0018] In an optional mode, the step of generating an interpretable reason for the travel time prediction value through the large language model based on the front real-time picture and the travel time prediction value comprises: The front real-time picture is subjected to feature extraction through a pre-trained visual Transformer model to obtain picture high-level features; The picture high-level features and the travel time prediction value are subjected to semantic understanding and reasoning analysis through the pre-trained large language model in combination with pre-set text prompt words to generate the interpretable reason for the travel time prediction value.

[0019] The beneficial effects of the above optional mode are that the front real-time picture high-level features are further extracted through the visual Transformer model, and the picture features and the prediction value are subjected to semantic reasoning through the large language model to generate the interpretable reason in the form of natural language, thereby enhancing the understandability and credibility of the prediction result.

[0020] In an optional mode, the method further comprises: The current latitude and longitude information of the target vehicle is acquired through a GPS carried by the target vehicle; The current accurate time of the target vehicle is acquired through a vehicle-mounted system clock or a GPS of the target vehicle; The current pose information of the target vehicle is acquired through an inertial measurement unit carried by the target vehicle; The front real-time picture of the target vehicle is acquired through a camera installed at the front of the target vehicle.

[0021] The advantages of adopting the above-mentioned optional methods are as follows: further latitude and longitude information is obtained through GPS, accurate time is obtained through vehicle clock or GPS, pose information is obtained through inertial measurement unit, and real-time images are obtained through front camera, ensuring accurate collection and synchronization of multimodal data sources, and providing a reliable data foundation for subsequent processing.

[0022] Secondly, the present invention provides a travel time estimation system that combines a large model, the technical solution of which is as follows: The first embedding module is used to embed the target vehicle's current latitude and longitude information, current pose information, real-time image ahead, and current precise time to obtain the first embedded data; The encoding processing module is used to input the first embedded data into a cross-attention encoder for capturing the correlation between multiple data to obtain optimized feature data. The second embedding module is used to embed the current latitude and longitude information, the current pose information, the real-time image in front, and the current precise time using a hard-suggestion word method to obtain the second embedded data; The splicing prediction module is used to splice the second embedded data with the optimized feature data to obtain spliced ​​feature data, and input the spliced ​​feature data into a pre-trained large language model for processing to obtain a travel time prediction value; The prediction and explanation module is used to generate an explainable reason for the predicted travel time based on the real-time image ahead and the predicted travel time using the large language model.

[0023] The beneficial effects of the travel time estimation system combining a large model according to the present invention are as follows: The system of this invention embeds vehicle latitude and longitude, pose, forward view, and precise time, uses a cross-attention encoder to capture multi-source data association features, and combines hard cue word embedding with a pre-trained large language model to generate predicted values ​​and explainable reasons. This solves the problems of traditional methods relying on static information, lacking multimodal fusion, and lacking interpretability, and improves the accuracy and reliability of travel time estimation results.

[0024] Thirdly, the technical solution of an electronic device according to the present invention is as follows: It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the travel time estimation method combining a large model as described in this invention.

[0025] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows: The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the travel time estimation method combining a large model as described in this invention.

[0026] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0027] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of the travel time estimation method combining a large model according to the present invention. Figure 2 This is a diagram of the overall algorithm framework; Figure 3 This is a schematic diagram of an embodiment of the travel time estimation system combining a large model according to the present invention. Figure 4 This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0028] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0029] Figure 1 This diagram illustrates a flowchart of an embodiment of a large-model-based travel time estimation method provided by the present invention. This method can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the large-model-based travel time estimation method by having its processor call computer-readable instructions stored in memory. Figure 1 As shown, it includes the following steps: S1. Embed the target vehicle's current latitude and longitude information, current pose information, real-time image ahead, and current precise time to obtain the first embedded data.

[0030] The target vehicle refers to a motor vehicle traveling in a road environment whose travel time needs to be estimated; for example, a smart electric vehicle traveling on a road. Current latitude and longitude information refers to the longitude and latitude coordinates of the target vehicle's current geographical location; for example, longitude 120.000 and latitude 30.000 indicates the vehicle is on a main road. Current pose information refers to the spatial attitude parameters of the target vehicle at the current moment; for example, a pitch angle of 1.5 degrees, a roll angle of 0.5 degrees, and a yaw angle of 90.0 degrees indicates the vehicle is turning right. Real-time forward view refers to continuous environmental images captured by the target vehicle's forward-facing camera; for example, video frames captured in real-time by a camera mounted on the front of the vehicle, including the road ahead, other vehicles, and traffic signals. Current precise time refers to a complete timestamp including year, month, day, hour, minute, and second; for example, August 15, 2025, 10:30:00, indicating the specific time of data collection. The first embedded data refers to the feature representation after embedding and splicing multiple source raw data; for example, a 2048-dimensional feature vector formed by splicing latitude, longitude, time, pose and image data after embedding them respectively.

[0031] S2. The first embedded data is input into a cross-attention encoder for capturing the correlation between multiple data to obtain optimized feature data.

[0032] The cross-attention encoder refers to a neural network module that models the correlation of multimodal data through an attention mechanism; for example, an encoder structure that uses a multi-head attention mechanism to calculate the relationship between latitude, longitude, time, pose, and image features. Optimized feature data refers to feature representations that enhance correlation after processing by the cross-attention encoder; for example, a 1024-dimensional feature vector containing information on the correlation between road congestion and vehicle steering behavior.

[0033] S3. The current latitude and longitude information, the current pose information, the real-time image in front, and the current precise time are embedded using the hard cue word method to obtain the second embedded data.

[0034] The hard cue word method refers to the technique of converting data into a model input format using structured templates; for example, constructing a text template such as "current location {longitude 120.000, latitude 30.000}, time {2025-08-15 10:30:00}". The second embedded data refers to the feature representation obtained by processing the hard cue word template through an embedding layer; for example, the 768-dimensional text feature vector output after inputting the hard cue word template into the embedding layer.

[0035] S4. The second embedded data and the optimized feature data are concatenated to obtain concatenated feature data, and the concatenated feature data is input into a pre-trained large language model for processing to obtain the travel time prediction value.

[0036] Among these, concatenated feature data refers to fused data obtained by concatenating different feature sources in terms of dimensions; for example, concatenating 1024-dimensional optimized features with 768-dimensional prompt word features to obtain 1792-dimensional fused features. Pre-trained large language models refer to deep learning models pre-trained on large-scale text; for example, using a model with the GPT-4 architecture as a base and loading pre-trained weight parameters. Travel time predictions refer to the model's estimate of the time required to travel from the current location to the destination; for example, predicting that it will take 30 minutes to travel from the current main road location to a commercial area.

[0037] S5. Based on the real-time image ahead and the predicted travel time, generate an explainable reason for the predicted travel time using the large language model.

[0038] Explainable reasons refer to the textual descriptions on which the prediction results are based; for example, the output of a large language model: "Due to road construction ahead, traffic is slow, and the estimated travel time will increase by 10 minutes."

[0039] The technical solution of this embodiment embeds vehicle latitude and longitude, pose, forward view, and precise time, uses a cross-attention encoder to capture multi-source data association features, and combines hard cue word embedding with a pre-trained large language model to generate predicted values ​​and explainable reasons. This solves the problems of traditional methods relying on static information, lacking multimodal fusion, and lacking interpretability, and improves the accuracy and reliability of travel time estimation results.

[0040] In one alternative approach, S1 specifically includes: The current latitude and longitude information is input into the first embedding layer for processing to obtain the latitude and longitude embedding vector.

[0041] The first embedding layer refers to a neural network layer that processes latitude and longitude data; for example, a fully connected layer that maps latitude and longitude coordinates into a 128-dimensional vector. The latitude and longitude embedding vector refers to the numerical representation of latitude and longitude data after transformation by the embedding layer; for example, longitude 120.000 and latitude 30.000 are converted into a 128-dimensional floating-point vector.

[0042] The current precise time is input into the second embedding layer for processing to obtain a time embedding vector.

[0043] The second embedding layer refers to a neural network layer that processes time data; for example, a temporal embedding layer that converts timestamps into 64-dimensional vectors. The time embedding vector refers to the numerical representation of time data after transformation by the embedding layer; for example, the timestamp 2025-08-15 10:30:00 is converted into a 64-dimensional periodic feature vector.

[0044] The current pose information is input into the third embedding layer for processing to obtain the pose embedding vector.

[0045] The third embedding layer refers to the neural network layer that processes vehicle pose data; for example, a fully connected layer that converts pitch, roll, and yaw angles into 96-dimensional vectors. The pose embedding vector refers to the numerical representation of the pose data after conversion by the embedding layer; for example, a pitch angle of 1.5 degrees, a roll angle of 0.5 degrees, and a yaw angle of 90.0 degrees are converted into 96-dimensional motion feature vectors.

[0046] The real-time image from the front is input into the visual embedding layer for processing to obtain the image embedding vector.

[0047] The visual embedding layer refers to a convolutional neural network layer that processes image data; for example, a network structure that uses ResNet-50 convolutional layers to extract 1024-dimensional visual features from the image. The image embedding vector refers to the feature representation of the image data after processing by the visual embedding layer; for example, a 1024-dimensional visual feature vector extracted from the real-time image in front of you by a convolutional network.

[0048] The latitude and longitude embedding vector, the time embedding vector, the pose embedding vector, and the image embedding vector are concatenated along the feature dimension to obtain the first embedding data.

[0049] Among them, the feature dimension refers to the arrangement direction of the numerical elements in the feature vector; for example, the latitude and longitude embedding vector (128 dimensions), the time embedding vector (64 dimensions), the pose embedding vector (96 dimensions), and the image embedding vector (1024 dimensions) are spliced ​​together in the feature dimension.

[0050] In the above-mentioned optional methods, latitude and longitude, time, pose, and image are further processed by inputting them into a dedicated embedding layer to obtain targeted embedding vectors, thereby realizing refined feature extraction of multi-source heterogeneous data and improving the feature expression capability of subsequent cross-attention encoding.

[0051] In one alternative approach, S2 specifically includes: The first embedded data is subjected to attention calculation through the multi-head attention mechanism in the cross-attention encoder to generate attention-weighted features.

[0052] Multi-head attention mechanisms refer to neural network components that perform multiple attention calculations in parallel; for example, using eight attention heads in a cross-attention encoder to simultaneously calculate the correlation of different feature subspaces. Attention calculation refers to the process of evaluating feature importance through a query-key mechanism; for example, calculating the attention weight distribution between vehicle pose features and the features of the scene ahead. Attention-weighted features refer to feature representations adjusted by attention weights; for example, a fused feature vector after applying a weight of 0.8 to the scene features and a weight of 0.2 to the latitude and longitude features.

[0053] Specifically, the first embedded data is input into the multi-head attention mechanism in the cross-attention encoder. The first embedded data is converted into query vector, key vector and value vector in parallel by multiple attention heads in the multi-head attention mechanism. The dot product of the query vector and the key vector is calculated and the softmax function is applied to obtain the attention weights. The attention weights are used to perform a weighted summation of the value vectors to generate attention-weighted features.

[0054] The attention-weighted features are linearly transformed through the linear layer in the cross-attention encoder to obtain the optimized feature data.

[0055] In this context, a linear layer refers to a neural network layer that performs linear transformations; for example, a fully connected layer that transforms 1024-dimensional attention-weighted features into 512-dimensional features using a weight matrix. A linear transformation refers to numerical operations using the weight matrix and bias vector; for example, performing matrix multiplication on 1024-dimensional input features using a 512×1024-dimensional weight matrix.

[0056] Specifically, attention-weighted features are input into the linear layer of the cross-attention encoder. The linear layer performs matrix multiplication and vector addition operations on the attention-weighted features using pre-trained weight matrices and bias vectors to complete the linear transformation, thereby generating optimized feature data.

[0057] In the above-mentioned optional approach, attention calculation is further performed on the first embedded data through the multi-head attention mechanism in the cross-attention encoder. Combined with linear layer transformation, the complex correlation between multimodal data is effectively captured, and more discriminative optimized feature data is generated.

[0058] In one alternative approach, S3 specifically includes: Construct a hard-coded prompt template that includes the current latitude and longitude information, the current pose information, the real-time view ahead, and the current precise time.

[0059] Hard cue word templates refer to text frames filled with structured data; for example, fixed format strings such as "latitude and longitude {lon}, {lat} time {time} pose {pitch}, {roll}, {yaw} image {image}".

[0060] Specifically, according to the preset text format, the numerical value of the current latitude and longitude information, the parameters of the current pose information, the image identifier of the real-time image in front, and the timestamp of the current precise time are filled into the corresponding template positions to form a structured text string as a hard prompt word template.

[0061] The hard prompt word template is input into the prompt word embedding layer for processing to obtain the second embedded data.

[0062] The cue word embedding layer refers to a neural network layer that converts text templates into vector representations; for example, an embedding layer that converts text tags in hard cue word templates into 768-dimensional word vectors.

[0063] Specifically, the hard cue word template is input into the cue word embedding layer, and the text content in the hard cue word template is converted into a numerical vector representation through the pre-trained embedding matrix in the cue word embedding layer to generate the second embedding data.

[0064] In the above optional methods, by constructing hard cue word templates and inputting them into the cue word embedding layer, the structured text information is transformed into semantic embedding, which effectively aligns the original data with the understanding form of the large language model and enhances the semantic integrity of the model input.

[0065] In one alternative approach, S4 specifically includes: The second embedded data and the optimized feature data are concatenated in the spatial dimension to obtain the concatenated feature data.

[0066] Spatial dimension refers to the dimension in the feature tensor that represents spatial relationships; for example, concatenating 768-dimensional second embedding data with 512-dimensional optimized feature data in terms of feature dimension yields 1280-dimensional features.

[0067] Specifically, the feature vector corresponding to the second embedded data and the feature vector corresponding to the optimized feature data are connected in the feature dimension direction to form a new fused feature vector, which is the spliced ​​feature data.

[0068] Intermediate feature data is obtained by performing contextual semantic modeling and feature transformation on the concatenated feature data through the Transformer network layer in the pre-trained large language model.

[0069] In this context, the Transformer network layer refers to a neural network layer based on a self-attention mechanism; for example, the 12-layer Transformer decoder layer in a large language model. Contextual semantic modeling refers to the process of establishing semantic relationships between features; for example, analyzing the semantic relationship between the time feature of "weekday morning rush hour" and the location feature of "commercial area". Feature transformation refers to the operation of mapping input features to a new feature space; for example, converting 1280-dimensional concatenated features into 1024-dimensional deep semantic features using a Transformer layer. Intermediate feature data refers to the feature representations generated by intermediate layers of the model; for example, the 1024-dimensional hidden state features after processing by the Transformer network layer.

[0070] Specifically, the Transformer network layer receives the concatenated feature data as input, uses a self-attention mechanism to calculate the association weights between different feature positions in the concatenated feature data, performs weighted fusion of features based on the association weights, and completes contextual semantic modeling; then, a feedforward neural network performs a nonlinear transformation on the weighted fused features to achieve feature transformation and output intermediate feature data.

[0071] The intermediate feature data is processed sequentially through the output aggregation layer and projection layer in the pre-trained large language model to obtain the predicted travel time value.

[0072] The output aggregation layer refers to a neural network layer that integrates the multiple outputs of the model; for example, an aggregation layer that performs a weighted summation of the last four hidden states of a Transformer. The projection layer refers to a neural network layer that maps features to the target dimension; for example, an output layer that maps 1024-dimensional aggregated features to 1-dimensional time prediction values ​​through a linear transformation.

[0073] Specifically, the output aggregation layer receives intermediate feature data, performs a weighted summation operation on multiple feature vectors contained in the intermediate feature data, and generates an aggregated feature vector. The aggregated feature vector is then input into the projection layer, which maps the aggregated feature vector into a one-dimensional value through a linear transformation. This one-dimensional value is the predicted travel time.

[0074] In the above-mentioned optional approach, the second embedded data and the optimized feature data are further concatenated in the spatial dimension, and then the context is modeled through the Transformer network layer of the large language model. After processing by the output aggregation layer and the projection layer, feature deep fusion and accurate time prediction are achieved.

[0075] In one alternative approach, S5 specifically includes: The high-level features of the image are obtained by extracting features from the real-time image in front of the camera using a pre-trained visual Transformer model.

[0076] Here, the pre-trained visual Transformer model refers to a pre-trained image model based on the Transformer architecture; for example, using the ViT-B / 16 model pre-trained on the ImageNet dataset to extract image features. High-level image features refer to abstract image features extracted by deep networks; for example, the 768-dimensional feature vector output by the ViT model contains semantic information such as road structure and vehicle positions.

[0077] Specifically, the real-time image in front is segmented into multiple image blocks by a visual Transformer model. Linear projection is performed on each image block to obtain a block embedding vector. After adding position encoding, the vector is input into the Transformer encoder. Feature processing is performed through a multi-layer self-attention mechanism and a feedforward neural network in the encoder. The feature vector corresponding to the [CLS] label is extracted from the encoder output as a high-level feature of the image.

[0078] By using the pre-trained large language model and combining it with pre-set text prompts, semantic understanding and reasoning analysis are performed on the high-level features of the image and the predicted travel time to generate the explainable reasons for the predicted travel time.

[0079] Among these, pre-defined text prompts refer to the instruction text that guides the model to generate explanations; for example, a fixed prompt statement such as "Please analyze the reason for traffic congestion based on the current scene and predicted time." Semantic understanding refers to the process of parsing the semantic connotation of data; for example, a large language model identifies the semantic association between the features of "dense traffic flow" and "traffic congestion." Inference analysis refers to the process of deriving conclusions based on logical rules; for example, combining "rain scenes" and "peak hours" to deduce the conclusion that "the combination of slippery road surface and traffic flow causes delays."

[0080] Specifically, high-level image features and predicted travel time are input into a pre-trained large language model, along with pre-defined text prompts. The pre-trained large language model performs semantic parsing and logical reasoning on the high-level image features and predicted travel time based on the text prompts. By analyzing the causal relationship between image content and time prediction, key factors affecting travel time are identified. The pre-trained large language model then transforms the causal relationship analysis results into a natural language text description, generating an explainable reason for the predicted travel time.

[0081] In the above-mentioned optional methods, high-level features of the real-time image ahead are further extracted through a visual Transformer model, and semantic reasoning of the image features and predicted values ​​is performed by combining a large language model to generate explainable reasons in natural language form, thereby enhancing the understandability and credibility of the prediction results.

[0082] In one alternative approach, it also includes: The current latitude and longitude information of the target vehicle is obtained through the GPS equipped on the target vehicle.

[0083] The current precise time of the target vehicle is obtained through the vehicle's onboard system clock or GPS.

[0084] Among them, the vehicle system clock refers to the time measuring device built into the vehicle; for example, the embedded real-time clock module that is synchronized with GPS time in a smart electric vehicle.

[0085] The current pose information of the target vehicle is obtained through the inertial measurement unit mounted on the target vehicle.

[0086] Inertial measurement unit (IMU) refers to a combination of sensors that measure the motion state of an object; for example, the MPU-6050 sensor module which includes a three-axis gyroscope and a three-axis accelerometer.

[0087] The camera installed at the front of the target vehicle acquires a real-time view of the area in front of the target vehicle.

[0088] Among them, the front-mounted camera refers to an image acquisition device fixed to the front of the vehicle; for example, an 8-megapixel autonomous driving camera installed behind the windshield.

[0089] In the above-mentioned optional methods, further latitude and longitude information is obtained through GPS, accurate time is obtained through vehicle clock or GPS, pose information is obtained through inertial measurement unit, and real-time images are obtained through front camera, so as to ensure accurate collection and synchronization of multimodal data sources and provide a reliable data foundation for subsequent processing.

[0090] like Figure 2 As shown, this embodiment processes multimodal sensor data using a pre-trained large language model to generate travel time prediction results and corresponding explanations. Specifically: 1) The target vehicle's GPS provides current latitude and longitude information, the onboard clock provides the current precise time, the inertial measurement unit collects current pose information, and the front-mounted camera captures real-time images of the front. The current latitude and longitude information is input into the first embedding layer and converted into a latitude and longitude embedding vector; the current precise time is input into the second embedding layer and converted into a time embedding vector; the current pose information is input into the third embedding layer and converted into a pose embedding vector; and the real-time image of the front is input into the visual embedding layer and converted into an image embedding vector. The latitude and longitude embedding vectors, time embedding vectors, pose embedding vectors, and image embedding vectors are concatenated along the feature dimension to form the first embedding data.

[0091] 2) The first embedded data is input into the cross-attention encoder. The cross-attention encoder performs attention calculation through a multi-head attention mechanism to generate attention-weighted features, which are then linearly transformed through the linear layer in the cross-attention encoder to obtain optimized feature data. At the same time, a hard cue word template containing current latitude and longitude information, current pose information, real-time image in front, and current precise time is constructed. The hard cue word template is processed by the cue word embedding layer to be transformed into the second embedded data.

[0092] 3) The second embedded data and the optimized feature data are concatenated spatially to form concatenated feature data. This concatenated feature data is input into a pre-trained large language model. The pre-trained large language model uses a Transformer network layer to perform contextual semantic modeling and feature transformation on the concatenated feature data, generating intermediate feature data. The intermediate feature data is then sequentially processed through the output aggregation layer of the pre-trained large language model for feature aggregation, and then through a projection layer for dimensional mapping, ultimately outputting the predicted travel time.

[0093] 4) The real-time video feed ahead is used for feature extraction via a pre-trained visual Transformer model. This model segments the video feed into image patches and converts them into patch embedding vectors. After adding positional encoding, these vectors are processed by a multi-layer Transformer encoder to output high-level video features. These high-level video features, combined with predicted travel time values ​​and pre-defined text prompts, are input into a pre-trained large language model. This model then uses semantic understanding and reasoning analysis to generate interpretable reasons for the travel time predictions.

[0094] Figure 3 A schematic diagram of an embodiment of a travel time estimation system 200 incorporating a large model, provided by the present invention, is shown. Figure 3 As shown, the combined large model travel time estimation system 200 includes: The first embedding module 201 is used to embed the target vehicle's current latitude and longitude information, current pose information, real-time front view and current precise time to obtain the first embedding data; Encoding processing module 202 is used to input the first embedded data into a cross-attention encoder for capturing the correlation between multiple data to obtain optimized feature data; The second embedding module 203 is used to embed the current latitude and longitude information, the current pose information, the real-time image in front and the current precise time using a hard prompt word method to obtain the second embedding data; The splicing prediction module 204 is used to splice the second embedded data with the optimized feature data to obtain spliced ​​feature data, and input the spliced ​​feature data into a pre-trained large language model for processing to obtain a travel time prediction value. The prediction and explanation module 205 is used to generate an explainable reason for the predicted travel time based on the real-time image ahead and the predicted travel time using the large language model.

[0095] In one alternative embodiment, the first embedding module 201 is specifically used for: The current latitude and longitude information is input into the first embedding layer for processing to obtain the latitude and longitude embedding vector; The current precise time is input into the second embedding layer for processing to obtain a time embedding vector; The current pose information is input into the third embedding layer for processing to obtain the pose embedding vector. The real-time image from the front is input into the visual embedding layer for processing to obtain the image embedding vector. The latitude and longitude embedding vector, the time embedding vector, the pose embedding vector, and the image embedding vector are concatenated along the feature dimension to obtain the first embedding data.

[0096] In an alternative embodiment, the encoding processing module 202 is specifically used for: Attention calculations are performed on the first embedded data using the multi-head attention mechanism in the cross-attention encoder to generate attention-weighted features; The attention-weighted features are linearly transformed through the linear layer in the cross-attention encoder to obtain the optimized feature data.

[0097] In one alternative embodiment, the second embedding module 203 is specifically used for: Construct a hard-coded prompt word template that includes the current latitude and longitude information, the current pose information, the real-time view ahead, and the current precise time; The hard prompt word template is input into the prompt word embedding layer for processing to obtain the second embedded data.

[0098] In an alternative embodiment, the stitching prediction module 204 is specifically used for: The second embedded data and the optimized feature data are concatenated in the spatial dimension to obtain the concatenated feature data; Intermediate feature data is obtained by performing contextual semantic modeling and feature transformation on the concatenated feature data through the Transformer network layer in the pre-trained large language model. The intermediate feature data is processed sequentially through the output aggregation layer and projection layer in the pre-trained large language model to obtain the predicted travel time value.

[0099] In an alternative embodiment, the prediction interpretation module 205 is specifically used for: The high-level features of the image are obtained by extracting features from the real-time image in front of the camera using a pre-trained visual Transformer model. By using the pre-trained large language model and combining it with pre-set text prompts, semantic understanding and reasoning analysis are performed on the high-level features of the image and the predicted travel time to generate the explainable reasons for the predicted travel time.

[0100] In one alternative embodiment, the data acquisition module is further included; the data acquisition module is used for: The current latitude and longitude information of the target vehicle is obtained through the GPS equipped on the target vehicle; The current precise time of the target vehicle is obtained through the vehicle's onboard system clock or GPS. The current pose information of the target vehicle is obtained through the inertial measurement unit mounted on the target vehicle. The camera installed at the front of the target vehicle acquires a real-time view of the area in front of the target vehicle.

[0101] It should be noted that the beneficial effects of the combined large model travel time estimation system 200 provided in the above embodiments are the same as those of the combined large model travel time estimation method, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0102] The combined large model travel time estimation system 200 of the present invention can be a computer program (including program code) running on a computer device. For example, the combined large model travel time estimation system 200 of the present invention is an application software that can be used to execute the corresponding steps in the combined large model travel time estimation method of the present invention.

[0103] In some embodiments, the combined large-model travel time estimation system 200 of the present invention can be implemented in a hardware-software hybrid manner. As an example, the combined large-model travel time estimation system 200 of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the combined large-model travel time estimation method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0104] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0105] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned methods for travel time estimation using a combined large model. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the method for travel time estimation using a combined large model as shown in any embodiment of the present invention by calling the computer program.

[0106] In one alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0107] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0108] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0109] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0110] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0111] Among them, electronic devices can also be terminal devices. A terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0112] It should be noted that, Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0113] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned methods for estimating travel time using a large model.

[0114] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0115] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned travel time estimation method incorporating a large model.

[0116] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0118] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0119] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0120] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0121] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0122] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0123] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for estimating travel time using a large-scale model, characterized in that, include: The target vehicle's current latitude and longitude information, current pose information, real-time front view, and current precise time are embedded to obtain the first embedded data; The first embedded data is input into a cross-attention encoder for capturing the correlation between multiple data to obtain optimized feature data; The current latitude and longitude information, the current pose information, the real-time image in front, and the current precise time are embedded using a hard cue word method to obtain the second embedded data; The second embedded data is concatenated with the optimized feature data to obtain concatenated feature data, and the concatenated feature data is input into a pre-trained large language model for processing to obtain a travel time prediction value. Based on the real-time video feed ahead and the predicted travel time, the interpretable reasons for the predicted travel time are generated using the large language model.

2. The travel time estimation method combining a large model according to claim 1, characterized in that, The step of embedding the target vehicle's current latitude and longitude information, current pose information, real-time front view, and current precise time to obtain the first embedded data includes: The current latitude and longitude information is input into the first embedding layer for processing to obtain the latitude and longitude embedding vector; The current precise time is input into the second embedding layer for processing to obtain a time embedding vector; The current pose information is input into the third embedding layer for processing to obtain the pose embedding vector. The real-time image from the front is input into the visual embedding layer for processing to obtain the image embedding vector. The latitude and longitude embedding vector, the time embedding vector, the pose embedding vector, and the image embedding vector are concatenated along the feature dimension to obtain the first embedding data.

3. The travel time estimation method combining a large model according to claim 2, characterized in that, The step of inputting the first embedded data into a cross-attention encoder for capturing the correlation between multiple data types to obtain optimized feature data includes: Attention calculations are performed on the first embedded data using the multi-head attention mechanism in the cross-attention encoder to generate attention-weighted features; The attention-weighted features are linearly transformed through the linear layer in the cross-attention encoder to obtain the optimized feature data.

4. The travel time estimation method combining a large model according to claim 3, characterized in that, The step of embedding the current latitude and longitude information, the current pose information, the real-time image in front, and the current precise time using a hard-signal method to obtain the second embedded data includes: Construct a hard-coded prompt word template that includes the current latitude and longitude information, the current pose information, the real-time view ahead, and the current precise time; The hard prompt word template is input into the prompt word embedding layer for processing to obtain the second embedded data.

5. The travel time estimation method combining a large model according to claim 4, characterized in that, The step of concatenating the second embedded data with the optimized feature data to obtain concatenated feature data, and inputting the concatenated feature data into a pre-trained large language model for processing to obtain the travel time prediction value, includes: The second embedded data and the optimized feature data are concatenated in the spatial dimension to obtain the concatenated feature data; Intermediate feature data is obtained by performing contextual semantic modeling and feature transformation on the concatenated feature data through the Transformer network layer in the pre-trained large language model. The intermediate feature data is processed sequentially through the output aggregation layer and projection layer in the pre-trained large language model to obtain the predicted travel time value.

6. The travel time estimation method combining a large model according to claim 5, characterized in that, The step of generating an explainable reason for the predicted travel time based on the real-time video feed and the predicted travel time using the large language model includes: The high-level features of the image are obtained by extracting features from the real-time image in front of the camera using a pre-trained visual Transformer model. By using the pre-trained large language model and combining it with pre-set text prompts, semantic understanding and reasoning analysis are performed on the high-level features of the image and the predicted travel time to generate the explainable reasons for the predicted travel time.

7. The travel time estimation method combining a large model according to any one of claims 1 to 6, characterized in that, Also includes: The current latitude and longitude information of the target vehicle is obtained through the GPS equipped on the target vehicle; The current precise time of the target vehicle is obtained through the vehicle's onboard system clock or GPS. The current pose information of the target vehicle is obtained through the inertial measurement unit mounted on the target vehicle. The camera installed at the front of the target vehicle acquires a real-time view of the area in front of the target vehicle.

8. A travel time estimation system combining a large model, characterized in that, include: The first embedding module is used to embed the target vehicle's current latitude and longitude information, current pose information, real-time image ahead, and current precise time to obtain the first embedded data; The encoding processing module is used to input the first embedded data into a cross-attention encoder for capturing the correlation between multiple data to obtain optimized feature data. The second embedding module is used to embed the current latitude and longitude information, the current pose information, the real-time image in front, and the current precise time using a hard-suggestion word method to obtain the second embedded data; The splicing prediction module is used to splice the second embedded data with the optimized feature data to obtain spliced ​​feature data, and input the spliced ​​feature data into a pre-trained large language model for processing to obtain a travel time prediction value; The prediction and explanation module is used to generate an explainable reason for the predicted travel time based on the real-time image ahead and the predicted travel time using the large language model.

9. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the travel time estimation method combining a large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, implements the travel time estimation method combining a large model as described in any one of claims 1 to 7.

Images