An end-to-end traffic road state perception method based on a multi-modal large model

Abstract

The application discloses an end-to-end traffic road state perception method based on a multi-modal large model, which comprises the following steps: collecting traffic video data sets and traffic text data sets; extracting each frame of image of at least part of the traffic video data and obtaining traffic features through feature extraction; labeling each frame of image based on a traffic element set; cleaning, segmenting and tokenizing the traffic text data sets; calculating a joint probability by counting a first ratio and a second ratio; judging whether the traffic elements and the traffic features are associated based on the joint probability; extracting the cleaned traffic text data sets by using a sliding window based on the associated traffic elements and traffic features, so as to obtain training samples; pre-training a large language model to obtain a pre-trained large language model; training a perception large model; and inputting a to-be-input video image into the trained perception large model to obtain a scene understanding text. The application simultaneously realizes accurate target detection and text description of a road network state.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing. Specifically, it relates to an end-to-end traffic road condition perception method based on a multimodal large model. Background Technology

[0002] In recent years, with the increasing complexity of urban traffic, video surveillance traffic network monitoring methods based on traditional deep learning object detection have been limited in terms of real-time performance, accuracy, and environmental adaptability, making it difficult to meet the needs of smart city traffic management.

[0003] Traditional visual models for road traffic condition perception primarily rely on image processing and computer vision techniques, extracting traffic flow information through the analysis of video surveillance or image data. This method constructs a neural network structure adapted to the characteristics of traffic data, extracting key features from videos or sensors to identify traffic condition information such as vehicle position, speed, and traffic flow. Specifically, traditional deep learning models can segment and detect image data to identify vehicles, pedestrians, and abnormal events on the road. While traditional deep learning methods can achieve high accuracy in identifying traffic conditions under specific conditions, their over-reliance on data annotation limits their performance in complex environments such as low light levels, significantly impacting the accuracy of traffic network condition descriptions. Therefore, a large-scale, end-to-end perception model with a large number of parameters is needed to improve road network condition perception capabilities. Summary of the Invention

[0004] The present invention is proposed based on the above-mentioned needs of the prior art. The technical problem to be solved by the present invention is to provide an end-to-end traffic road state perception method based on a multimodal large model to improve the road network state perception capability.

[0005] To solve the above problems, the present invention is implemented using the following technical solution:

[0006] This paper provides an end-to-end traffic road state perception method based on a multimodal large model. The method includes: acquiring a traffic video dataset and a traffic text dataset; extracting each frame of the traffic video data and performing feature extraction on each frame to obtain traffic features; labeling each frame based on a traffic element set, where if an image contains a certain traffic element, the corresponding traffic element is labeled as 1; otherwise, it is labeled as 0; cleaning, segmenting, and tokenizing the traffic text dataset; calculating a first ratio and a second ratio, where the first ratio represents the ratio of the frequency of traffic elements appearing in the traffic video dataset to the total number of words in the traffic vocabulary, and the second ratio represents the ratio of the frequency of traffic features appearing in the traffic video dataset to the total number of words in the traffic vocabulary; calculating a joint probability based on the first and second ratios; determining whether the joint probability exceeds a threshold, and if it does, associating the corresponding traffic element with the corresponding traffic feature; and based on... The associated traffic elements and features are extracted from the cleaned traffic text dataset using a sliding window to obtain training samples. A pre-trained large language model is then developed based on these training samples. This pre-training process includes: inputting labeled phrases corresponding to traffic elements into the large language model to facilitate its understanding of semantic information within the traffic scene; using the large language model to pre-train the training samples to predict descriptive words for traffic state changes; training the perception model based on a traffic video dataset, corresponding traffic text data, and labeled images and corresponding traffic elements. The perception model sequentially includes a pre-trained image encoder, a location embedding layer, a Q-Former, a linear layer, and the pre-trained large language model. During training, the parameters of the pre-trained image encoder and the pre-trained large language model are frozen; finally, the input video image is input into the trained perception model to obtain scene-understanding text.

[0007] Optionally, the method is characterized by training the labeled traffic video dataset based on the Sigmoid cross-entropy loss function to obtain a trained automatic annotation model; and using the automatic annotation model to automatically annotate images; the expression for the Sigmoid cross-entropy loss function is: Where L represents the cross-entropy loss function for image annotation, m represents the number of images after frame extraction from the traffic video data, n represents the sequence number of the image, and y n P represents an image of sequence n. n This represents the image annotation results for sequence n.

[0008] Optionally, the joint probability is calculated based on the first ratio and the second ratio, and its expression is:

[0009] Where A represents the occurrence of traffic elements in the traffic video dataset, B represents the occurrence of traffic features in the traffic video dataset, P(A|B) represents the joint probability, P(B|A) represents the probability of event B occurring given that event A has occurred, P(A) represents the first ratio, and P(B) represents the second ratio.

[0010] Optionally, L2 regularization can be used to control the complexity of the model during the pre-training process of the large language model.

[0011] Optionally, the Lion optimizer can be used to adjust the batch size of the input data for the pre-trained large language model.

[0012] Optionally, each time the batch size is adjusted, the weights of the pre-trained large language model are initialized using the learning decay rate, the expression for which the learning decay rate is calculated is:

[0013] Where α represents the learning decay rate, DecayRate represents the initial large language model decay rate, EpochNumber represents the number of model training iterations, and α0 represents the initial learning rate.

[0014] Optionally, during pre-training, the large language model is fine-tuned based on the LoRA algorithm.

[0015] Optionally, the large language model is LLaMA3, and the network structure of LLaMA3, from top to bottom, includes: a first layer normalization, a residual connection, a multi-head self-attention layer, a second layer normalization, and a fully connected layer.

[0016] Compared with existing technologies, this invention provides an end-to-end traffic and road condition perception method based on a multimodal large model. It enables automatic image annotation and enhances the ability to extract correlation characteristics between different samples by constructing the correlation between traffic features and traffic elements. Furthermore, the end-to-end perception large model of this invention outputs scene-understanding textual descriptions, greatly enhancing the reasoning ability of automatic scene understanding. It achieves end-to-end perception and description of traffic conditions, while simultaneously enabling accurate target detection and textual description of road network conditions in complex environments. This satisfies the need for accurate perception of urban road traffic networks and reduces the cost of manual road traffic condition detection. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart of an end-to-end traffic road state perception method based on a multimodal large model provided in this embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the scope of protection of the present invention.

[0021] This embodiment provides an end-to-end traffic road condition perception method based on a multimodal large model, the process of which is as follows: Figure 1 As shown, it includes:

[0022] S1 collects traffic video datasets and traffic text datasets.

[0023] Collect traffic video datasets and traffic text datasets in the field of urban road traffic.

[0024] S2 extracts at least a portion of the traffic video data from each frame, and performs feature extraction on each frame to obtain traffic features.

[0025] Using a convolutional network as the input layer, a convolution operation is performed on the pixel matrix corresponding to the image to be labeled. During the convolution operation, the number of elements in the resulting feature matrix is ​​typically reduced compared to the input matrix, necessitating zero-padding (padding the input image matrix with zeros). This zero-padding process effectively preserves the size of the feature matrix. Subsequently, the defined convolution kernel slides progressively across the padded input matrix, performing dot product operations to extract feature information from the image. The formula for the convolution operation is as follows:

[0026]

[0027] In the formula, X i Let X be the feature obtained after the i-th convolutional layer, and W be the input feature. i b represents the convolution kernel in the i-th convolutional layer. i This represents the bias of the i-th convolutional layer. This represents the operation between the convolution kernel and image features, where f represents the activation function used after the convolution operation.

[0028] After padding, it is necessary to calculate the size of the output image after the convolution operation. The size of the convolution kernel is F×F, and the stride of the convolution kernel is S. Assuming the padding is P, then the output size of an input image of size W×W×C1 after passing through the convolution layer is Q×Q×C2, where Q is calculated using the following formula:

[0029] Q = (W - F + 2P) / S + 1

[0030] Convolutional operations can reduce the number of parameters in a function; however, video image data is massive, and feature extraction still faces the challenge of high-parameter computation. To reduce the computational cost of image feature extraction, pooling layers are introduced to reduce the spatial size of features. Max pooling is used for feature repetitive sampling. Traffic monitoring feature maps contain extremely rich information, covering edge features, local features, and main features. However, not all extracted features are actually what needs to be labeled. Before inputting them into the next layer of the network, measures are taken to remove these unnecessary features, retaining key information. This process not only effectively reduces the number of parameters during network training but also effectively controls overfitting and improves the model's generalization ability.

[0031] S3 labels each frame of image based on the traffic element set. If an image contains a certain traffic element, the corresponding traffic element is labeled as 1; otherwise, it is labeled as 0.

[0032] The performance of large-scale visual models largely depends on training with a large amount of accurate and rich video image data. Video data provides spatiotemporal information about dynamic scenes, helping models not only recognize objects but also understand their motion trajectories, behavior patterns, and scene changes. Video image annotation is the process of adding labels to targets and scenes in videos to train visual models; image annotation is the foundation of video annotation. Image annotation methods include object detection annotation, semantic segmentation annotation, instance segmentation annotation, keypoint annotation, and tracking annotation. Through these annotation methods, video data can be structured and parsed, helping multimodal models identify, understand, and predict different targets and behaviors in complex dynamic scenes during training. Object detection annotation is mainly used to determine the information and location of traffic elements such as cars and pedestrians in images. Objects are selected using bounding boxes, and each box is labeled.

[0033] Suppose that the sequence of extracted frames x from a set of video X is X' = {x1, x2, ..., x...} n}. Y = {y1, y2, ..., y j ,…,y m} represents the set of traffic elements that have already been annotated in the traffic element database. Such an automatic video annotation task can be represented as an N-pair 1-M mapping set of image-traffic element pairs from the video frame-by-frame image set and the traffic element set, denoted as P = {(x1,Y1),(x2,Y2),…,(x...}. i ,Y i ),…(x n ,Y n )}, Y i Y is an M-dimensional vector whose elements take values ​​of 0 or 1, indicating whether the image contains that element. i y in j When y = 1, it indicates that the traffic element y j In image x i The middle label. When Y i y in j When y = 0, it means y j In image x i There is no label in it.

[0034] For the multi-traffic element labeling problem, the Sigmoid cross-entropy loss function is used to train the automatic labeling model, and its expression is:

[0035]

[0036] Where L represents the cross-entropy loss function for image annotation, m represents the number of images after frame extraction from the traffic video data, n represents the sequence number of the image, and y n P represents an image of sequence n. n This represents the image annotation results for sequence n.

[0037] The derivation of the above expression is as follows: The probability output by the Sigmoid cross-entropy function needs to be mapped to the [0,1] interval, and this probability needs to reflect the probability of predicting a positive class. The prediction output is the probability when the sample label is 1:

[0038] P n =P(y=1|x)

[0039] If the image sample is 0, the probability is as follows:

[0040] 1-P n =P(y=0|x)

[0041] The formulas for constructing the maximum likelihood value are combined as follows:

[0042]

[0043] When the traffic element label of the image in the next video segment is y=0, the first term in the following formula becomes 1, and the probability needs to be further transformed, calculated as follows:

[0044] P(y=0|x)=1-P n

[0045] When the traffic element label of the image in the next video segment is y=1, the probability needs to be further transformed and calculated as follows:

[0046] P(y=1|x)=P n

[0047] The probability expression remains unchanged in both cases. Under the overall probability expression, a higher probability indicates better feature extraction of traffic elements. To reduce the computational complexity of the exponential function without altering the monotonicity of the probability calculation formula, a log function is introduced, as follows:

[0048] logP(y|x)=y n logP n +(1-y n log(1-P) n )

[0049] For m traffic element labels, a Sigmoid cross-entropy loss function is constructed.

[0050] The semantic annotation of the multimodal large model divides the pixels in the image into different category regions. Each traffic element is labeled as a specific category, such as road, vehicle, pedestrian, etc., and relevant text augmentation training is performed on these categories to improve the richness of the descriptive content and reasoning ability. The specific text augmentation training process is shown in S4-S8.

[0051] S4 cleans, segments, and tokenizes traffic text datasets.

[0052] Traffic text datasets often contain noise and errors, and data from different sources may have inconsistent formats. Data cleaning is necessary to ensure the quality of data training, including removing HTML tags, handling missing data, and removing duplicate samples.

[0053] The cleaned data is segmented and tokenized. Specifically, the traffic text data is divided into sequences of words or sub-words. After segmentation, each word can be mapped to a corresponding identifier, and the text can be converted to lowercase, punctuation marks and special characters can be removed using the index in the vocabulary.

[0054] S5 calculates the first and second ratios.

[0055] The first ratio represents the ratio of the number of times traffic elements appear in the traffic video dataset to the total number of words in the traffic vocabulary, and the second ratio represents the ratio of the number of times traffic features appear in the traffic video dataset to the total number of words in the traffic vocabulary.

[0056] S6 calculates the joint probability based on the first ratio and the second ratio; determines whether the joint probability exceeds a threshold; if it exceeds the threshold, the corresponding traffic element and the corresponding traffic feature are associated.

[0057] The expression for calculating the joint probability P(A|B) is as follows:

[0058]

[0059] Where A represents the occurrence of traffic elements in the traffic video dataset, B represents the occurrence of traffic features in the traffic video dataset, P(A|B) represents the joint probability, P(B|A) represents the probability of event B occurring given that event A has occurred, P(A) represents the first ratio, and P(B) represents the second ratio.

[0060] In this embodiment, the threshold is set to 0.8. When P(A|B) exceeds 0.8 during training, traffic elements and traffic features are considered to be associated.

[0061] In this embodiment, a large language model can also be used to train traffic text data to output the correlation between traffic elements and traffic features.

[0062] S7 uses a sliding window to extract training samples from the cleaned traffic text dataset based on associated traffic elements and features.

[0063] Based on the characteristics of urban traffic, the data is organized into training samples. For large language models, continuous sequences can be extracted from the text using a sliding window based on associated traffic elements and features, with the goal of inferring descriptive words for possible changes in traffic status. The dataset can be expanded by performing random transformations on the training data. For example, text can be randomly truncated, noise added, or synonyms replaced to improve the model's robustness and generalization ability.

[0064] S8 pre-trains the large language model based on training samples to obtain a pre-trained large language model.

[0065] A large language model is pre-trained based on a large dataset of labeled traffic text. Preferably, the large language model is LLaMA3, whose basic structure is a Transformer structure. Its core is a self-attention mechanism to capture dynamic textual language relevance, and a multi-head attention mechanism to realize feature calculation for multiple outputs.

[0066] To improve the stability of LLaMA3 during training, a pre-layer normalization method was introduced. The first layer normalization was moved before the multi-head self-attention layer, and the second layer normalization was also moved before the fully connected layer. Simultaneously, the residual connection was adjusted to be placed after the multi-head self-attention layer and the fully connected layer. The adjusted LLaMA3 network structure, from top to bottom, includes: first layer normalization, residual connection, multi-head self-attention layer, second layer normalization, and fully connected layer.

[0067] The RMSNorm normalization function was used in the layer normalization. The expression for the RMSNorm function to process the input vector is as follows:

[0068]

[0069] Where RMS(a) represents root mean square layer normalization, n represents the feature dimension, and a i This represents the input feature vector. This represents the normalized output feature vector or tensor.

[0070] For positional encoding, Rotational Position Encoding (RoPE) is used instead of the original absolute positional encoding. Based on complex number theory, RoPE aims to achieve relative positional encoding through absolute positional encoding. Its goal is to add absolute positional information to q and k through the following operations:

[0071]

[0072] Where q represents the query vector, m represents the position index, indicating the current position encoding, and k represents the key vector. This represents the absolute positional information that the large language model provides to q. Let f(k,m) represent the absolute positional information that the large language model brings to k, and let f(k,m) represent the encoding function of the absolute positional information.

[0073] Based on the above methods, and It brings absolute positional information of text position embedding to large language models.

[0074] The pre-training process includes:

[0075] Input the labeled phrases corresponding to traffic elements into the large language model so that the large language model can understand the semantic information in the traffic scene.

[0076] The labeled phrases include entity classes, attribute classes, state classes, and relationship classes.

[0077] The training samples are pre-trained using a large language model to enable the large language model to predict traffic state change descriptive words.

[0078] Preferably, during the pre-training of the large language model, L2 regularization is used to control the complexity of the model.

[0079] L2 regularization is used to help control the model's complexity and calculate the model's loss on the original data. total Overfitting is prevented by weight decay.

[0080]

[0081] In the formula, LOSS data Let ||w|| represent the data loss term, where λ is the L2 regularization coefficient, used to control the contribution of the regularization term to the total loss; 2 It is the square of the L2 norm of the weight vector w.

[0082] Preferably, the Lion optimizer is used to adjust the batch size of the input data for the pre-trained large language model.

[0083] Batch input facilitates data parallelism in large language models. During training, a larger batch size typically improves training efficiency, using more samples for each weight update and thus reducing the number of updates, which is crucial for the training speed of large language models. This embodiment considers the difficulty of multimodal training and employs the Lion optimizer to adjust the batch size. It also introduces functions to make the program more compact, such as the linear interpolation function interp(x,y,a), whose optimized original function is (1-a)·x+a·y.

[0084] Preferably, each time the batch size is adjusted, the weights of the pre-trained large language model are initialized using the learning decay rate.

[0085] Batch size is a hyperparameter that needs to be tuned, and should be chosen based on the characteristics of the perceptual large model architecture, task, and dataset. Experimentation is needed in traffic networks to find the optimal batch size. This method starts with a large batch size and then gradually decreases it to improve model stability. A learning decay rate α is used to initialize the weights of the pre-trained model, which helps the model converge quickly. The decay formula is as follows:

[0086]

[0087] Where α0 represents the initial learning rate, which decreases continuously with the number of iterations, DecayRate represents the initial large language model decay rate, EpochNumber represents the number of model training iterations, and α0 represents the initial learning rate.

[0088] S9 trains a large-scale perception model based on traffic video datasets, corresponding traffic text data, and labeled images and corresponding traffic elements.

[0089] An end-to-end perception model is primarily a combined model integrating a perception module and a large language model. Its output includes question-and-answer formats and perceived content output, requiring data to be organized into question-and-answer pairs, source-target pairs, etc. For supervised learning tasks, data labels need to be associated with input samples to enable supervised model training.

[0090] The perceptual large model consists of a pre-trained image encoder, a location embedding layer, a Q-Former, a linear layer, and a pre-trained large language model. During training, the parameters of the pre-trained image encoder and the pre-trained large language model are frozen.

[0091] The core of building a large multimodal perception model lies in video encoding understanding and large language model output. This embodiment proposes a video Q-former to capture the temporal changes of visual scenes. The model is trained on a large dataset of video image caption pairs and visual instruction tuning to align the output of the visual encoder with the embedding space of the LLM.

[0092] The main steps of the end-to-end sensing big data model are as follows:

[0093] Features are extracted from input video frames using a frozen pre-trained image encoder, comprising ViT-G / 14 from EVA-CLIP and a pre-trained Q-Former. Temporal information is injected into the video frames using a location embedding layer. A visual Q-Former is employed to capture the temporal changes in road traffic scenes, providing a dynamic understanding of road traffic conditions. The training strategy for the visual Q-Former is as follows: on the one hand, the visual Q-Former performs multi-objective training on text pairs, aligning visual and text representations to learn the visual representation that best matches the text; on the other hand, a linear layer projects the output of the visual Q-Former into a vector with the same dimension as the text embedding of a large language model, enabling the visual representation learned by the Q-Former to be interpretable by the large language model.

[0094] A visual representation model is trained on a large dataset of videos of various types of traffic incidents to generate video-to-text data. Annotated image-traffic feature pairs are added to the pre-trained dataset to enhance the understanding of static traffic scene concepts. A BLIP image-text matching loss coefficient is constructed through contrastive learning. v2t :

[0095]

[0096] Among them, vi The image embedding representation of the i-th feature, t i The text embedding representation of the i-th feature, sim(v i ,t i ) represents the similarity calculation function, and τ represents the variation parameter of contrastive learning

[0097] Preferably, during the pre-training process, the large language model is fine-tuned based on the LoRA algorithm.

[0098] The text output of the large language model is based on the text prediction and text classification after self-attention layer normalization. There is a large amount of professional domain text content in the urban traffic system. In order to significantly improve the representation ability of the large model in the urban traffic road perception task, it is necessary to finely tune the large model through a certain amount of road traffic text data to improve the task applicability of the pre-trained large model. This embodiment adopts a traffic scene fine-tuning method based on LoRA, and the basic steps are as follows:

[0099] Freeze the parameters of the large language model and use the large number of original parameters of LLaMA3 for inference without updating to ensure the basic language logic. Then create two low-rank matrices A and B, so that the size of the matrix after their multiplication will be the same as the size of the weight matrix of the model being fine-tuned. There are multiple weight matrices in the large language model of this embodiment, and similar matrix pairs are created for each weight matrix.

[0100] Only train the low-rank matrices A and B, while keeping the pre-trained weight matrix W unchanged. The pre-trained weight matrix W is a d×k matrix, which represents the weights in the original model and remains frozen during the adaptation process without training and updating. The low-rank matrix B is a d×r matrix, and the low-rank matrix A is an r×k matrix, where r<<min(d,k), which is the maximum rank of LoRA. A is initialized using a random Gaussian distribution, and its elements are sampled from a Gaussian distribution with a mean of 0. B is a zero matrix at initialization. Based on this reparameterization, the expression of Lora fine-tuning is:

[0101] h = Wx + ΔWx = Wx + BAx

[0102] Where, ΔW represents the parameter update during fine-tuning. LoRA restricts the weight update, making ΔW = BA. W0x represents the original forward propagation process, that is, the input x passes through the weight matrix W to obtain the output. In the large language model, x can be a word embedding vector, and W is the weight matrix from the word embedding layer to the hidden layer. BAx represents the correction term introduced in the fine-tuning process, and the product BA represents the correction to the original weight matrix W.

[0103] Through this reparameterization, LoRa fine-tuning can adapt to the road traffic perception content text generation task by learning a low-rank matrix BA while keeping the pre-trained weights unchanged. This significantly reduces the number of parameters that need to be trained and improves the efficiency of traffic perception. Furthermore, since the matrix W is pre-trained and remains unchanged during training, the computational and storage costs of large perception models can be further reduced.

[0104] S10 inputs the video image to be input into the trained perception model to obtain scene understanding text.

[0105] This embodiment provides an end-to-end traffic and road condition perception method based on a multimodal large model. It enables automatic image annotation and enhances the ability to extract correlation characteristics between different samples by constructing the correlation between traffic features and traffic elements. Furthermore, the end-to-end perception large model outputs scene-understanding text descriptions, greatly enhancing the reasoning ability for automatic scene understanding. This allows for end-to-end perception and description of traffic conditions, while simultaneously achieving accurate target detection and text description of road network conditions in complex environments. This meets the requirements for accurate perception of urban road traffic networks and reduces the cost of manual road traffic condition detection.

[0106] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An end-to-end traffic road state perception method based on a multimodal large model, characterized in that, include: Collect traffic video datasets and traffic text datasets; Extract at least a portion of the traffic video data from each frame, and perform feature extraction on each frame to obtain traffic features; Each frame of the image is labeled based on the traffic element set. If an image contains a certain traffic element, the corresponding traffic element is labeled as 1. Otherwise, mark it as 0; Cleaning, segmentation, and tokenization of traffic text datasets; The first ratio and the second ratio are calculated. The first ratio represents the ratio of the number of times traffic elements appear in the traffic video dataset to the total number of words in the traffic vocabulary. The second ratio represents the ratio of the number of times traffic features appear in the traffic video dataset to the total number of words in the traffic vocabulary. The joint probability is calculated based on the first ratio and the second ratio; it is then determined whether the joint probability exceeds a threshold. If it exceeds the threshold, the corresponding traffic element and the corresponding traffic feature are associated. Based on the associated traffic elements and traffic features, a sliding window is used to extract training samples from the cleaned traffic text dataset. The large language model is pre-trained based on training samples to obtain a pre-trained large language model, including: inputting the labeled word groups corresponding to traffic elements into the large language model so that the large language model can understand the semantic information in the traffic scene; and using the large language model to pre-train the training samples so that the large language model can predict traffic state change descriptive words. The perception model is trained based on traffic video dataset, corresponding traffic text data, and data consisting of labeled images and corresponding traffic elements. The perception model includes a pre-trained image encoder, a location embedding layer, a Q-Former, a linear layer, and a pre-trained large language model. During the training process, the parameters of the pre-trained image encoder and the pre-trained large language model are frozen. The video image to be input is fed into the trained perception model to obtain scene understanding text.

2. The end-to-end traffic road state perception method based on a multimodal large model according to claim 1, characterized in that, The automatic annotation model is trained on the labeled traffic video dataset based on the Sigmoid cross-entropy loss function. Automatic annotation of images is performed using an automatic annotation model; The expression for the Sigmoid cross-entropy loss function is: Where L represents the cross-entropy loss function for image annotation, m represents the number of images after frame extraction from the traffic video data, n represents the sequence number of the image, and y n P represents an image of sequence n. n This represents the image annotation results for sequence n.

3. The end-to-end traffic road state perception method based on a multimodal large model according to claim 1, characterized in that, The joint probability is calculated based on the first ratio and the second ratio, and its expression is as follows: Where A represents the occurrence of traffic elements in the traffic video dataset, B represents the occurrence of traffic features in the traffic video dataset, P(A|B) represents the joint probability, P(B|A) represents the probability of event B occurring given that event A has occurred, P(A) represents the first ratio, and P(B) represents the second ratio.

4. The end-to-end traffic road state perception method based on a multimodal large model according to claim 1, characterized in that, During the pre-training of large language models, L2 regularization is used to control the complexity of the model.

5. The end-to-end traffic road state perception method based on a multimodal large model according to claim 1, characterized in that, The Lion optimizer is used to adjust the batch size of the input data for the pre-trained large language model.

6. The end-to-end traffic road state perception method based on a multimodal large model according to claim 5, characterized in that, Each time the batch size is adjusted, the weights of the pre-trained large language model are initialized using the learning decay rate. The expression for calculating the learning decay rate is as follows: Where α represents the learning decay rate, DecayRate represents the initial large language model decay rate, EpochNumber represents the number of model training iterations, and α0 represents the initial learning rate.

7. The end-to-end traffic road state perception method based on a multimodal large model according to claim 1, characterized in that, During the pre-training process, the large language model is fine-tuned based on the LoRA algorithm.

8. The end-to-end traffic road state perception method based on a multimodal large model according to claim 1, characterized in that, The large language model is LLaMA3, and the network structure of LLaMA3, from top to bottom, includes: a first layer normalization, a residual connection, a multi-head self-attention layer, a second layer normalization, and a fully connected layer.

Images