Highway gantry scene multi-source vehicle information matching and fusion method based on single-agent structure large language model

Abstract

The application discloses a kind of multi-source vehicle information matching fusion methods in highway gantry scene based on single-agent structure large language model, comprising: multi-source vehicle information perception data in highway gantry scene is mined and labeled;Multi-source vehicle information matching fusion dataset in highway gantry scene is constructed;Based on multi-source vehicle information matching process, construct large language single-agent structure model LLMSA-MVIMF-EGS for multi-source vehicle information matching fusion in highway gantry scene;Multi-source vehicle information matching fusion result is output by large language single-agent structure model LLMSA-MVIMF-EGS.The application is matched training on the basis of large language model, for specific multi-source vehicle information data in highway gantry scene, and provides certain technical support for highway gantry ETC charging.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and relates to the identification of vehicle information on highways. Specifically, it relates to a method for matching and fusing multi-source vehicle information in highway gantry scenarios based on a single-agent structure large language model. Background Technology

[0002] As a vital transportation artery in my country, expressways bear a tremendous burden of transport tasks. In the process of intelligent development of expressways, multi-source data fusion technology is of great significance for the analysis of ETC toll collection data. By fusing video-captured vehicle holographic perception data and RSU vehicle information perception data, it is possible to analyze transportation demand more comprehensively and accurately, formulate reasonable toll policies, and simultaneously improve toll collection efficiency, alleviate road pressure, and enhance road safety and highway service levels.

[0003] Currently, most ETC toll collection on highways uses a method of matching vehicle toll data based on the similarity of vehicle license plate characters. However, this technology faces problems such as reduced matching accuracy and inability to effectively identify large vehicles with small license plates when dealing with inconsistencies in license plate information collected by multiple sensors.

[0004] Therefore, a new technological solution is needed to address these issues. Summary of the Invention

[0005] Purpose of the invention: In order to overcome the shortcomings of the existing technology, this invention provides a method for matching and fusing multi-source vehicle information in highway gantry scenarios based on a single-agent structure large language model. On the basis of the large language model, it performs matching training on multi-source vehicle information data in specific highway gantry scenarios, providing certain technical support for highway gantry ETC toll collection.

[0006] Technical Solution: To achieve the above objectives, this invention provides a method for matching and fusing multi-source vehicle information in highway gantry scenarios based on a single-agent structure large language model, comprising the following steps:

[0007] S1: Mining and labeling multi-source vehicle information perception data in highway gantry scenarios;

[0008] S2: Based on the data labeled in step S1, construct a multi-source vehicle information matching and fusion dataset in the highway gantry scenario;

[0009] S3: Based on the multi-source vehicle information matching process, construct the large language single agent structure model LLMSA-MVIMF-EGS for multi-source vehicle information matching and fusion in highway gantry scenarios;

[0010] S4: Based on the fused dataset from step S2, output the multi-source vehicle information matching and fusion results through the Large Language Model based Single-Agent for Multi-source Vehicle Information Matching and Fusion in Expressway Gantry Scenes (LLMSA-MVIMF-EGS).

[0011] Furthermore, in step S1, the multi-source vehicle information perception data in the highway gantry scene includes video capture vehicle holographic perception data and RSU vehicle information perception data.

[0012] The video capture vehicle holographic perception data includes vehicle license plate information, vehicle passage time, vehicle type information, etc., and the vehicle license plate information and vehicle passage time are preprocessed separately; the RSU vehicle perception data mainly includes vehicle license plate information, vehicle passage time, vehicle transaction result, vehicle type information, etc., and the vehicle passage time, vehicle transaction result, and vehicle type information are preprocessed separately.

[0013] Further, in step S1, a process for labeling multi-source vehicle information perception data in highway gantry scenarios is constructed by combining a text information matching fusion method based on character comparison with a text information matching fusion method based on unsupervised contrastive learning. Specifically, the process involves labeling multi-source vehicle information perception data in highway gantry scenarios. During the labeling process, data matching is mainly based on license plate features and time features, and it is assumed that the license plate information characters of the same vehicle transaction data are allowed to have two or fewer errors. For the time feature, given that vehicles with the same license plate may pass through the gantry multiple times on the same day, in order to reduce computational costs and improve the timeliness and accuracy of data labeling, a time window of five minutes above and below the time of a vehicle's passage can be set to narrow the data retrieval range. For any data A in the video capture vehicle holographic perception data... i First, extract the vehicle passage time data t. i The data is divided into five-minute intervals before and after the interval, and then data with time information matching the requirements of these intervals is searched in the RSU vehicle information perception dataset (B). i1 B i2 ...B ij The search scope was reduced from the entire RSU vehicle information perception dataset to j data points that met the time criteria.

[0014] After labeling, a certain number of data were obtained, including data of successful matching and fusion, data of failed matching and fusion, and data of no vehicle license plate. The data of successful matching and fusion included data of completely identical vehicle license plates and data of roughly identical vehicle license plates.

[0015] Furthermore, the specific process of constructing the multi-source vehicle information matching and fusion dataset in the highway gantry scenario in step S2 is as follows:

[0016] S2-1: Data that simultaneously satisfies both vehicle license plate characteristics and vehicle type characteristics are taken as positive samples, and data that do not simultaneously satisfy both vehicle license plate characteristics and vehicle type characteristics are taken as negative samples.

[0017] S2-2: Add label "1" to positive sample data pairs and label "0" to negative sample data pairs; the training set contains positive and negative samples of the dataset with completely identical license plates and roughly identical license plates, while the validation and test sets contain only positive and negative samples of the dataset with roughly identical license plates; the ratio of positive to negative samples in the dataset is 1:1, the ratio of completely identical to roughly identical license plates in the training set is 1:1, and the ratio of positive to negative samples of the dataset with roughly identical license plates in the training, validation, and test sets is 6:2:2.

[0018] Furthermore, the large language single-agent structure model LLMSA-MVIMF-EGS in step S3 includes a Single-Agent module, a large language model, and an information output module;

[0019] The Single-Agent module is used to generate prompt words and matching data in the highway gantry scenario and feed them into the large language model.

[0020] The large language model is used to predict the matching results based on the prompt words and the data to be matched.

[0021] The information output module is used to classify and save the matching result data.

[0022] Furthermore, the prompt words constructed in the Single-Agent module consist of six parts: the role played by the large language model, the question background, the contextual information, the input data, the expected output result, and the special requirements. For the role played by the large language model, the prompt words are set as a professional data analyst. For the question background, the prompt words are set as a classification problem between vehicle license plates and vehicle types. For the contextual information, the prompt words input the classification criteria for vehicle license plates and vehicle types. For the input data, the prompt words use a test set of multi-source vehicle information matching and fusion data from a highway gantry scenario as input. For the expected output result, the prompt words require the large language model to output "yes" or "no" based on the recognition result. For the special requirements, the prompt words require the output result of the large language model to be within one character in length. When comparing vehicle license plate area codes and alphanumeric sequences, the model needs to focus on the inclusion relationship of the sequences rather than the corresponding relationship, and characters with differences will not be calculated additionally during the comparison process.

[0023] Furthermore, the large language model is the Classify GPT4 module. The Transformer module in the Classify GPT4 module adopts the Causal Decoder architecture, uses Full Language Modeling as the prediction target, and generates the classification result (0 / 1) of the data to be matched.

[0024] Furthermore, this invention constructs a multi-source vehicle information matching process based on a large language single agent: First, multi-source vehicle perception data from the highway gantry scenario is input into a single agent structural model. The single agent structural model performs a question content enhancement task and combines vehicle license plate data with vehicle type data, transforming it into a data format input to the large language model, generating prompt words and matching data in the highway gantry scenario respectively. Second, the single agent structural model inputs the prompt words into the large language model. The large language model learns the task content based on information such as the role played by the large language model in the prompt words, the question background, contextual information, input data examples, expected output results, and special requirements. Then, the single agent structural model sequentially inputs the matching data into the large language model. The large language model predicts the matching data according to the requirements of the prompt words and outputs the matching result (0 / 1). Finally, the single agent structural model classifies and saves the matching data according to the matching results. If the matching result is 0, the data set is marked as a failed match; if the matching result is 1, the vehicle passage time difference of the data set is calculated, and the data whose time difference falls within the range of successful matching in this invention is marked as successfully matched data, otherwise it is marked as data with questionable results for manual secondary verification later.

[0025] Based on the above matching process, the specific process of step S4 is as follows:

[0026] S4-1: Input the multi-source vehicle information test set from the highway gantry scene into the Single-Agent module; among them, the video capture vehicle holographic perception dataset includes vehicle license plate data (PA1, PA2…PA). n Vehicle type data (VA1, VA2…VA) n ), through time data (TA1, TA2…TA) n The RSU vehicle information perception dataset contains vehicle license plate data (PB1, PB2…PB). n Vehicle type data (VB1, VB2…VB) n ), through time data (TB1, TB2…TB) n );

[0027] S4-2: The Single-Agent module inputs information such as the model's role, the question's background, contextual information, input data samples, expected output results, and special requirements into the large language model, helping Classify GPT4, which classifies vehicle license plate information and vehicle type information, understand the task requirements; the Single-Agent module inputs vehicle license plate data PA from the video capture vehicle holographic perception dataset. i Vehicle type data (VA) i Vehicle license plate data PB in the RSU vehicle information perception dataset i Vehicle type data VB i Transformed into the data format input to the large language model (PA) i +VA i ,PB i +VB i ), which is used as the input data to be matched into the Classify GPT4 module;

[0028] S4-3: The Transformer module of the Classify GPT4 module adopts a Causal Decoder architecture, using Full Language Modeling as the prediction target to generate classification results (0 / 1) for the data to be matched. The Single-Agent module saves the classification results as the data to be matched (PA). i +VA i ,PB i +VB i The label is set to 'Label-i', and then the next data to be matched (PA) is entered again. i+1 +VA i+1 ,PB i+1 +VB i+1 This process continues until all data is matched and merged, and then all data is output to the information output module.

[0029] S4-4: The information output module outputs data based on the data to be matched (PA). i +VA i ,PB i +VB i The label Label-i is used to categorize and save the data. Specifically, if Label-i is 0, the original data pairs (PA) of the group are saved. i ,VA i ,TA i ), (PB i ,VB i ,TB i The label (Label-i) is saved to the failed matching dataset; if Label-i is 1, the next step of discrimination is performed; the information output module calculates the transit time data (TA) in the video capture vehicle holographic perception dataset. i With the RSU vehicle information perception dataset, the transit time data TB i Perform the difference operation to calculate the time difference T between vehicle passages. i =(TA) i -TB i The information output module outputs T... i Perform a judgment, if T i If the range is (-30s, 120s), then the original data pair of this set of data (PA) i ,VA i ,TA i ), (PB i ,VB i ,TB i Save the label (Label-i) to the successfully matched dataset; if T i If the value exceeds the range of (-30s, 120s), then the original data will be paired with (PA). i ,VA i ,TA i ), (PB i ,VB i ,TB i Save the label (Label-i) to the dataset of questionable results.

[0030] Beneficial effects: Compared with existing technologies, this invention, based on a large language model, performs matching training and prediction on multi-source vehicle information data in specific highway gantry scenarios. The matching and fusion accuracy exceeds 98%, which can effectively match video-captured vehicle holographic perception data and RSU vehicle information perception data, providing certain technical support for highway ETC intelligent toll collection systems. Attached Figure Description

[0031] Figure 1This is a flowchart of the matching and fusion process based on the large language single-agent structure model in this invention;

[0032] Figure 2 The running diagram of the LLMSA-MVIMF-EGS model provided by this invention;

[0033] Figure 3 This is a structural diagram of the CDSM-MVIMF-EGS model;

[0034] Figure 4 Here is a diagram of the MVLSTM-MVIMF-EGS model structure;

[0035] Figure 5 This is a structural diagram of the ALBERT-MVIMF-EGS model. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0037] like Figure 1 and Figure 2 As shown, this invention provides a method for matching and fusing multi-source vehicle information in highway gantry scenarios based on a single-agent structure large language model, comprising the following steps:

[0038] S1: Mining and labeling multi-source vehicle information perception data in highway gantry scenarios;

[0039] The dataset used in step S1 of this embodiment comes from four consecutive days of relevant data collected from four different gantries in January 2024. The multi-source vehicle information perception data in the highway gantry scene includes video capture vehicle holographic perception data and RSU vehicle information perception data.

[0040] Table 1 shows the video capture data of vehicle holographic perception and RSU vehicle information perception data collected by each gantry:

[0041] Table 1. Number of multi-source vehicle information perception data in highway gantry scenarios (unit: data items)

[0042]

[0043] The video capture data for vehicle holographic perception includes vehicle license plate information, vehicle passage time, and vehicle type information. The vehicle license plate information and vehicle passage time are preprocessed separately. The RSU vehicle perception data mainly includes vehicle license plate information, vehicle passage time, vehicle transaction results, and vehicle type information. The vehicle passage time, vehicle transaction results, and vehicle type information are preprocessed separately. First, the vehicle passage time information is uniformly converted into a date format that can be recognized and calculated by the Datetime module in Python. Second, the numbers in the vehicle transaction results column are converted into the corresponding transaction status. Finally, the numbers in the vehicle type information column are converted into corresponding textual descriptions of the vehicle type information.

[0044] This paper constructs a process for annotating multi-source vehicle information perception data in highway gantry scenarios by combining a character comparison-based text information matching fusion method with an unsupervised contrastive learning-based text information matching fusion method. Specifically, the process involves annotating multi-source vehicle information perception data in highway gantry scenarios. During the annotation process, data matching is primarily based on license plate features and time features, assuming that the license plate information characters of the same vehicle transaction data are allowed to have two or fewer errors. Regarding time features, given that vehicles with the same license plate may pass through the gantry multiple times on the same day, to reduce computational costs and improve the timeliness and accuracy of data annotation, a five-minute time window can be set before and after a vehicle's passage time to narrow the data retrieval range. For any data A in the video-captured vehicle holographic perception data... i First, extract the vehicle passage time data t. i The data is divided into five-minute intervals before and after the interval, and then data with time information matching the requirements of these intervals is searched in the RSU vehicle information perception dataset (B). i1 B i2 ...B ij The search scope was reduced from the entire RSU vehicle information perception dataset to j data points that met the time criteria.

[0045] In this embodiment, after annotation, a total of 462,199 positive samples with successful matching and fusion, 40,167 negative samples with failed matching and fusion, and 65,690 samples without vehicle license plates were obtained. Among the positive samples with successful matching and fusion, there were 442,643 samples with completely identical vehicle license plates and 19,556 samples with approximately identical vehicle license plates. The specific annotation results are shown in Table 2.

[0046] Table 2. Annotation results of multi-source vehicle information perception data in highway gantry scenarios (unit: data items)

[0047]

[0048] S2: Based on the data labeled in step S1, construct a multi-source vehicle information matching and fusion dataset for highway gantry scenarios. The specific process is as follows:

[0049] S2-1: Data that simultaneously satisfies both vehicle license plate characteristics and vehicle type characteristics are considered positive samples, while data that do not simultaneously satisfy both vehicle license plate characteristics and vehicle type characteristics are considered negative samples; the number of data in each category is shown in Table 3.

[0050] Table 3. Number of sample data of each type in the multi-source vehicle information matching and fusion dataset in the highway gantry scenario.

[0051]

[0052] S2-2: Add label "1" to positive sample data pairs and label "0" to negative sample data pairs; the training set contains positive and negative samples of the datasets with completely identical and roughly identical license plates, while the validation and test sets contain only positive and negative samples of the datasets with roughly identical license plates; the ratio of positive to negative samples in the datasets is 1:1, the ratio of completely identical to roughly identical license plates in the training set is 1:1, and the ratio of positive to negative samples of roughly identical license plates in the training, validation, and test sets is 6:2:2; the number of samples of each type in the constructed training, validation, and test sets is shown in Table 4.

[0053] Table 4. Number of sample data of each type in the training set, validation set, and test set.

[0054]

[0055] S3: Based on the multi-source vehicle information matching process, construct the large language single agent structure model LLMSA-MVIMF-EGS for multi-source vehicle information matching and fusion in highway gantry scenarios;

[0056] The large language single-agent structural model LLMSA-MVIMF-EGS includes a Single-Agent module, a large language model, and an information output module.

[0057] The Single-Agent module generates prompt words and matching data for highway gantry scenarios and feeds them into the large language model; the large language model makes predictions based on the prompt words and matching data, and outputs the matching results; the information output module classifies and saves the matching results data.

[0058] The prompts constructed in the Single-Agent module consist of six parts: the role of the large language model, the question background, contextual information, input data, expected output, and special requirements. For the role of the large language model, the prompts define it as a professional data analyst. For the question background, the prompts define it as a vehicle license plate and vehicle type classification problem. For the contextual information, the prompts input the classification criteria for vehicle license plates and vehicle types. For the input data, the prompts use a test set of multi-source vehicle information matching and fusion data from a highway gantry scenario as input. For the expected output, the prompts require the large language model to output "yes" or "no" based on the recognition result. For the special requirements, the prompts require the output length of the large language model to be controlled within one character. When comparing vehicle license plate area codes and alphanumeric sequences, the model needs to focus on the inclusion relationship of the sequences rather than their correspondence, and characters with differences will not be calculated additionally during the comparison process.

[0059] In this embodiment, the large language model is the Classify GPT4 module. The Transformer module in the Classify GPT4 module adopts the Causal Decoder architecture, uses Full Language Modeling as the prediction target, and generates the classification result (0 / 1) of the data to be matched.

[0060] S4: Based on the fused dataset from step S2, output the multi-source vehicle information matching and fusion results through the large language single-agent structure model LLMSA-MVIMF-EGS.

[0061] This invention constructs a multi-source vehicle information matching process based on a large language single agent: First, multi-source vehicle perception data from a highway gantry scenario is input into a single agent structural model. The single agent structural model performs a question content enhancement task and combines vehicle license plate data with vehicle type data, transforming it into a data format input to the large language model, generating prompt words and matching data for the highway gantry scenario. Second, the single agent structural model inputs the prompt words into the large language model. The large language model learns the task content based on information such as the role of the large language model in the prompt words, the question background, contextual information, input data examples, expected output results, and special requirements. Then, the single agent structural model sequentially inputs the matching data into the large language model. The large language model predicts the matching data based on the requirements of the prompt words and outputs a matching result (0 / 1). Finally, the single agent structural model classifies and saves the matching data according to the matching results. If the matching result is 0, the data set is marked as a failed match; if the matching result is 1, the vehicle passage time difference of the data set is calculated, and the data whose time difference falls within the range of successful matching in this invention is marked as successfully matched data, otherwise it is marked as data with questionable results for manual secondary verification later.

[0062] Based on the above matching process, the specific process of step S4 is as follows:

[0063] S4-1: Input the multi-source vehicle information test set from the highway gantry scene into the Single-Agent module; among them, the video capture vehicle holographic perception dataset includes vehicle license plate data (PA1, PA2…PA). n Vehicle type data (VA1, VA2…VA) n ), through time data (TA1, TA2…TA) n The RSU vehicle information perception dataset contains vehicle license plate data (PB1, PB2…PB). n Vehicle type data (VB1, VB2…VB) n ), through time data (TB1, TB2…TB) n );

[0064] S4-2: The Single-Agent module inputs information such as the model's role, the question's background, contextual information, input data samples, expected output results, and special requirements into the large language model, helping Classify GPT4, which classifies vehicle license plate information and vehicle type information, understand the task requirements; the Single-Agent module inputs vehicle license plate data PA from the video capture vehicle holographic perception dataset. i Vehicle type data (VA) i Vehicle license plate data PB in the RSU vehicle information perception dataseti Vehicle type data VB i Transformed into the data format input to the large language model (PA) i +VA i ,PB i +VB i ), which is used as the input data to be matched into the Classify GPT4 module;

[0065] S4-3: The Transformer module of the Classify GPT4 module adopts a Causal Decoder architecture, using Full Language Modeling as the prediction target to generate classification results (0 / 1) for the data to be matched. The Single-Agent module saves the classification results as the data to be matched (PA). i +VA i ,PB i +VB i The label is set to 'Label-i', and then the next data to be matched (PA) is entered repeatedly. i+1 +VA i+1 ,PB i+1 +VB i+1 This process continues until all data is matched and merged, and then all data is output to the information output module.

[0066] S4-4: The information output module outputs data based on the data to be matched (PA). i +VA i ,PB i +VB i The label Label-i is used to categorize and save the data. Specifically, if Label-i is 0, the original data pairs (PA) of the group are saved. i ,VA i ,TA i ), (PB i ,VB i ,TB i The label (Label-i) is saved to the failed matching dataset; if Label-i is 1, the next step of discrimination is performed; the information output module calculates the transit time data (TA) in the video capture vehicle holographic perception dataset. i With the RSU vehicle information perception dataset, the transit time data TB i Perform the difference operation to calculate the time difference T between vehicle passages. i =(TA) i -TB i The information output module outputs T... i Perform a judgment, if T i If the range is (-30s, 120s), then the original data pair of this set of data (PA)i ,VA i ,TA i ), (PB i ,VB i ,TB i Save the label (Label-i) to the successfully matched dataset; if T i If the value exceeds the range of (-30s, 120s), then the original data will be paired with (PA). i ,VA i ,TA i ), (PB i ,VB i ,TB i Save the label (Label-i) to the dataset of questionable results.

[0067] Based on the above, in order to verify the superiority of the GPT4SA-MVIMF-EGS model provided by this invention, this embodiment constructs six models—CDSSM-MVIMF-EGS, MVLSTM-MVIMF-EGS, ALBERT-MVIMF-EGS, BaiChuan2SA-MVIMF-EGS, ChatGLM3SA-MVIMF-EGS, and the GPT4SA-MVIMF-EGS of this invention—for matching experiments. The specific construction method is as follows:

[0068] 1. Construct a CDSSM-MVIMF-EGS model for multi-source vehicle information matching and fusion in highway gantry scenarios based on convolutional neural networks. Its main contents are as follows: Convolutional Deep Structured Semantic Models (CDSSM). CDSSM enhances the ability to capture contextual relationships by incorporating word-level n-gram analysis and convolutional pooling mechanisms, thereby achieving a significant improvement in model performance. The CDSSM-MVIMF-EGS model structure is as follows: Figure 3 As shown in Table 5, the parameters of the CDSSM-MVIMF-EGS model are as follows:

[0069] Table 5 - CDSM-MVIMF-EGS Model Parameters

[0070]

[0071] For text A and text B input into the CDSM-MVIMF-EGS model, the text is first decomposed using a character-level n-gram technique through the Embedding Layer. Each word is then converted into a corresponding vector representation of size (None, 64, 200) using a Letter-Trigram matrix, thereby capturing local features of the text. This refined text information representation strategy not only keenly captures subtle semantic changes within the text information but also improves the model's generalization ability to process different text types. Secondly, convolutional layers are used to extract local features of the text information. These convolutional layers obtain rich local text features by sliding windows on the character-level n-gram. The feature matrix of text A is transformed into a feature vector of size (None, 64, 300) through the Conv1D_1 convolutional layer, and the feature matrix of text B is transformed into a feature vector of size (None, 64, 300) through the Conv1D_2 convolutional layer. Each convolutional operation can be regarded as a feature detection, which can... The CDSSM-MVIMF-EGS model first identifies key patterns and structures in the text. Then, it uses Max-Pooling layers to reduce feature dimensionality, aiming to reduce overfitting risk while improving generalization ability. The feature vector of text A is transformed into a (None, 300) feature vector through the pooling layer Global_Max_Pooling1d_1, and the feature vector of text B is transformed into a (None, 300) feature vector through the pooling layer Global_Max_Pooling1d_2. Finally, the CDSSM-MVIMF-EGS model outputs the matching and fusion results between text information through fully connected layers and similarity dot product calculation. The feature vector of text A is transformed into a (None, 128) feature vector through two dense layers Dense_1 and Dense_2, and the feature vector of text B is transformed into a (None, 128) feature vector through two dense layers Dense_3 and Dense_4. The two feature vectors are input into the Dot Layer for similarity dot product calculation and connected to the dense layer Dense_5 to output the matching and fusion results of text A and text B.

[0072] 2. The specific content of constructing the MVLSTM-MVIMF-EGS model for multi-source vehicle information matching and fusion in highway gantry scenarios based on recurrent neural networks is as follows: Multi-View Long Short-Term Memory (MVLSTM) is an improved version of Long Short-Term Memory (LSTM) networks used to capture long-distance dependencies in text sequences and introduces multi-view strategies to enhance the effectiveness of text information representation and inter-sentence interactions. The MVLSTM-MVIMF-EGS model mainly consists of a sentence representation layer, a multi-view matching layer, and an interaction aggregation layer. In the sentence representation layer, the text information of each input model is processed by a bidirectional LSTM network to obtain bidirectional semantic information. The bidirectional semantic information of each word is concatenated as the semantic representation of that word as the output. Assume that text A and text B input to the MVLSTM-MVIMF-EGS model are transformed into sentence representations u and v. In the multi-view matching layer, the sentence representations are processed through cosine similarity, bilinearity, and tensor layers to extract fine-grained interaction relationships between sentences. The cosine similarity calculation method is as follows: The bilinear calculation method is s(u,v)=u T M1v+b, the tensor layer calculation method is as follows: Where M1 represents the matrix weights, b represents the bias parameters, M2 represents the tensor parameters, and f represents the nonlinear function. In the interaction pooling layer, the similarity matrix between text A and text B is calculated through inter-local interactions, and a K-Max pooling layer is connected to extract features from the similarity matrix. The result is then output as a matching fusion result after passing through a fully connected layer. The structure of the MVLSTM-MVIMF-EGS model is as follows: Figure 4 As shown in Table 6, the parameters of the MVLSTM-MVIMF-EGS model are as follows:

[0073] Table 6. Parameters of each layer in the MVLSTM-MVIMF-EGS neural network

[0074]

[0075]

[0076] First, texts A and B input to the MVLSTM-MVIMF-EGS model are processed by the EmbeddingLayer and transformed into feature vectors of size (None, 64, 100). Second, the feature vectors of texts A and B are respectively input into the bidirectional long short-term memory network in the Bidirectional Layer to extract the bidirectional semantic information of each word in the text, outputting a feature vector of size (None, 64, 128). Then, the feature vectors of texts A and B are respectively input into the Lambda_1 and Lambda_2 layers to extract fine-grained cosine similarity, bilinearity, and tensor layer relationships between sentences. Next, in the Dot... Layer performs inter-regional interaction calculations on the multi-view features of text A and text B, generating a similarity matrix of size (None, 64, 64). A Reshape layer is then connected to transform the similarity matrix into a size of (None, 4096). A max pooling layer K_Max_Pooling is then connected to generate a feature vector of size (None, 50). Finally, the MVLSTM-MVIMF-EGS model outputs the matching and fusion results between text information through fully connected layers. Two dense layers, Dense_1 and Dense_2, are connected to a Dropout layer to transform the result into a feature vector of size (None, 128). A dense layer, Dense_3, is then connected to output the matching and fusion results of text A and text B.

[0077] 3. Construct an ALBERT-MVIMF-EGS model based on Transformer for multi-source vehicle information matching and fusion in highway gantry scenarios. The specific method is: A Lite BERT. The ALBERT model is an improved pre-trained language model based on BERT. This model employs two key techniques: embedding parameter sharing and cross-layer parameter sharing. While maintaining or even improving model performance, it significantly reduces model size and training costs. Furthermore, the ALBERT model introduces sentence order prediction (SOP), enhancing the model's depth of understanding of text structure. The constructed ALBERT-MVIMF-EGS model structure is as follows: Figure 5 As shown in Table 7, the parameters of the ALBERT-MVIMF-EG model are as follows:

[0078] Table 7 Parameters of each layer in the AERT-MVIMF-EGS model

[0079]

[0080]

[0081] The ALBERT-MVIMF-EGS model consists of three parts: an input layer, an encoder layer, and an output layer. In the input layer, texts A and B are input to the ALBERT-MVIMF-EGS model. First, they are subdivided into the smallest units, Tokens, through word segmentation and standardization. Then, classification tags [CLS] and delimiters [SEP] are added to each Token to complete the format conversion. Second, each Token undergoes three layers of processing: Token Embedding, Segment Embedding, and Position Embedding, to obtain a rich semantic representation vector that integrates the lexical features, paragraph origin, and positional information of the token. During word vector generation, the ALBERT-MVIMF-EGS model uses embedding parameter decomposition to split the large word embedding matrix into two smaller matrices multiplied together, thereby reducing the number of parameters. Finally, the ALBERT-MVIMF-EGS model combines the outputs of these three layers, integrating the lexical features, paragraph origin, and positional information of the token, to construct a semantic representation vector of size (None, None, 1024). This vector is then used as input to the encoder layer for deeper semantic understanding and processing. In the encoder layer, the ALBERT-MVIMF-EGS model contains 24 Transformer encoder modules, each containing a multi-head self-attention layer and a feedforward neural network, designed to encode and model the input text information. The ALBERT-MVIMF-EGS model employs cross-layer parameter sharing, sharing the parameters of the multi-head self-attention layer and the feedforward neural network across all layers. In this model, only the parameters of the first Transformer encoder are learned; the remaining 23 Transformer encoders share the parameters of a single Transformer encoder. In the output layer, the ALBERT-MVIMF-EGS model includes pooling layers and classification layers. The pooling layers Pooler-Lambda and Pooler-Dense are responsible for pooling the output of the encoder layer to obtain a fixed-dimensional representation of the input sequence (None, 1024). The classification layer contains a fully connected layer and a Softmax activation function, which is responsible for transforming the fixed-dimensional output of the pooling layer into a probability distribution for text matching, and outputting the prediction results of text matching.

[0082] In this embodiment, six models were used to conduct matching experiments: CDSM-MVIMF-EGS, MVLSTM-MVIMF-EGS, ALBERT-MVIMF-EGS, GPT4SA-MVIMF-EGS, BaiChuan2SA-MVIMF-EGS, and ChatGLM3SA-MVIMF-EGS. The detection results of the test set are shown in Table 8.

[0083] Table 8. Analysis of Experimental Results

[0084]

[0085] As shown in Table 8, the overall detection performance of the GPT4SA-MVIMF-EGS model provided by this invention is better than that of the other five models. The first three are matching models built on convolutional neural networks, recurrent neural networks and Transformer architectures, respectively. BaiChuan2SA-MVIMF-EGS and ChatGLM3SA-MVIMF-EGS are two other matching models built on a single agent structure of large language. As a single agent structure matching model based on the GPT4 large language model, GPT4SA-MVIMF-EGS (1) can understand the needs of classification tasks faster and more accurately than convolutional neural networks, recurrent neural networks and Transformer architectures; (2) GPT4 has advantages over other large language models in terms of model structure, parameters and training set, and its performance is also better. Therefore, the GPT4SA-MVIMF-EGS model is more suitable for matching multi-source vehicle information in highway gantry scenarios. The model uses the Causal Decoder architecture and Full Language Modeling as the prediction target to generate the classification results of the data to be matched.

Claims

1. A method for matching and fusing multi-source vehicle information in a highway gantry scenario based on a single-agent structure large language model, characterized in that, Includes the following steps: S1: Mining and labeling multi-source vehicle information perception data in highway gantry scenarios; S2: Based on the data labeled in step S1, construct a multi-source vehicle information matching and fusion dataset in the highway gantry scenario; S3: Based on the multi-source vehicle information matching process, construct the large language single agent structure model LLMSA-MVIMF-EGS for multi-source vehicle information matching and fusion in highway gantry scenarios; S4: Based on the fused dataset from step S2, output the multi-source vehicle information matching and fusion results through the large language single-agent structure model LLMSA-MVIMF-EGS; In step S3, the large language single-agent structure model LLMSA-MVIMF-EGS includes a Single-Agent module, a large language model, and an information output module; The Single-Agent module is used to generate prompt words and matching data in the highway gantry scenario and feed them into the large language model. The large language model is used to predict the matching results based on the prompt words and the data to be matched. The information output module is used to classify and save the matching result data; The prompts constructed in the Single-Agent module consist of six parts: the role of the large language model, the question background, contextual information, input data, expected output, and special requirements. For the role of the large language model, the prompts define it as a professional data analyst. For the question background, the prompts define it as a vehicle license plate and vehicle type classification problem. For the contextual information, the prompts input the classification criteria for vehicle license plates and vehicle types. For the input data, the prompts use a test set of multi-source vehicle information matching and fusion data from a highway gantry scenario as input. For the expected output, the prompts require the large language model to output "yes" or "no" based on the recognition result. For the special requirements, the prompts require the output length of the large language model to be controlled within one character. When comparing vehicle license plate area codes and alphanumeric sequences, the model needs to focus on the inclusion relationship of the sequences rather than their correspondence, and characters with differences will not be calculated additionally during the comparison process. The large language model is the Classify GPT4 module. The Transformer module in the Classify GPT4 module adopts the Causal Decoder architecture, uses Full Language Modeling as the prediction target, and generates the classification results of the data to be matched.

2. The method for matching and fusing multi-source vehicle information in a highway gantry scenario based on a single-agent structure large language model as described in claim 1, characterized in that, In step S1, the multi-source vehicle information perception data in the highway gantry scene includes video capture vehicle holographic perception data and RSU vehicle information perception data. The video capture vehicle holographic perception data includes vehicle license plate information, vehicle passage time, and vehicle type information, and the vehicle license plate information and vehicle passage time are preprocessed separately; the RSU vehicle perception data includes vehicle license plate information, vehicle passage time, vehicle transaction result, and vehicle type information, and the vehicle passage time, vehicle transaction result, and vehicle type information are preprocessed separately.

3. The method for matching and fusing multi-source vehicle information in a highway gantry scenario based on a single-agent structure large language model, as described in claim 2, is characterized in that... Step S1 involves combining a character-based text information matching fusion method with an unsupervised contrastive learning-based text information matching fusion method to construct a process for labeling multi-source vehicle information perception data in highway gantry scenarios. Specifically, this involves labeling multi-source vehicle information perception data in highway gantry scenarios. During the labeling process, data matching is performed based on license plate features and time features, assuming that the license plate information characters of the same vehicle transaction data are allowed to have two or fewer errors. For the time feature, a five-minute time window is set before and after the time a vehicle passes to narrow the data retrieval range. For any data point in the video-captured vehicle holographic perception data... First, extract the vehicle passage time data. The time period is divided into five minutes before and after the time period, and then data with time information that meets the requirements of the time period is searched in the RSU vehicle information perception dataset. , The search scope was reduced from the entire RSU vehicle information perception dataset to j data points that met the time criteria. After labeling, we obtained data on successful matching and fusion, failed matching and fusion, and no vehicle license plate. Successful matching and fusion data included data with completely identical vehicle license plates and data with roughly identical vehicle license plates.

4. The method for matching and fusing multi-source vehicle information in a highway gantry scenario based on a single-agent structure large language model, as described in claim 3, is characterized in that... The specific process of constructing the multi-source vehicle information matching and fusion dataset in the highway gantry scenario in step S2 is as follows: S2-1: Data that simultaneously satisfies both vehicle license plate characteristics and vehicle type characteristics are taken as positive samples, and data that do not simultaneously satisfy both vehicle license plate characteristics and vehicle type characteristics are taken as negative samples. S2-2: Add label "1" to positive sample data pairs and label "0" to negative sample data pairs; the training set contains positive and negative samples of the dataset with completely identical license plates and roughly identical license plates, while the validation and test sets contain only positive and negative samples of the dataset with roughly identical license plates.

5. The method for matching and fusing multi-source vehicle information in a highway gantry scenario based on a single-agent structure large language model, as described in claim 4, is characterized in that... The specific process of step S4 is as follows: S4-1: Input the multi-source vehicle information test set from the highway gantry scene into the Single-Agent module; among which, the video capture vehicle holographic perception dataset includes vehicle license plate data. Vehicle type data ), through time data The RSU vehicle information perception dataset contains vehicle license plate data. Vehicle type data ), through time data ); S4-2: The Single-Agent module inputs information such as the model's role, the question's background, contextual information, input data samples, expected output results, and special requirements into the large language model, helping Classify GPT4, which classifies vehicle license plate information and vehicle type information, understand the task requirements; the Single-Agent module also inputs vehicle license plate data from the video capture vehicle holographic perception dataset. Vehicle type data Vehicle license plate data in the RSU vehicle information perception dataset Vehicle type data Transformed into the data format input to the large language model. , The data to be matched is input into the Classify GPT4 module; S4-3: The Transformer module of the Classify GPT4 module adopts a Causal Decoder architecture, using Full Language Modeling as the prediction target to generate classification results (0 / 1) for the data to be matched. The Single-Agent module saves the classification results as the data to be matched. , The label is set to "Label-i", and then the next data to be matched is entered repeatedly. , This continues until all data is matched and merged, and then all data is output to the information output module. S4-4: The information output module outputs data according to the data to be matched. , The label Label-i categorizes and saves the data. Specifically, if Label-i is 0, the original data of that group is stored separately. , , , The label (Label-i) is saved to the dataset of failed matches; if Label-i is 1, the next step of discrimination is performed; the information output module calculates the transit time data in the video capture vehicle holographic perception dataset. With the transit time data in the RSU vehicle information perception dataset Perform the difference to calculate the time difference between vehicle passages. =( The information output module supports... Perform a judgment, if If the range is (-30s, 120s), then the original data of this set of data is compared with... , , , Save the label (Label-i) to the successfully matched dataset; if If the value exceeds the range of (-30s, 120s), then the original data will be recalculated. , , , Save the label (Label-i) to the dataset of questionable results.

Images