Highway gantry scene multi-source vehicle information matching and fusion method based on improved multi-agent structure large language model

Abstract

The application discloses a kind of based on improved multi-agent structure large language model's highway gantry scene in multi-source vehicle information matching fusion method, comprising: to the multi-source vehicle information perception data in highway gantry scene is mined and labeled;Highway gantry scene in multi-source vehicle information matching fusion dataset is constructed;Based on multi-source vehicle information matching process, construct the multi-source vehicle information matching fusion model LLMMADA-MVIMF-EGS based on improved multi-agent structure large language model;According to fusion dataset, the multi-source vehicle information matching fusion result is output by model LLMMADA-MVIMF-EGS, realizes multi-source vehicle information matching fusion in highway gantry scene.The application is based on large language model, and the multi-source vehicle information data in specific highway gantry scene is matched and trained, to provide certain technical support for highway gantry ETC charging.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and relates to the identification of vehicle information on highways. Specifically, it relates to a method for matching and fusing multi-source vehicle information in highway gantry scenarios based on an improved multi-agent structure large language model. Background Technology

[0002] Currently, highway toll data fusion still faces certain limitations, specifically: 1) Due to the complexity and diversity of multi-source vehicle perception data in highway gantry scenarios, existing data collection, standardization, and fusion methods have not yet formed a systematic application; 2) Due to the differences in information collection capabilities among various highway sensors, the actual collected data suffers from inconsistencies, incompleteness, or distortion, affecting the accuracy and effectiveness of multi-source data fusion for highway ETC toll collection. Therefore, in the development of intelligent highway ETC toll collection systems, it is necessary to further optimize data processing methods and improve data fusion methods to fully leverage the potential of multi-source data fusion technology in highway ETC toll management. Summary of the Invention

[0003] Purpose of the invention: In order to overcome the shortcomings of the existing technology, this invention provides a method for matching and fusing multi-source vehicle information in highway gantry scenarios based on an improved multi-agent structure large language model. On the basis of the large language model, it performs matching training on multi-source vehicle information data in specific highway gantry scenarios, providing certain technical support for highway gantry ETC toll collection.

[0004] Technical Solution: To achieve the above objectives, this invention provides a method for matching and fusing multi-source vehicle information in highway gantry scenarios based on an improved multi-agent structure large language model, comprising the following steps:

[0005] S1: Mining and labeling multi-source vehicle information perception data in highway gantry scenarios;

[0006] S2: Based on the data labeled in step S1, construct a multi-source vehicle information matching and fusion dataset in the highway gantry scenario;

[0007] S3: Based on the multi-source vehicle information matching process, construct a multi-source vehicle information matching and fusion model LLMMADA-MVIMF-EGS based on an improved multi-agent structure large language model;

[0008] S4: Based on the fusion dataset from step S2, the multi-source vehicle information matching and fusion results are output through the model LLMMADA-MVIMF-EGS to achieve multi-source vehicle information matching and fusion in the highway gantry scenario.

[0009] Furthermore, in step S1, the multi-source vehicle information perception data in the highway gantry scene includes video capture vehicle holographic perception data and RSU vehicle information perception data.

[0010] The video capture vehicle holographic perception data includes vehicle license plate information, vehicle passage time, and vehicle type information, and the vehicle license plate information and vehicle passage time are preprocessed separately; the RSU vehicle perception data includes vehicle license plate information, vehicle passage time, vehicle transaction result, and vehicle type information, and the vehicle passage time, vehicle transaction result, and vehicle type information are preprocessed separately.

[0011] Further, in step S1, a process for labeling multi-source vehicle information perception data in highway gantry scenarios is constructed by combining a text information matching fusion method based on character comparison with a text information matching fusion method based on unsupervised contrastive learning. Specifically, the process involves labeling multi-source vehicle information perception data in highway gantry scenarios. During the labeling process, data matching is performed based on license plate features and time features, assuming that the license plate information characters of the same vehicle transaction data are allowed to have two or fewer errors. For the time feature, a time window of five minutes above and below the time a vehicle passes is set to narrow the data retrieval range. For any data A in the video capture vehicle holographic perception data... i First, extract the vehicle passage time data t. i The data is divided into five-minute intervals before and after the interval, and then data with time information matching the requirements of these intervals is searched in the RSU vehicle information perception dataset (B). i1 B i2 ...B ij The search scope was reduced from the entire RSU vehicle information perception dataset to j data points that met the time criteria.

[0012] After labeling, we obtained data on successful matching and fusion, failed matching and fusion, and no vehicle license plate. Successful matching and fusion data included data with completely identical vehicle license plates and data with roughly identical vehicle license plates.

[0013] Furthermore, the specific process of constructing the multi-source vehicle information matching and fusion dataset in the highway gantry scenario in step S2 is as follows:

[0014] S2-1: Data that simultaneously satisfies both vehicle license plate characteristics and vehicle type characteristics are taken as positive samples, and data that do not simultaneously satisfy both vehicle license plate characteristics and vehicle type characteristics are taken as negative samples.

[0015] S2-2: Add label "1" to positive sample data pairs and label "0" to negative sample data pairs; the training set contains positive and negative samples of the dataset with completely identical license plates and roughly identical license plates, while the validation and test sets contain only positive and negative samples of the dataset with roughly identical license plates.

[0016] Furthermore, the multi-source vehicle information matching and fusion model LLMMADA-MVIMF-EGS in step S3 includes a controller agent, a vehicle license plate classification agent Identifier agent, a vehicle type classification agent Category agent, a large language model GPT4, a data alignment agent Alignment agent, and an information output module.

[0017] The controller-Agent is used to split the multi-source vehicle information data matching and fusion task into a vehicle license plate matching task and a vehicle type matching task.

[0018] The vehicle license plate classification agent Identifier-Agent is used to generate prompt words and vehicle license plate data in the highway gantry scenario and feed them into the large language model GPT4.

[0019] The vehicle type classification agent, Category-Agent, is used to generate prompt words and vehicle type data in the highway gantry scenario and feed them into the large language model GPT4.

[0020] The large language model GPT4 is used to predict the matching data based on the prompt words and output the matching results to the vehicle license plate classification agent Identifier-Agent and the vehicle type classification agent Category-Agent, respectively.

[0021] The data alignment agent Alignment-Agent is used to perform data trimming and alignment between the classification result P-Label-i stored by Identifier-Agent and the classification result V-Label-i stored by Category-Agent.

[0022] The information output module is used to classify and save the data output by the data alignment agent, Alignment-Agent.

[0023] Furthermore, the large language model GPT4 includes an Identifier GPT4 module and a Category GPT4 module;

[0024] The Identifier GPT4 module is used to output classification results based on the prompt words generated by the vehicle license plate classification agent Identifier-Agent and the vehicle license plate data.

[0025] The Category GPT4 module is used to classify the prompts generated by the Category-Agent based on vehicle type and output the classification results.

[0026] Furthermore, the Transformer modules in both the Identifier GPT4 module and the Category GPT4 module adopt the Causal Decoder architecture, with Full Language Modeling as the prediction target, to generate the classification result (0 / 1) of the data to be matched.

[0027] Furthermore, the specific process of step S4 includes:

[0028] 1) Input the multi-source vehicle information test set from the highway gantry scenario into the Controller-Agent; among which, the video capture vehicle holographic perception dataset includes vehicle license plate data (PA1, PA2…PA). n Vehicle type data (VA1, VA2…VA) n ), through time data (TA1, TA2…TA) n The RSU vehicle information perception dataset contains vehicle license plate data (PB1, PB2…PB). n Vehicle type data (VB1, VB2…VB) n ), through time data (TB1, TB2…TB) n The Controller-Agent breaks down the multi-source vehicle information data matching and fusion task into a vehicle license plate matching task and a vehicle type matching task; it combines the vehicle license plate data (PA1, PA2…PA) from the video capture vehicle holographic perception dataset with the RSU vehicle information perception dataset. n ) and (PB1,PB2…PB) n The data is input into the vehicle license plate classification agent Identifier-Agent, which combines the video-captured vehicle holographic perception dataset with the vehicle type data (VA1, VA2…VA) from the RSU vehicle information perception dataset. n ) and (VB1,VB2…VB n Input the vehicle type classification agent (Category-Agent);

[0029] 2) The Identifier-Agent inputs information such as the model's role, the question's background, contextual information, input data samples, expected output results, and special requirements into the Identifier GPT4 module; the Identifier-Agent combines the vehicle license plate data from the video-captured vehicle holographic perception dataset with the vehicle license plate data from the RSU vehicle information perception dataset, transforming it into the data format input to the large language model (PA). i ,PB i (), as the input of the data to be matched and fused into the IdentifierGPT4 module;

[0030] Category-Agent inputs information such as the model's role, question background, contextual information, input data samples, expected output results, and special requirements into the Category GPT4 module; Category-Agent combines vehicle type data from the video capture vehicle holographic perception dataset with vehicle type data from the RSU vehicle information perception dataset, transforming it into the data format input to the large language model (VA). i ,VB i ), which is input as the data to be matched into the Category GPT4 module;

[0031] 3) In the Identifier GPT4 module, the Transformer module of the GPT4 large language model adopts the CausalDecoder architecture, uses Full Language Modeling as the prediction target, and generates the classification result (0 / 1) of the data to be matched; the Identifier-Agent saves the classification result as the vehicle license plate data to be matched (PA). i ,PB i The label P-Label-i is entered, and then the next piece of data to be matched and fused (PA) is entered again. i+1 ,PB i+1 This process continues until all data is matched and fused, and then all data is output to the data alignment agent, Alignment-Agent.

[0032] In the Category GPT4 module, the Transformer module of the GPT4 large language model adopts the CausalDecoder architecture, uses Full Language Modeling as the prediction target, and generates classification results (0 / 1) for the data to be matched; the Category-Agent saves the classification results as fused vehicle type data to be matched (VA). i ,VB i) label V-Label-i, then repeatedly input the next piece of data to be matched and fused (VA i+1 ,VB i+1 ) until all data are matched and fused, and output all data to the data alignment agent Alignment-Agent;

[0033] 4) Alignment-Agent performs data pruning and data alignment on the classification result P-Label-i from Identifier-Agent and the classification result V-Label-i from Category-Agent;

[0034] If the first character appearing in the classification result is the number "1" or the character "yes", the number "1" is taken as the output result; otherwise, "0" is taken as the output result; Alignment-Agent aggregates the classification result P-Label-i from Identifier-Agent and the classification result V-Label-i from Category-Agent; if both output results of P-Label-i and V-Label-i are the number "1", the original data pair (PA i ,VA i ,TA i ), (PB i ,VB i ,TB i ) has its data classification label Label-i saved as the number "1", which represents that the data pair matches each other in terms of vehicle license plate information and vehicle type information; if the output results of P-Label-i and V-Label-i are not both the number "1", the original data pair (PA i ,VA i ,TA i ), (PB i ,VB i ,TB i ) has its data classification label Label-i saved as the number "0", which represents that the data pair does not match each other in terms of vehicle license plate information and vehicle type information; after the classification labels Label-i of all data pairs are generated, all data are output to the information output module.

[0035] 5) The information output module classifies and saves the data according to the data classification labels Label-i of the original data pairs (PA i ,VA i ,TA i ), (PB i ,VB i ,TB i );

[0036] If Label-i is 0, then the original data pairs (PA) of this group of data will be... i ,VA i ,TA i ), (PB i ,VB i ,TB i The label (Label-i) is saved to the failed matching dataset; if Label-i is 1, the next step of discrimination is performed; the information output module calculates the transit time data (TA) in the video capture vehicle holographic perception dataset. i With the pass-through time data TB in the RSU vehicle perception dataset i Perform the difference operation to calculate the time difference T between vehicle passages. i =(TA) i -TB i The information output module outputs T... i Perform a judgment, if T i If the range is (-30s, 120s), then the original data pair of this set of data (PA) i ,VA i ,TA i ), (PB i ,VB i ,TB i Save the label (Label-i) to the successfully matched dataset; if T i If the value exceeds the range of (-30s, 120s), then the original data will be paired with (PA). i ,VA i ,TA i ), (PB i ,VB i ,TB i Save the label (Label-i) to the dataset of questionable results.

[0037] In the present invention, in consideration of the fact that in a large-language large-language multi-agent structural model, the length of the output result of an agent used for processing subtasks may be greater than 1, or the output result may be the character "yes" which expresses the same meaning as the number "1", it is therefore necessary to perform data pruning and data alignment on the classification results of the subtasks, so as to improve the effectiveness of vehicle information matching and fusion. In the matching and fusion process, firstly, multi-source vehicle information data in a highway gantry scenario is input to a controller agent (Controller-Agent), the controller agent splits a multi-source vehicle information data matching task into a vehicle license plate matching task and a vehicle type matching task, inputs the vehicle license plate matching data to a vehicle license plate classification agent (Identifier-Agent), and inputs the vehicle type data to a vehicle type classification agent (Category-Agent); secondly, the vehicle license plate classification agent and the vehicle type classification agent respectively input prompt words to a large language model, and the large language model learns task content based on information such as the role played by the large language model in the prompt words, the question background, context information, input data examples, expected output results and special requirements; then, the vehicle license plate classification agent and the vehicle type classification agent sequentially input the to-be-matched and fused data into the large language model, the large language model performs prediction according to the requirements of the prompt words and the to-be-matched data, and outputs a matching and fusion result (0 / 1); finally, a data alignment agent (Alignment-Agent) summarizes the classification results of the vehicle license plate classification agent and the vehicle type classification agent, and obtains a final matching and fusion result through data pruning and data alignment. If the matching and fusion result is 0, the group of data is labeled as matching and fusion failure data; if the matching and fusion result is 1, the vehicle passage time difference of the group of data is calculated, otherwise the data is labeled as result uncertain data.

[0038] Beneficial effects: Compared with the prior art, the present invention, based on a large language model, performs matching training and prediction on multi-source vehicle information data for specific highway gantry scenarios, the matching and fusion accuracy reaches 98.50%, can effectively match the holographic perception data of vehicles captured by videos and the vehicle information perception data of RSU, and provides certain technical support for intelligent highway ETC toll collection systems. Description of Drawings

[0039] Figure 1 is an improved matching and fusion flow chart based on a large-language multi-agent structural model provided by the present invention;

[0040] Figure 2 is a structural diagram of the LLMMADA-MVIMF-EGS model provided by the present invention;

[0041] Figure 3 is a structural diagram of the Match-Pyramid-MVIMF-EGS model;

[0042] Figure 4 Here is a structural diagram of the ESIM-MVIMF-EG model;

[0043] Figure 5 This is a structural diagram of the BERT-MVIMF-EGS model. Detailed Implementation

[0044] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0045] This invention provides a method for matching and fusing multi-source vehicle information in highway gantry scenarios based on an improved multi-agent structure large language model, such as... Figure 1 As shown, it includes the following steps:

[0046] S1: Mining and labeling multi-source vehicle information perception data in highway gantry scenarios;

[0047] The dataset used in step S1 of this embodiment comes from four consecutive days of relevant data collected from four different gantries in January 2024. The multi-source vehicle information perception data in the highway gantry scene includes video capture vehicle holographic perception data and RSU vehicle information perception data.

[0048] Table 1 shows the video capture data of vehicle holographic perception and RSU vehicle information perception data collected by each gantry:

[0049] Table 1. Number of multi-source vehicle information perception data in highway gantry scenarios (unit: data items)

[0050]

[0051] The video capture data for vehicle holographic perception includes vehicle license plate information, vehicle passage time, and vehicle type information. The vehicle license plate information and vehicle passage time are preprocessed separately. The RSU vehicle perception data mainly includes vehicle license plate information, vehicle passage time, vehicle transaction results, and vehicle type information. The vehicle passage time, vehicle transaction results, and vehicle type information are preprocessed separately. First, the vehicle passage time information is uniformly converted into a date format that can be recognized and calculated by the Datetime module in Python. Second, the numbers in the vehicle transaction results column are converted into the corresponding transaction status. Finally, the numbers in the vehicle type information column are converted into corresponding textual descriptions of the vehicle type information.

[0052] This paper constructs a process for annotating multi-source vehicle information perception data in highway gantry scenarios by combining a character comparison-based text information matching fusion method with an unsupervised contrastive learning-based text information matching fusion method. Specifically, the process involves annotating multi-source vehicle information perception data in highway gantry scenarios. During the annotation process, data matching is primarily based on license plate features and time features, assuming that the license plate information characters of the same vehicle transaction data are allowed to have two or fewer errors. Regarding time features, given that vehicles with the same license plate may pass through the gantry multiple times on the same day, to reduce computational costs and improve the timeliness and accuracy of data annotation, a five-minute time window can be set before and after a vehicle's passage time to narrow the data retrieval range. For any data A in the video-captured vehicle holographic perception data... i First, extract the vehicle passage time data t. i The data is divided into five-minute intervals before and after the interval, and then data with time information matching the requirements of these intervals is searched in the RSU vehicle information perception dataset (B). i1 B i2 ...B ij The search scope was reduced from the entire RSU vehicle information perception dataset to j data points that met the time criteria.

[0053] In this embodiment, after annotation, a total of 462,199 positive samples with successful matching and fusion, 40,167 negative samples with failed matching and fusion, and 65,690 samples without vehicle license plates were obtained. Among the positive samples with successful matching and fusion, there were 442,643 samples with completely identical vehicle license plates and 19,556 samples with approximately identical vehicle license plates. The specific annotation results are shown in Table 2.

[0054] Table 2. Annotation results of multi-source vehicle information perception data in highway gantry scenarios (unit: data items)

[0055]

[0056]

[0057] S2: Based on the data labeled in step S1, construct a multi-source vehicle information matching and fusion dataset for highway gantry scenarios. The specific process is as follows:

[0058] S2-1: Data that simultaneously satisfies both vehicle license plate characteristics and vehicle type characteristics are considered positive samples, while data that do not simultaneously satisfy both vehicle license plate characteristics and vehicle type characteristics are considered negative samples; the number of data in each category is shown in Table 3.

[0059] Table 3. Number of sample data of each type in the multi-source vehicle information matching and fusion dataset in the highway gantry scenario.

[0060]

[0061] S2-2: Add label "1" to positive sample data pairs and label "0" to negative sample data pairs; the training set contains positive and negative samples of the datasets with completely identical and roughly identical license plates, while the validation and test sets contain only positive and negative samples of the datasets with roughly identical license plates; the ratio of positive to negative samples in the datasets is 1:1, the ratio of completely identical to roughly identical license plates in the training set is 1:1, and the ratio of positive to negative samples of roughly identical license plates in the training, validation, and test sets is 6:2:2; the number of samples of each type in the constructed training, validation, and test sets is shown in Table 4.

[0062] Table 4. Number of sample data of each type in the training set, validation set, and test set.

[0063]

[0064] S3: Based on the multi-source vehicle information matching process, construct a multi-source vehicle information matching and fusion model LLMMADA-MVIMF-EGS based on an improved multi-agent structure large language model;

[0065] Reference Figure 2 The multi-source vehicle information matching and fusion model LLMMADA-MVIMF-EGS includes a controller agent, a vehicle license plate classification agent Identifier agent, a vehicle type classification agent Category agent, a large language model GPT4, a data alignment agent Alignment agent, and an information output module.

[0066] The Controller-Agent is used to break down the multi-source vehicle information data matching and fusion task into a vehicle license plate matching task and a vehicle type matching task.

[0067] The vehicle license plate classification intelligent agent Identifier-Agent is used to generate prompt words and vehicle license plate data in highway gantry scenarios and feed them into the large language model GPT4.

[0068] The Category-Agent vehicle type classification agent is used to generate prompt words and vehicle type data in highway gantry scenarios and feed them into the large language model GPT4.

[0069] The large language model GPT4 is used to make predictions based on the requirements of prompt words and the data to be matched, and outputs the matching results to the vehicle license plate classification agent Identifier-Agent and the vehicle type classification agent Category-Agent respectively.

[0070] The data alignment agent Alignment-Agent is used to perform data trimming and alignment between the classification result P-Label-i stored by Identifier-Agent and the classification result V-Label-i stored by Category-Agent.

[0071] The information output module is used to classify and save the data output by the data alignment agent, Alignment-Agent.

[0072] The large language model GPT4 includes the Identifier GPT4 module and the Category GPT4 module;

[0073] The Identifier GPT4 module is used to output classification results based on the prompt words generated by the vehicle license plate classification agent Identifier-Agent and the vehicle license plate data.

[0074] The Category GPT4 module is used to classify the prompts generated by the Category-Agent based on vehicle type and output the classification results.

[0075] Both the Identifier GPT4 module and the Category GPT4 module's Transformer module adopt the CausalDecoder architecture, using Full Language Modeling as the prediction target to generate classification results (0 / 1) for the data to be matched.

[0076] S4: Based on the fusion dataset from step S2, the multi-source vehicle information matching and fusion results are output through the model LLMMADA-MVIMF-EGS to achieve multi-source vehicle information matching and fusion in the highway gantry scenario.

[0077] Reference Figure 1 and Figure 2 The specific process of outputting the multi-source vehicle information matching and fusion results through the model LLMMADA-MVIMF-EGS in step S4 includes:

[0078] 1) Input the multi-source vehicle information test set from the highway gantry scenario into the Controller-Agent; among which, the video capture vehicle holographic perception dataset includes vehicle license plate data (PA1, PA2…PA). nVehicle type data (VA1, VA2…VA) n ), through time data (TA1, TA2…TA) n The RSU vehicle information perception dataset contains vehicle license plate data (PB1, PB2…PB). n Vehicle type data (VB1, VB2…VB) n ), through time data (TB1, TB2…TB) n The Controller-Agent breaks down the multi-source vehicle information data matching and fusion task into a vehicle license plate matching task and a vehicle type matching task; it combines the vehicle license plate data (PA1, PA2…PA) from the video capture vehicle holographic perception dataset with the RSU vehicle information perception dataset. n ) and (PB1,PB2…PB) n The data is input into the vehicle license plate classification agent Identifier-Agent, which combines the video-captured vehicle holographic perception dataset with the vehicle type data (VA1, VA2…VA) from the RSU vehicle information perception dataset. n ) and (VB1,VB2…VB n Input the vehicle type classification agent (Category-Agent);

[0079] 2) The Identifier-Agent inputs information such as the model's role, the question's background, contextual information, input data samples, expected output results, and special requirements into the Identifier GPT4 module; the Identifier-Agent combines the vehicle license plate data from the video-captured vehicle holographic perception dataset with the vehicle license plate data from the RSU vehicle information perception dataset, transforming it into the data format input to the large language model (PA). i ,PB i (), as the input of the data to be matched and fused into the IdentifierGPT4 module;

[0080] Category-Agent inputs information such as the model's role, question background, contextual information, input data samples, expected output results, and special requirements into the Category GPT4 module; Category-Agent combines vehicle type data from the video capture vehicle holographic perception dataset with vehicle type data from the RSU vehicle information perception dataset, transforming it into the data format input to the large language model (VA). i ,VB i ), which is input as the data to be matched into the Category GPT4 module;

[0081] 3) In the Identifier GPT4 module, the Transformer module in the large language model GPT4 adopts a CausalDecoder architecture, takes Full Language Modeling as the prediction objective, and generates the classification result (0 / 1) of the data to be matched; Identifier-Agent saves the classification result as fused vehicle license plate data to be matched (PA i ,PB i ) as label P-Label-i, then repeatedly inputs the next fused data to be matched (PA i+1 ,PB i+1 ) until all data are matched and fused, and outputs all data to the data alignment agent Alignment-Agent;

[0082] In the Category GPT4 module, the Transformer module in the large language model GPT4 adopts a CausalDecoder architecture, takes Full Language Modeling as the prediction objective, and generates the classification result (0 / 1) of the data to be matched; Category-Agent saves the classification result as fused vehicle type data to be matched (VA i ,VB i ) as label V-Label-i, then repeatedly inputs the next fused data to be matched (VA i+1 ,VB i+1 ) until all data are matched and fused, and outputs all data to the data alignment agent Alignment-Agent;

[0083] 4) Alignment-Agent performs data pruning and data alignment on the classification result P-Label-i from Identifier-Agent and the classification result V-Label-i from Category-Agent;

[0084] If the first character appearing in the classification result is the number "1" or the character "yes", the number "1" is taken as the output result; otherwise, "0" is taken as the output result; Alignment-Agent summarizes the classification result P-Label-i from Identifier-Agent and the classification result V-Label-i from Category-Agent; if the output results of both P-Label-i and V-Label-i are the number "1", the original data pairs (PA i ,VA i ,TA i ), (PB i ,VBi ,TB i The data category label Label-i is saved as the number "1", indicating that the data pair matches in terms of vehicle license plate information and vehicle type information; if the output results of P-Label-i and V-Label-i are not all "1", then the original data pair (PA) is saved as "1". i ,VA i ,TA i ), (PB i ,VB i ,TB i The data category label Label-i is saved as the number "0", which means that the data pair matches each other in terms of vehicle license plate information and vehicle type information; after the category labels Label-i of all data pairs are generated, all data is output to the information output module.

[0085] 5) The information output module outputs data based on the original data (PA). i ,VA i ,TA i ), (PB i ,VB i ,TB i The data classification label Label-i categorizes and saves the data.

[0086] If Label-i is 0, then the original data pairs (PA) of this group of data will be... i ,VA i ,TA i ), (PB i ,VB i ,TB i The label (Label-i) is saved to the failed matching dataset; if Label-i is 1, the next step of discrimination is performed; the information output module calculates the transit time data (TA) in the video capture vehicle holographic perception dataset. i With the pass-through time data TB in the RSU vehicle perception dataset i Perform the difference operation to calculate the time difference T between vehicle passages. i =(TA) i -TB i The information output module outputs T... i Perform a judgment, if T i If the range is (-30s, 120s), then the original data pair of this set of data (PA) i ,VA i ,TA i ), (PB i ,VB i ,TB i Save the label (Label-i) to the successfully matched dataset; if Ti If the value exceeds the range of (-30s, 120s), then the original data will be paired with (PA). i ,VA i ,TA i ), (PB i ,VB i ,TB i Save the label (Label-i) to the dataset of questionable results.

[0087] Based on the above, in order to verify the superiority of the GPT4MA-MVIMF-EGS model provided by this invention, this embodiment constructs six models—Match-Pyramid-MVIMF-EGS, ESIM-MVIMF-EGS, BERT-MVIMF-EGS, BaiChuan2MA-MVIMF-EGS, ChatGLM3MA-MVIMF-EGS, and the GPT4MA-MVIMF-EGS of this invention—for matching experiments. The specific construction method is as follows:

[0088] 1. Construct a Match-Pyramid-MVIMF-EGS model for multi-source vehicle information matching and fusion in highway gantry scenarios based on convolutional neural networks. Specifically, the Match-Pyramid neural network treats the semantic similarity between words as the pixel similarity between images, capturing basic matching and fusion features between words by generating a word-level similarity matrix. Similarity can be viewed as a binary image ranging from 0 to 1, where each element's value represents whether corresponding word pairs are equivalent; alternatively, similarity can be defined as a grayscale image with real-valued values, in which case it is usually obtained by calculating the inner product or cosine similarity of word vectors. By applying convolution operations to this matching matrix, the matching fusion model can identify and extract meaningful local matching patterns. The structure of the Match-Pyramid-MVIMF-EGS model is as follows: Figure 3 As shown in Table 5, the model parameters are as follows. First, for text A and text B input to the Match-Pyramid-MVIMF-EGS model, it is assumed that each consists of 8 words, where text A contains (W1, W2, W3, W4, W5, W6, W7, W8) and text B contains (V1, V2, V3, V4, V5, V6, V7, V8). These are transformed into feature vectors of size (None, 64, 100) through the embedding layer. Second, the word W is compared... i With V jThe similarity between words is stored in the j-th row and i-th column of the Matching Matrix, representing word consistency, cosine similarity of word vectors, or inner product between word vectors, to generate a (None, 64, 64, 1) 2D convolutional layer. Then, the image processing steps using a convolutional neural network are performed, with convolution operations in the Matching Matrix, processed by two convolutional layers Conv2D_1 and Conv2D_2, generating a (None, 64, 64, 32) feature vector, which is then connected to a max pooling layer Max_Pooling2d_1 to generate a (None, 21, 21, 32) feature vector, and finally connected to a Flattening layer. The Layer generates a feature vector of size (None, 14112). Finally, the Match-Pyramid-MVIMF-EGS model outputs the matching and fusion results between text information through the processing of fully connected layers. The two dense layers Dense_1 and Dense_2 are connected to the Dropout layer to transform into a feature vector of size (None, 128). The dense layer Dense_3 is then connected to output the matching and fusion results between text A and text B.

[0089] Table 5 Match-Pyramid-MVIMF-EGS Model Parameters

[0090]

[0091] 2. The specific content of constructing the ESIM-MVIMF-EGS model for multi-source vehicle information matching and fusion in highway gantry scenarios based on recurrent neural networks is as follows: Enhanced Sequential Inference Model. ESIM is a model constructed by integrating an attention mechanism and a bidirectional long short-term memory network. This integration not only strengthens the ability to capture contextual information of sequential data, but also significantly improves the model's ability to interpret semantic connections between two texts by incorporating an attention mechanism. The ESIM-MVIMF-EGS model consists of four layers: Input Encoding, Local Inference Modeling, Inference Composition, and Prediction. In the input encoding structure, the ESIM-MVIMF-EGS model first divides the two input text information into premises and hypotheses, and converts each word in the text into a fixed-length word vector. These word vectors are processed based on a BiLSTM network to obtain the representation of each word in its context. The structure of the ESIM-MVIMF-EGS model is as follows: Figure 4As shown in Table 6, the model parameters are as follows. First, for text A and text B input to the ESIM-MVIMF-EGS model, the embedding layer processes them and transforms them into feature vectors of size (None, 64, 100). Second, the ESIM-MVIMF-EGS model extracts the contextual information from the feature vectors of text A and text B based on the BiLSTM network in the Bidirectional_1 layer and transforms it into feature vectors of size (None, 64, 128). Then, the ESIM-MVIMF-EGS model uses an attention mechanism to detect the relationship between text A and text B. After the integration calculation of the Concatenate_1 and Concatenate_2 layers, it is transformed into alignment features of size (None, 64, 512). Next, the feature vector of text A is connected to the BiLSTM network in the Bidirectional_2 layer, the Global_Average_Pooling1d_1 average pooling layer, and the Global_Max_P max pooling layer. In layer ooling1d_1, the feature vector of text B is connected to the BiLSTM network in layer Bidirectional_3, the average pooling layer Global_Average_Pooling1d_2, and the max pooling layer Global_Max_Pooling1d_2, fusing local information into a feature vector of size (None, 128). This is then connected to layer Concatenate_3 and layer Dropout_1, aggregating the feature vectors of text A and text B into a feature vector of size (None, 512). Finally, the ESIM-MVIMF-EGS model outputs the matching and fusion results between text information through fully connected layers. It connects two dense layers Dense_1 and Dense_2 to layer Dropout_2, transforming them into a feature vector of size (None, 128), and connects to dense layer Dense_3 to output the matching and fusion results of text A and text B.

[0092] Table 6. Parameters of each layer in the ESIM-MVIMF-EGS model

[0093]

[0094] 3. Construct a BERT-MVIMF-EGS model for multi-source vehicle information matching and fusion in highway gantry scenarios based on Transformer. Specifically, it involves Bidirectional Encoder Representations from Transformers. The BERT model is a pre-trained deep bidirectional representation model built on the Transformer architecture. This model is based on a bidirectional Transformer encoder and adopts two pre-training techniques: Masked Language Model (MLM) and Next Sentence Prediction (NSP), to extract semantic representations at the lexical and sentence levels, respectively. The structure of the constructed BERT-MVIMF-EGS model is as follows: Figure 5 As shown in Table 7, the model parameters are as follows. The model mainly consists of three parts: an input layer, an encoder layer, and an output layer. In the input layer, texts A and B input to the BERT-MVIMF-EGS model are first subdivided into the smallest unit, Token, through word segmentation and standardization. Then, classification tags [CLS] and delimiters [SEP] are added to each Token to complete the format conversion. Next, each Token undergoes three layers of processing: TokenEmbedding, Segment Embedding, and Position Embedding, to obtain a rich semantic representation vector integrating the lexical features, paragraph origin, and positional information of the token. Finally, the BERT-MVIMF-EGS model integrates the outputs of these three layers, combining the lexical features, paragraph origin, and positional information of the token, to construct a semantic representation vector of size (None, None, 768). This vector is then used as input to the encoder layer for deeper semantic understanding and processing. In the encoder layer, the BERT-MVIMF-EGS model contains 12 Transformer encoder modules. Each Transformer encoder module includes a multi-head self-attention layer and a feedforward neural network, designed to encode and model the input text information. The multi-head self-attention layer consists of multiple self-attention mechanisms, where self-attention refers to obtaining multiple tensor representations through different matrix multiplications when performing attention calculations on the input. Each self-attention mechanism independently learns contextual information to capture different contextual features, thereby enriching the contextual modeling capabilities of the multi-head self-attention layer.

[0095] Table 7 BERT-MVIMF-EGS Model Parameters

[0096]

[0097]

[0098] In this embodiment, based on the constructed multi-source vehicle information matching and fusion dataset in the highway gantry scene, matching experiments were conducted using six models: Match-Pyramid-MVIMF-EGS, ESIM-MVIMF-EGS, BERT-MVIMF-EGS, GPT4MA-MVIMF-EGS, BaiChuan2MA-MVIMF-EGS, and ChatGLM3MA-MVIMF-EGS. The detection results of the test set are shown in Table 8. It can be seen that the comprehensive detection performance of the GPT4MA-MVIMF-EGS model provided by this invention is better than that of the other five models. The first three are matching models built on convolutional neural networks, recurrent neural networks and Transformer architectures respectively. BaiChuan2MA-MVIMF-EGS and ChatGLM3MA-MVIMF-EGS are two other matching models built on a single agent structure of large language. Because GPT4MA-MVIMF-EGS is a multi-agent structure matching model based on the GPT4 large language model, (1) compared with convolutional neural networks, recurrent neural networks and Transformer architectures, the large language model can understand the needs of classification tasks faster and more accurately; (2) GPT4 has advantages over other large language models in terms of model structure, parameters and training set, and its performance is also better. Therefore, the GPT4MA-MVIMF-EGS model is more suitable for matching multi-source vehicle information in highway gantry scenarios. The model uses the Causal Decoder architecture and Full Language Modeling as the prediction target to generate the classification results of the data to be matched.

[0099] Table 8. Analysis of Experimental Results

[0100]

[0101] In addition, to verify the effectiveness of the method of the present invention, an ablation experiment was conducted in this case. The experimental results are shown in Table 9. The experimental results show that the optimization method of the present invention can significantly improve the matching accuracy of large language models, which demonstrates the effectiveness of the method of the present invention.

[0102] Table 9. Experimental results of multi-source vehicle information matching, fusion, and ablation in highway gantry scenarios based on large language models.

[0103]

Claims

1. A method for matching and fusing multi-source vehicle information in highway gantry scenarios based on an improved multi-agent structural large language model, characterized in that, Includes the following steps: S1: Mining and labeling multi-source vehicle information perception data in highway gantry scenarios; S2: Based on the data labeled in step S1, construct a multi-source vehicle information matching and fusion dataset in the highway gantry scenario; S3: Based on the multi-source vehicle information matching process, construct a multi-source vehicle information matching and fusion model LLMMADA-MVIMF-EGS based on an improved multi-agent structure large language model; S4: Based on the fusion dataset from step S2, the multi-source vehicle information matching and fusion results are output through the model LLMMADA-MVIMF-EGS to achieve multi-source vehicle information matching and fusion in the highway gantry scenario; The multi-source vehicle information matching and fusion model LLMMADA-MVIMF-EGS in step S3 includes a controller agent, a vehicle license plate classification agent Identifier agent, a vehicle type classification agent Category agent, a large language model GPT4, a data alignment agent Alignment agent, and an information output module. The controller-Agent is used to split the multi-source vehicle information data matching and fusion task into a vehicle license plate matching task and a vehicle type matching task. The vehicle license plate classification agent Identifier-Agent is used to generate prompt words and vehicle license plate data in the highway gantry scenario and feed them into the large language model GPT4. The vehicle type classification agent, Category-Agent, is used to generate prompt words and vehicle type data in the highway gantry scenario and feed them into the large language model GPT4. The large language model GPT4 is used to predict the matching data based on the prompt words and output the matching results to the vehicle license plate classification agent Identifier-Agent and the vehicle type classification agent Category-Agent, respectively. The data alignment agent Alignment-Agent is used to perform data trimming and alignment between the classification result P-Label-i stored by Identifier-Agent and the classification result V-Label-i stored by Category-Agent. The information output module is used to classify and save the data output by the data alignment agent Alignment-Agent; The large language model GPT4 includes an Identifier GPT4 module and a Category GPT4 module; The Identifier GPT4 module is used to output classification results based on the prompt words generated by the vehicle license plate classification agent Identifier-Agent and the vehicle license plate data. The Category GPT4 module is used to classify the prompt words generated by the Category-Agent based on vehicle type and output the classification results based on the vehicle type data. Both the Identifier GPT4 module and the Category GPT4 module use the CausalDecoder architecture in their Transformer modules, with Full Language Modeling as the prediction target, to generate classification results for the data to be matched.

2. The method for matching and fusing multi-source vehicle information in highway gantry scenarios based on an improved multi-agent structural large language model as described in claim 1, characterized in that, In step S1, the multi-source vehicle information perception data in the highway gantry scene includes video capture vehicle holographic perception data and RSU vehicle information perception data. The video capture vehicle holographic perception data includes vehicle license plate information, vehicle passage time, and vehicle type information, and the vehicle license plate information and vehicle passage time are preprocessed separately; the RSU vehicle perception data includes vehicle license plate information, vehicle passage time, vehicle transaction result, and vehicle type information, and the vehicle passage time, vehicle transaction result, and vehicle type information are preprocessed separately.

3. The method for matching and fusing multi-source vehicle information in highway gantry scenarios based on an improved multi-agent structural large language model as described in claim 1, characterized in that, The specific process of step S4 includes: 1) Input the multi-source vehicle information test set from the highway gantry scenario into the Controller-Agent; among which, the video capture vehicle holographic perception dataset includes vehicle license plate data. Vehicle type data ), through time data The RSU vehicle information perception dataset contains vehicle license plate data. Vehicle type data ), through time data The Controller-Agent breaks down the multi-source vehicle information data matching and fusion task into a vehicle license plate matching task and a vehicle type matching task; it combines the vehicle license plate data from the video capture vehicle holographic perception dataset with the RSU vehicle information perception dataset. )and The data is input into the vehicle license plate classification agent Identifier-Agent, which combines the vehicle type data from the video-captured vehicle holographic perception dataset with the vehicle type data from the RSU vehicle information perception dataset. )and Input the vehicle type classification agent (Category-Agent); 2) The Identifier-Agent inputs the model's role, the question's background, contextual information, input data samples, expected output results, and special requirements into the Identifier GPT4 module; the Identifier-Agent combines the vehicle license plate data from the video-captured vehicle holographic perception dataset with the vehicle license plate data from the RSU vehicle information perception dataset, transforming it into the data format input to the large language model. , The data to be matched and fused is input into the IdentifierGPT4 module; Category-Agent inputs the model's role, question background, contextual information, input data samples, expected output results, and special requirements into the Category GPT4 module; Category-Agent combines vehicle type data from the video capture vehicle holographic perception dataset with vehicle type data from the RSU vehicle information perception dataset, transforming it into the data format input to the large language model. , The data to be matched is input into the Category GPT4 module; 3) In the Identifier GPT4 module, the Transformer module of the GPT4 large language model adopts the CausalDecoder architecture, uses Full Language Modeling as the prediction target, and generates the classification results of the data to be matched; the Identifier-Agent saves the classification results as the vehicle license plate data to be matched and fused. , The label P-Label-i is entered, and then the next piece of data to be matched and fused is entered again. , This process continues until all data is matched and fused, and then all data is output to the data alignment agent, Alignment-Agent. In the Category GPT4 module, the Transformer module of the GPT4 large language model adopts a Causal Decoder architecture, uses Full Language Modeling as the prediction target, and generates classification results for the data to be matched. The Category-Agent saves the classification results as vehicle type data to be matched and fused. , The label is V-Label-i, and then the next piece of data to be matched and fused is entered repeatedly. , This process continues until all data is matched and fused, and then all data is output to the data alignment agent, Alignment-Agent. 4) Alignment-Agent performs data trimming and alignment on the classification results P-Label-i of Identifier-Agent and V-Label-i of Category-Agent; If the first character occurring in the classification result is the number "1" or the character "yes", the number "1" is taken as the output result; otherwise, "0" is taken as the output result; Alignment-Agent aggregates the classification result P-Label-i from Identifier-Agent and the classification result V-Label-i from Category-Agent; if the output results of both P-Label-i and V-Label-i are the number "1", the original data pair , , 、 the data classification label Label-i is saved as the number "1", which represents that the data pair matches each other in terms of vehicle license plate information and vehicle type information; if the output results of P-Label-i and V-Label-i are not both the number "1", the original data pair , , 、 the data classification label Label-i is saved as the number "0", which represents that the data pair does not match each other in terms of vehicle license plate information and vehicle type information; after the classification labels Label-i of all data pairs are generated, all data are output to the information output module; 5) The information output module processes the raw data... , , , The data classification label Label-i categorizes and saves the data; If Label-i is 0, then the original data of this group will be... , , , The label (Label-i) is saved to the dataset of failed matches; if Label-i is 1, the next step of discrimination is performed; the information output module calculates the transit time data in the video capture vehicle holographic perception dataset. With the transit time data in the RSU vehicle perception dataset Perform the difference to calculate the time difference between vehicle passages. =( The information output module supports... Perform a judgment, if If the range is (-30s, 120s), then the original data of this set of data is compared with... , , , Save the label (Label-i) to the successfully matched dataset; if If the value exceeds the range of (-30s, 120s), then the original data will be recalculated. , , , Save the label (Label-i) to the dataset of questionable results.

Images