Multi-modal large model training system and method applied to traffic field

Abstract

The invention provides a multi-modal large model training system and method applied to the traffic field. The system comprises a large language model module, a feature alignment module, a multi-modal module and a user interaction module. Wherein the large language model module adopts a decoder architecture improved in Transform, and traffic text data and non-traffic text data which are collected in advance are used for training to obtain a language model; the feature alignment module extracts high-level semantic features of an image through an image encoder, and performs image and text alignment training in combination with a language model to obtain a text image alignment model; the multi-modal module adopts multi-modal instruction data containing a text image question and answer pair and combines the text image alignment model to carry out fusion training to obtain a multi-modal large model; and the user interaction module is used for receiving a user request, performing reasoning prediction through a multi-modal large model, and providing prediction and interaction results. The method solves various complex and tedious problems in the traffic field, and has excellent expansibility and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to a multimodal large model training system and method applied to the field of transportation. Background Art

[0002] Large models and multimodal large model technology is a hot topic in the current field of artificial intelligence. Larger neural network models are usually used to improve the performance and representation of tasks, and information from multiple input modalities is combined for joint learning and reasoning.

[0003] As computing power and data sets grow, researchers continue to work hard to design and train larger neural network models. These models have more parameters and more layers to better learn and represent complex features. The most famous examples are models such as GPT, BERT, and AlphaGo. These large models have demonstrated excellent performance in tasks such as natural language processing, speech recognition, and computer vision. At the same time, researchers are also faced with the challenges of computing resources and time required to train and deploy large models.

[0004] Multimodal large models further develop the concept of large models. They can process multiple input modalities (such as text, images, audio, etc.) at the same time, so as to understand and process information more comprehensively. The training and design of multimodal large models involve knowledge from multiple fields, such as computer vision, natural language processing, and speech processing. These models can jointly learn features from different modalities, take into account the correlation between modalities, and show better performance than unimodal models on multimodal tasks.

[0005] In recent years, the development of artificial intelligence technology in the field of transportation has made significant progress and led to many revolutionary changes, such as using autonomous driving technology to identify and understand the surrounding environment, planning and executing the vehicle's driving path, using computer vision technology to accurately identify pedestrians and traffic signs, improving traffic safety, etc. With the development of big model technology, all walks of life have been transformed by big model technology.

[0006] Existing technologies applied in the field of transportation usually lack generalization capabilities and often have an over-reliance on specific scenarios and problems. This makes these models poorly generalized and difficult to adapt to changes and new problems in different traffic environments. In addition, it is difficult to process large-scale complex data. Traditional artificial intelligence technologies have difficulty in processing large-scale and complex traffic data. The scale of traffic data in large cities is huge, and traditional technologies may not be able to process and analyze it efficiently. Existing technologies have limited interpretability, and some traditional artificial intelligence technologies often lack interpretability in the field of transportation. This makes the decision-making process of the model difficult to understand and explain, limiting the credibility and acceptability of the model in some key application areas. Summary of the invention

[0007] The technical problem to be solved by the present invention is how to obtain a reliable reasoning interaction model in the field of transportation. The present invention proposes a multimodal large model training system applied in the field of transportation, comprising:

[0008] The large language model module adopts the improved decoder architecture in Transformer, uses the pre-collected traffic text data and non-traffic text data, and trains the language model according to the preset ratio;

[0009] A feature alignment module extracts high-level semantic features of an image through an image encoder, and performs image and text alignment training in combination with the language model to obtain a text-image alignment model;

[0010] The multimodal module uses multimodal instruction data containing text-image question-answer pairs and combines it with the text-image alignment model for fusion training to obtain a multimodal large model;

[0011] The user interaction module is used to receive user requests, perform inference and prediction through a multimodal large model, and provide prediction and interaction results.

[0012] According to some embodiments of the present invention, in the improved decoder architecture, the absolute position coding in the architecture is replaced by NTK-AliBi position coding interpolation, the NTK-ALiBi coding encodes the position space into the frequency domain space, the coefficient of the bias term is the frequency domain value, and the improved bias term coefficient m h for:

[0013] m h =1 / (2( 8h / H )*a(( h - 1 )( H - 1 )));

[0014] Among them, h is the current attention head index, H is the number of attention heads, and a is the scaling factor.

[0015] In some embodiments of the present invention, in the improved decoder architecture, the MQA structure in the architecture is replaced with a GQA structure, the LayerNorm in the architecture is replaced with RMSNorm, and a Dropout layer is added after the first fully connected layer in the Feed Forward of the architecture.

[0016] According to some embodiments of the present invention, the traffic text data includes:

[0017] Scientific and technological literature, including scientific and technological papers and research reports related to the field of transportation at home and abroad;

[0018] Industry information, transportation industry development reports, transportation industry related documents, traffic safety education documents, domestic transportation related laws and regulations;

[0019] Engineering construction information, domestic and international transportation infrastructure;

[0020] The non-traffic text data includes: high-quality Internet text data, works, and papers;

[0021] When performing model training, the preset ratio of the traffic text data to the non-traffic text data is 1:5.

[0022] According to an embodiment of the present invention, a multimodal large model training method applied to the transportation field is provided. The method adopts the multimodal large model training system applied to the transportation field as described above to perform multimodal large model training. The method includes:

[0023] S10 uses the improved decoder architecture in Transformer and uses pre-collected traffic text data and non-traffic text data to train a language model according to a preset ratio.

[0024] S20, extracting high-level semantic features of the image through an image encoder, and performing image and text alignment training in combination with the language model to obtain a text-image alignment model;

[0025] S30, using multimodal instruction data including text-image question-answer pairs and combining them with the text-image alignment model to perform fusion training to obtain a multimodal large model;

[0026] S40, receiving user requests, performing inference and prediction through a multimodal large model, and providing prediction and interaction results.

[0027] According to some embodiments of the present invention, step S10 specifically includes:

[0028] A10, collect traffic-related text data and non-traffic text data in advance, and prepare pre-training text data according to a preset ratio of 1:5;

[0029] A20, cleans, de-duplicates, deletes invalid data, and vectorizes the pre-trained text data;

[0030] A30, uses the improved decoder architecture in Transformers to build the model network;

[0031] A40, randomly initializes the parameters of each layer of the model network using a preset learning rate;

[0032] A50, inputs the vectorized training text data into the model network and calculates the activation value and output of each layer;

[0033] A60, calculates the cross entropy loss between the model network output and the true value;

[0034] A70 uses the back-propagation algorithm to propagate the loss back to calculate the parameter update gradient;

[0035] A80, uses the optimizer to update the parameters of each layer of the model network according to gradient descent;

[0036] A90, repeat A40-A80, and continuously update network parameters until convergence;

[0037] A100, saves the checkpoints of each epoch to form a pre-training model library;

[0038] A110, evaluate the models in the pre-trained model library and select the best model as the language model.

[0039] In some embodiments of the present invention, step S20 specifically includes:

[0040] B10, obtain text-image pair training data;

[0041] B20, vectorize the acquired text-image pair training data;

[0042] B30, aligns the visual features extracted by the visual encoder and the text embedding obtained by the language model embedding layer through the Q-Former architecture;

[0043] B40, inputting the vectorized image data into the visual encoder, extracting the text visual information through the projection layer and inputting it into the language model, and the language model generates an image representation;

[0044] B50, the difference between the image representation and the original text calculated using the cross entropy loss function;

[0045] B60, uses the back-propagation algorithm to propagate the loss back to calculate the parameter update gradient;

[0046] B70, use the optimizer to update the parameters of each layer of the network according to gradient descent;

[0047] B80, repeat B40-B70, and continuously update network parameters until convergence;

[0048] B90, saves the checkpoint of each epoch to form a feature alignment model library;

[0049] B100, evaluates the models in the feature alignment model library and selects the best model as the text-image alignment model.

[0050] According to some embodiments of the present invention, step S30 specifically includes:

[0051] C10, obtains traffic-related image data, combines it with human preference questions and answers, and generates corresponding graphic and text instruction interaction data through GPT-4;

[0052] C20, vectorization processing of the acquired graphic and text instruction interaction data;

[0053] C30, inputs the vectorized text-image interaction data into the text-image alignment model for further training, and calculates the activation value and output of each layer;

[0054] C40, calculates the cross entropy loss between the output of the text-image alignment model and the true value;

[0055] C50, using the back-propagation algorithm to propagate the loss back to calculate the parameter update gradient;

[0056] C60, uses the optimizer to update the parameters of each layer of the network according to gradient descent;

[0057] C70, repeat C30-C60, and continuously update network parameters until convergence;

[0058] C80, saves the checkpoint of each epoch to form a multimodal model library;

[0059] C90, evaluates the models in the multimodal model library and selects the best model as the multimodal large model.

[0060] In some embodiments of the present invention, step S40 specifically includes:

[0061] D10, issuing user request instructions through the webui interface;

[0062] D20, the WebUI interface passes the user request instruction to the model inference module;

[0063] D30, the model reasoning module performs reasoning through a multimodal large model;

[0064] D40, the inference result is returned to the WebUI interface;

[0065] D50, the WebUI interface generates a friendly response based on the reasoning result and completes an interaction;

[0066] D60, repeat steps D10-D50 to realize the interactive dialogue between the user and the multimodal large model.

[0067] The instruction reasoning interaction method in the traffic field based on a multimodal large model according to an embodiment of the present invention includes:

[0068] receiving user instructions for the traffic picture;

[0069] Reasoning the traffic picture and the user instruction through a pre-trained multimodal large model;

[0070] Return the inference result to complete the interaction;

[0071] The multimodal large model is obtained by training using the multimodal large model training method applied to the transportation field as described above.

[0072] The present invention has the following beneficial effects:

[0073] The present invention proposes a multimodal large model training method in the field of transportation, innovatively optimizes the Transformers-decoder architecture, accelerates model training and improves performance. The present invention uses a multimodal large model to solve various complex and cumbersome problems in the field of transportation. Compared with the existing solutions, the present invention implements an end-to-end multimodal large model training method, which can achieve deep fusion of different modes and handle open domain problems. It also has excellent scalability and robustness, and is the best solution to achieve the purpose. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A diagram of a multimodal large model training and application architecture according to an embodiment of the present invention;

[0075] Figure 2 is a schematic diagram of an improved decoder structure according to an embodiment of the present invention;

[0076] Figure 3 A schematic diagram showing a comparison of the Attention calculation methods of MQA and GQA according to an embodiment of the present invention;

[0077] Figure 4 A schematic diagram of a graphic-text feature alignment structure according to an embodiment of the present invention;

[0078] Figure 5Schematic diagram of a user interaction process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the present invention is described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0080] The description of the method flow in the specification of the present invention and the steps of the flowchart in the drawings of the specification of the present invention do not have to be strictly executed according to the step numbers, and the method steps can be executed in a different order. Moreover, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0081] The present invention applies large model technology to the field of transportation. Compared with traditional methods, the multimodal large model described in the present invention has the ability to understand semantic intent, the ability to learn context and the ability to have multiple rounds of dialogue, the ability to generate content, and the ability of complex logical reasoning based on thought chains. The present invention is committed to playing a practical value in the real transportation industry, integrating information through the two modes of text and image, and realizing functions such as intelligent consulting assistant, traffic safety education, traffic accident reporting and analysis. The present invention uses a multimodal large model to solve the problems of poor generalization ability of traditional models in traffic scenes, inability to process large-scale complex data, and limited interpretability. For example, in a specific application example, the user can input instructions on whether there are any violations of traffic laws for a specific picture taken by a road camera, and the multimodal large model will give the reasoning interaction results, such as outputting the existence of traffic violations such as running red lights and pressing solid lines and the corresponding pictures.

[0082] This invention describes in detail the training method and practical application of a large multi-modal model in the field of transportation. Figure 1 As shown, the multimodal large model training system applied to the transportation field of the present invention includes: a large language model module, a text-image feature alignment module, a multimodal module, and a user interaction module.

[0083] The large language model module adopts the Transformer architecture and obtains a language model through pre-training on large-scale unsupervised text data. The large language model trained by this module will be used in the text-image feature alignment module.

[0084] The text-image feature alignment module is based on the language model trained by the large language model module, combined with the image encoder to extract high-level semantic features of the image, and jointly encodes the image and text to obtain the visual semantic representation of the image. The model obtained by feature alignment training will be used in the multimodal module.

[0085] The multimodal module combines the model trained by the feature alignment module with multimodal instruction fine-tuning data containing text-image question-answer pairs, and integrates the training through the Transformer architecture. This module combines the large language model module and the feature alignment module for joint training, and achieves deep interaction and understanding between different modalities by establishing a correlation between language and vision.

[0086] The user interaction module accepts user requests and combines the multimodal large model trained by the multimodal module to perform inference and prediction, and provides prediction and interaction results to achieve a friendly human-computer interaction experience.

[0087] The following is a detailed introduction to each module:

[0088] 1. Large language model module;

[0089] The large language model module is mainly used for unsupervised text pre-training to obtain a generative large language model. The steps of this module are as follows:

[0090] A10, prepare pre-training corpus: In this stage, traffic-related text data and other text data from the Internet are collected. According to the conclusion obtained from experimental verification, the ratio of traffic text and other text data is about 1:5, and the large model has the best effect. The present invention collects a total of 80G of pre-training corpus, which comes from:

[0091] Scientific and technological literature: including domestic and foreign scientific and technological papers and research reports related to the field of transportation;

[0092] Industry information: transportation industry development reports, transportation industry related documents, traffic safety education documents, domestic transportation related laws and regulations;

[0093] Engineering construction information: domestic and international transportation infrastructure;

[0094] Others: high-quality text data, books, and papers on the Internet;

[0095] A20, data preprocessing: clean the collected raw data, deduplicate it, delete invalid data, and finally vectorize it as the input of the network.

[0096] A30, building a network model structure: This invention only adopts the improved decoder architecture in Transformers. The entire network uses a 40-layer decoder architecture. The decoder architecture in Transformers includes an embedding layer (Input & Positional Embedding Layer), an attention layer (Masked Multi-Head Attention Layer), a normalization layer (Add & Norm Layer), and a feedforward neural network layer (Feed Forward).

[0097] like Figure 2 As shown, the present invention improves the decoder architecture, and the specific improvements are as follows:

[0098] 1) NTK (Neural Tangent Kernel)-ALiBi interpolation;

[0099] Due to computing power and video memory bottlenecks, the maximum input length of large models is usually set to 2k or 4k during training, but there is a need to input longer context during reasoning. This requires large models to have stronger memory and splice historical conversations into current conversations to reduce forgetting.

[0100] The present invention improves the absolute position coding in the decoder into NTK-AliBi position coding interpolation. NTK-ALiBi coding encodes the position space into the frequency domain space, and the coefficient of the bias term is the frequency domain value. The improved bias term coefficient is:

[0101] m h =1 / (2( 8h / H )*a(( h - 1 )( H - 1 )));

[0102] Where h is the current attention head index, H is the number of attention heads, and a is the scaling factor. The improved NTK-ALiBi can achieve high-frequency resolution unchanged and low-frequency field of view enlarged. When h = 1, the field of view is small, which is a high-frequency situation. Bias term m h =1 / 2 (8-H) The same as the original bias coefficient, equivalent to high-frequency extrapolation, the high-frequency field of view resolution remains unchanged. When h = H, the field of view is larger, which is the low-frequency case. Bias term m h =1 / (2 8 *a), which is reduced by a times based on the original bias, which is equivalent to interpolating the encoding position, and the low-frequency field of view becomes a times larger.

[0103] 2)Grouped-queryAttention(GQA);

[0104] This paper replaces the Multi-head attention (MQA) structure in the decoder with GQA. When calculating Attention, the query is grouped and K (Key) and V (value) are shared within the group to improve the performance of the attention mechanism and the speed of reasoning calculation. Figure 3 shown.

[0105] 3) RMSNorm (Root Mean Square LayerNormalization);

[0106] The present invention replaces LayerNorm in the decoder architecture with RMSNorm, and the formula is as follows:

[0107]

[0108] Compared with LayerNorm, RMSNorm removes the mean subtraction part, which greatly reduces the amount of calculation.

[0109] 4) Feed Forward;

[0110] Feed Forward in the Transformer-decoder architecture contains two layers of fully connected layers. After the first fully connected layer is dimensioned, the feature space is high and overfitting is prone to occur. In addition, graphic data is difficult to obtain and costly. The present invention adds a Dropout layer after the first fully connected layer in Feed Forward to alleviate this problem.

[0111] A40, initialization of model parameters: After the data and network structure are prepared, the parameters of each layer of the network are randomly initialized using a smaller learning rate;

[0112] A50, forward propagation: send the vectorized data into the network and calculate the activation value and output of each layer;

[0113] A60, calculate the loss function: calculate the cross entropy loss between the network output and the true value;

[0114] A70, back propagation: Use the back propagation algorithm to propagate the loss back to calculate the parameter update gradient;

[0115] A80, update parameters: use the optimizer (Adamw) to update the parameters of each layer of the network according to gradient descent;

[0116] A90, repeat A40-A70, and continuously update network parameters until convergence;

[0117] A100, saves the checkpoints of each epoch to form a pre-training model library;

[0118] A110, evaluates the models in the pre-trained model library and selects the best model.

[0119] 2. Text-image feature alignment module;

[0120] The text-image feature alignment module mainly performs joint encoding of text and image to extract high-level semantic features of the image. The steps of this module are as follows:

[0121] B10, prepare training data: The text-image feature alignment data of this module comes from general data containing images and corresponding texts on the Internet. In addition, the present invention also uses GPT-4 to generate a small amount of text-image pair data related to the transportation field;

[0122] B20, data preprocessing: vectorization of text-image data; pre-trained model: the model is trained by the large language model module; visual encoder: the present invention uses CLIP's ViT-L / 14, which is used as a visual encoder to extract visual features;

[0123] B30, feature alignment architecture: This stage of training introduces the Q-Former architecture of BLIP2, freezes the parameters of the visual encoder and the pre-trained model, and only trains the parameters of the Q-Former. The Q-Former architecture aligns the visual features extracted by the visual encoder with the text embedding obtained by the pre-trained model embedding layer. Figure 4 shown.

[0124] B40, forward propagation: the vectorized image data is input into the visual encoder, the text visual information is extracted through the projection layer and input into the pre-trained model, the pre-trained model combines the input to perform next token prediction and generate image representation;

[0125] B50, calculate the loss function: use the cross entropy loss function to calculate the difference between the image representation generated by the network and the original text;

[0126] B60, back propagation: Use the back propagation algorithm to propagate the loss back to calculate the parameter update gradient;

[0127] B70, update parameters: use the optimizer (Adamw) to update the parameters of each layer of the network according to gradient descent;

[0128] B80, repeat B40-B70, and continuously update network parameters until convergence;

[0129] B90, saves the checkpoint of each epoch to form a feature alignment model library;

[0130] B100, evaluates the models in the feature alignment model library and selects the best model.

[0131] 3. Multimodal module;

[0132] The multimodal module is mainly used to establish the relationship between language and vision, and to achieve deep interaction and understanding between different modalities. The steps of this module are as follows:

[0133] C10, prepare training data: In this stage, we collected high-quality traffic-related image data from the Internet, combined it with human preference questions and answers, and sent it to GPT-4 to generate corresponding graphic and text instruction interaction data. The data quality was manually checked and rewritten into multiple rounds of dialogue;

[0134] C20, data preprocessing: vectorization of graphic and text data;

[0135] Multimodal architecture: The multimodal architecture has the same network structure as the large language model module and the text-image feature alignment module. In this stage, we will continue training based on the text-image alignment model and combined with multimodal instruction data.

[0136] C30, forward propagation: the vectorized image and text instruction data is sent to the text-image alignment model for further training, and the activation value and output of each layer are calculated;

[0137] C40, calculate the loss function: calculate the cross entropy loss between the network output and the true value;

[0138] C50, back propagation: Use the back propagation algorithm to propagate the loss back to calculate the parameter update gradient;

[0139] C60, update parameters: use the optimizer (Adamw) to update the parameters of each layer of the network according to gradient descent.

[0140] C70, repeat C30-C60, and continuously update network parameters until convergence;

[0141] C80, saves the checkpoint of each epoch to form a multimodal model library;

[0142] C90, evaluates the models in the multimodal model library and selects the best model;

[0143] 4. User interaction module;

[0144] The user interaction module is mainly used to realize the reasoning and user-friendly interaction of the multimodal large model. This module consists of three parts: Webui, model reasoning, and multimodal model. The Webui interface is a bridge connecting users and multimodal models. Users can interact with the multimodal model directly through the Webui interface. The multimodal model is trained through the large language model module, text-image alignment module, and multimodal module. The model reasoning part is used to improve the model throughput and response speed, and improve the user interaction experience. Users can use the interaction module to realize functions such as intelligent consulting assistant, traffic safety education, traffic accident reporting and analysis. The user interaction steps are as follows: Figure 5 shown.

[0145] D10, the user issues a command through the webui interface;

[0146] D20, the WebUI interface passes the user request to the model inference part;

[0147] D30, the model reasoning part uses pipeline and batch processing technology to perform efficient reasoning on user requests;

[0148] D40, the inference result is returned to the WebUI interface;

[0149] D50, the WebUI interface generates a friendly reply based on the reasoning result and presents it to the user, completing an interaction;

[0150] D60, repeat the above process to enable users to interact with the multimodal model anytime and anywhere.

[0151] Compared with the prior art, the present invention has the following advantages:

[0152] The present invention proposes a multimodal large model training method in the field of transportation, innovatively optimizes the Transformers-decoder architecture, accelerates model training and improves performance. The present invention uses a multimodal large model to solve various complex and cumbersome problems in the field of transportation. Existing solutions include:

[0153] Multiple unimodal models implement different functions: language models are used to process text requests, visual recognition models are used to process images, knowledge graphs are used to match queries, etc. However, it is difficult to deeply integrate the unimodal models and achieve deep cross-modal understanding.

[0154] Using traditional machine learning methods: manually designing features using feature engineering, and using classification, regression and other algorithms to query and answer questions in the transportation field. However, it is difficult to handle complex and diverse data, and has low scalability and robustness.

[0155] Using traditional dialogue systems: Simple dialogues are achieved based on rule matching, but they cannot handle complex problems in open domains and it is difficult to achieve true human-computer dialogue.

[0156] Compared with existing solutions, the present invention implements an end-to-end multimodal large model training method, which can achieve deep fusion of different modalities and handle open domain problems. It also has excellent scalability and robustness, and is the best solution to achieve the goal.

[0157] Through the description of the specific implementation methods, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the predetermined purpose should be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not intended to limit the present invention.

Claims

1. A multi-modal large model training system applied in the field of transportation, characterized in that: include: The large language model module adopts the improved decoder architecture in Transformer, uses the pre-collected traffic text data and non-traffic text data, and trains the language model according to the preset ratio; A feature alignment module extracts high-level semantic features of an image through an image encoder, and performs image and text alignment training in combination with the language model to obtain a text-image alignment model; The multimodal module uses multimodal instruction data containing text-image question-answer pairs and combines it with the text-image alignment model for fusion training to obtain a multimodal large model; The user interaction module is used to receive user requests, perform inference and prediction through a multimodal large model, and provide prediction and interaction results.

2. The multimodal large model training system applied to the transportation field according to claim 1 is characterized in that: In the improved decoder architecture, the absolute position encoding in the architecture is replaced by NTK-AliBi position encoding interpolation. NTK-ALiBi encoding encodes the position space into the frequency domain space, and the coefficient of the bias term is the frequency domain value. The improved bias term coefficient m h for: m h =1 / (2( 8h / H )*a(( h - 1 )( H - 1 ))); Among them, h is the current attention head index, H is the number of attention heads, and a is the scaling factor.

3. The multimodal large model training system applied to the transportation field according to claim 1 is characterized in that: In the improved decoder architecture, the MQA structure in the architecture is replaced with the GQA structure, the LayerNorm in the architecture is replaced with the RMSNorm, and a Dropout layer is added after the first fully connected layer in the Feed Forward of the architecture.

4. The multimodal large model training system for the transportation field according to claim 1 is characterized in that: The traffic text data includes: Scientific and technological literature, including scientific and technological papers and research reports related to the field of transportation at home and abroad; Industry information, transportation industry development reports, transportation industry related documents, traffic safety education documents, domestic transportation related laws and regulations; Engineering construction information, domestic and international transportation infrastructure; The non-traffic text data includes: high-quality Internet text data, works, and papers; When performing model training, the preset ratio of the traffic text data to the non-traffic text data is 1:

5.

5. A multimodal large model training method applied in the field of transportation, characterized in that: The method uses a multimodal large model training system applied to the transportation field according to any one of claims 1 to 4 to perform multimodal large model training, and the method includes: S10 uses the improved decoder architecture in Transformer and uses pre-collected traffic text data and non-traffic text data to train a language model according to a preset ratio. S20, extracting high-level semantic features of the image through an image encoder, and performing image and text alignment training in combination with the language model to obtain a text-image alignment model; S30, using multimodal instruction data including text-image question-answer pairs and combining them with the text-image alignment model to perform fusion training to obtain a multimodal large model; S40, receiving user requests, performing inference and prediction through a multimodal large model, and providing prediction and interaction results.

6. The multimodal large model training method applied to the traffic field according to claim 5 is characterized in that: Step S10 specifically includes: A10, collect traffic-related text data and non-traffic text data in advance, and prepare pre-training text data according to a preset ratio of 1:5; A20, cleans, de-duplicates, deletes invalid data, and vectorizes the pre-trained text data; A30, uses the improved decoder architecture in Transformers to build the model network; A40, randomly initializes the parameters of each layer of the model network using a preset learning rate; A50, inputs the vectorized training text data into the model network and calculates the activation value and output of each layer; A60, calculates the cross entropy loss between the model network output and the true value; A70 uses the back-propagation algorithm to propagate the loss back to calculate the parameter update gradient; A80, uses the optimizer to update the parameters of each layer of the model network according to gradient descent; A90, repeat A40-A80, and continuously update network parameters until convergence; A100, saves the checkpoints of each epoch to form a pre-training model library; A110, evaluate the models in the pre-trained model library and select the best model as the language model.

7. The multimodal large model training method applied to the traffic field according to claim 5 is characterized in that: Step S20 specifically includes: B10, obtain text-image pair training data; B20, vectorize the acquired text-image pair training data; B30, aligns the visual features extracted by the visual encoder and the text embedding obtained by the language model embedding layer through the Q-Former architecture; B40, inputting the vectorized image data into the visual encoder, extracting the text visual information through the projection layer and inputting it into the language model, and the language model generates an image representation; B50, the difference between the image representation and the original text calculated using the cross entropy loss function; B60, uses the back-propagation algorithm to propagate the loss back to calculate the parameter update gradient; B70, use the optimizer to update the parameters of each layer of the network according to gradient descent; B80, repeat B40-B70, and continuously update network parameters until convergence; B90, saves the checkpoint of each epoch to form a feature alignment model library; B100, evaluates the models in the feature alignment model library and selects the best model as the text-image alignment model.

8. The multimodal large model training method applied to the traffic field according to claim 5 is characterized in that: Step S30 specifically includes: C10, obtains traffic-related image data, combines it with human preference questions and answers, and generates corresponding graphic and text instruction interaction data through GPT-4; C20, vectorization processing of the acquired graphic and text instruction interaction data; C30, inputs the vectorized text-image interaction data into the text-image alignment model for further training, and calculates the activation value and output of each layer; C40, calculates the cross entropy loss between the output of the text-image alignment model and the true value; C50, using the back-propagation algorithm to propagate the loss back to calculate the parameter update gradient; C60, uses the optimizer to update the parameters of each layer of the network according to gradient descent; C70, repeat C30-C60, and continuously update network parameters until convergence; C80, saves the checkpoint of each epoch to form a multimodal model library; C90, evaluates the models in the multimodal model library and selects the best model as the multimodal large model.

9. The multimodal large model training method applied to the traffic field according to claim 5 is characterized in that: Step S40 specifically includes: D10, issuing user request instructions through the webui interface; D20, the WebUI interface passes the user request instruction to the model inference module; D30, the model reasoning module performs reasoning through a multimodal large model; D40, the inference result is returned to the WebUI interface; D50, the WebUI interface generates a friendly response based on the reasoning result and completes an interaction; D60, repeat steps D10-D50 to realize the interactive dialogue between the user and the multimodal large model.

10. A command reasoning interaction method in the field of traffic based on a multimodal large model, characterized in that: include: receiving user instructions for the traffic picture; Reasoning the traffic picture and the user instruction through a pre-trained multimodal large model; Return the inference result to complete the interaction; Wherein, the multimodal large model is trained using the multimodal large model training method applied to the transportation field as described in any one of claims 5-9.

Images