Training method and system for multimodal model of urban rail transit

Abstract

The present invention relates to a training method and system for a multimodal model of urban rail transit, and belongs to the field of artificial intelligence technology, including: extracting text data from urban rail transit field data to generate structured question-answer pairs to form a unimodal data set; extracting and aligning each image and text from urban rail transit field data to form a multimodal data set; inputting the unimodal data set into the ChatGLM2‑6B model for training to form a unimodal model; inputting the multimodal data set into the VisualGLM‑6B model for training to form a multimodal model; and using the trained model to answer questions from users in the field of urban rail transit. The present invention collects text and image data in the field of urban rail transit, so that the model can make full use of different types of information during training, thereby enhancing the accuracy and reliability of the model. In addition, the present invention can select the most suitable model to answer questions by training unimodal and multimodal models, thereby enhancing the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of urban rail transit, and in particular to a training method and system for a multimodal model of urban rail transit. Background Art

[0002] Urban rail transit is an important part of modern society and has a profound impact on economic development, environmental quality, and residents' lives. With the acceleration of urbanization, transportation systems face many challenges, including traffic congestion, environmental pollution, safety issues, and operational efficiency. To meet these challenges, it is particularly important to develop intelligent transportation systems (ITS), which can provide solutions by analyzing, predicting, and optimizing traffic status and behavior. However, the development of ITS requires a deep understanding of specific knowledge and data in the transportation field, as well as the ability to process multimodal inputs and outputs. In recent years, artificial intelligence systems based on large language models have demonstrated their powerful capabilities in multiple fields, from medical diagnosis to legal analysis to financial risk assessment. Large language models (LLMs) learn from large amounts of text data through deep neural networks, capture the semantic and syntactic patterns of language, and generate coherent and accurate text for various applications. However, most LLMs are pre-trained on general corpora and fail to fully cover the particularities and details of the transportation field. Therefore, directly applying these models to the transportation field may produce inaccurate or irrelevant outputs. Summary of the invention

[0003] To solve the above problems, an embodiment of the present invention aims to provide a training method and system for a multimodal model of urban rail transit.

[0004] The training method for the multimodal model of urban rail transit includes:

[0005] Step 1: Collect data in the field of urban rail transit;

[0006] Step 2: extracting text data from the urban rail transit field data, and generating structured question-answer pairs based on the text data to form a unimodal data set;

[0007] Step 3: extracting images and texts from the urban rail transit field data, and aligning each image and text to form a multimodal dataset;

[0008] Step 4: Input the unimodal dataset into the ChatGLM2-6B model for training to form the UrbanTrackAI-SM model;

[0009] Step 5: Input the multimodal dataset into the VisualGLM-6B model for training to form the UrbanTrackAI-MM model;

[0010] Step 6: Use the UrbanTrackAI-SM model or the UrbanTrackAI-MM model to answer users’ questions in the field of urban rail transit.

[0011] Preferably, the step 2: extracting text data from the urban rail transit field data, and generating structured question-answer pairs based on the text data to form a unimodal data set, includes:

[0012] Step 2.1: Use document processing tools to extract text data from urban rail transit data;

[0013] Step 2.2: Use paragraph separators, sentence boundaries, or section headings to divide the text data into multiple chunks, and each chunk contains a different topic;

[0014] Step 2.3: Use the language model to generate one or more questions based on the corresponding paragraphs in each chunk;

[0015] Step 2.4: Merge the generated questions with the corresponding chunks, and then use the language model to generate answers to each question based on the content of each paragraph in the corresponding chunk to form structured question-answer pairs.

[0016] Preferably, the step 3: extracting images and texts from the urban rail transit field data, and aligning each image and text to form a multimodal data set, includes:

[0017] Step 3.1: construct text-image samples based on the aligned images and texts in a preset order;

[0018] Step 3.2: Set corresponding questions for each text-image sample to form a multimodal driving test sample;

[0019] Step 3.3: Convert all text-image samples into Chinese versions to form a general dataset, and use the multimodal driving test samples and the general dataset as a multimodal dataset.

[0020] Preferably, the step 4: inputting the unimodal data set into the ChatGLM2-6B model for training to form the UrbanTrackAI-SM model comprises:

[0021] The initial value of the learning rate is set to 1e-4, the batch size is set to 32, and the ChatGLM2-6B model is fine-tuned for 3 epochs using the single-modal dataset to form the UrbanTrackAI-SM model.

[0022] Preferably, the step 5: inputting the multimodal data set into the VisualGLM-6B model for training to form the UrbanTrackAI-MM model comprises:

[0023] Step 5.1: Set the initial value of the learning rate to 1e-4, the lora level to 32, the batch size to 32, and use the multimodal dataset to fine-tune the VisualGLM-6B model for 300 epochs using the Lora technology to form a one-stage optimization model;

[0024] Step 5.2: Use the common dataset to train the first-stage optimization model for 120 epochs to form the UrbanTrackAI-MM model.

[0025] Preferably, in step 5.1, when fine-tuning the first-stage optimization model, the ratio of the multimodal driving test samples to the general data set is 1:1.

[0026] Preferably, in the process of training the ChatGLM2-6B model or the VisualGLM-6B model, a cross entropy loss function is used to measure the difference between the predicted value and the actual value; wherein the cross entropy loss function is:

[0027]

[0028] in, is the number of categories, is the true label distribution, The ChatGLM2-6B model or VisualGLM-6B model predicts Probability distribution of the class.

[0029] Preferably, in the process of training the ChatGLM2-6B model or the VisualGLM-6B model, the formula is used:

[0030]

[0031]

[0032]

[0033] Update the learning rate of the ChatGLM2-6B model or VisualGLM-6B model; where the subscript represents the current time step, and the subscript represents the previous time step, is the time step The first moment estimate of is the time step The second moment estimate of is the gradient, and is a hyperparameter, is the learning rate, is a constant, is the time step The first moment estimate of is the time step The second moment estimate of is the bias-corrected value of the first-order moment estimate at time step t, is a hyperparameter of The power is used to adjust the smoothness of the first-order moment estimate. For hyperparameters The power is used to adjust the smoothness of the second-order moment estimate. is the time step The model parameters when is the time step The model parameters at .

[0034] The present invention also provides a training system for a multimodal model of urban rail transit, comprising:

[0035] Data collection module, used to collect data in the field of urban rail transit;

[0036] A unimodal data set construction module, used to extract text data from the urban rail transit field data, and generate structured question-answer pairs based on the text data to form a unimodal data set;

[0037] A multimodal dataset construction module, used to extract images and texts from the urban rail transit field data, and align each image and text to form a multimodal dataset;

[0038] The unimodal model training module is used to input the unimodal dataset into the ChatGLM2-6B model for training to form the UrbanTrackAI-SM model;

[0039] Multimodal model training module, used to input multimodal datasets into the VisualGLM-6B model for training to form the UrbanTrackAI-MM model;

[0040] The model application module is used to answer users' questions in the field of urban rail transit using the UrbanTrackAI-SM model or the UrbanTrackAI-MM model.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps in the above-mentioned training method for a multimodal model of urban rail transit are implemented.

[0042] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0043] The present invention relates to a training method for a multimodal model of urban rail transit. Compared with the prior art, the present invention collects text and image data in the field of urban rail transit so that the model can make full use of different types of information during training, thereby enhancing the accuracy and reliability of the model. In addition, the present invention can select the most suitable model to answer questions by training single-modal and multi-modal models, thereby enhancing user experience.

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flow chart of a training method for a multimodal model of urban rail transit provided by the present invention;

[0047] Figure 2 The ChatGLM2-6B network structure diagram provided by the present invention;

[0048] Figure 3 The VisualGLM-6B network structure provided by the present invention. DETAILED DESCRIPTION

[0049] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0050] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0051] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0052] See also Figure 1 , a training method for a multimodal model of urban rail transit, comprising:

[0053] Step 1: Collect data in the field of urban rail transit;

[0054] The diversity of data sources is crucial to building a comprehensive and representative transportation dataset. The data collection in this invention covers the following areas:

[0055] Scientific and technological literature: including scientific and technological papers, research reports and patent standards related to urban rail transit.

[0056] Scientific research foundation: involving transportation technology projects and expert information from the government, research institutions, universities and enterprises.

[0057] Statistical data: covers statistical information on road transport, water transport, port throughput, urban passenger transport, highway construction investment, etc.

[0058] Engineering construction information: including construction information of domestic and international transportation infrastructure, such as highways, bridges, tunnels and ports.

[0059] Management decision-making information: Transportation industry development reports and think tank reports to provide support for policy making.

[0060] Scientific data: pavement damage images and their annotation information for image recognition and processing.

[0061] Others: including transport statistics terminology and knowledge of transport infrastructure construction.

[0062] Step 2: extracting text data from the urban rail transit field data, and generating structured question-answer pairs based on the text data to form a unimodal data set;

[0063] Unimodal data mainly refers to text data. The present invention collects a large amount of text data from the above sources to form a domain pre-training data set. These data are mainly used for the initial training of the model to help the model understand professional terms and concepts in the field of transportation.

[0064] Furthermore, step 2 includes:

[0065] Step 2.1: Use document processing tools to extract text data from urban rail transit data;

[0066] Step 2.2: Use paragraph separators, sentence boundaries, or section headings to divide the text data into multiple chunks, and each chunk contains a different topic;

[0067] Step 2.3: Use the language model to generate one or more questions based on the corresponding paragraphs in each chunk;

[0068] Step 2.4: Merge the generated questions with the corresponding chunks, and then use the language model to generate answers to each question based on the content of each paragraph in the corresponding chunk to form structured question-answer pairs.

[0069] In practical applications, the present invention first needs to extract text content from files of different formats. This step mainly relies on document processing tools, such as Apache PDFBox for PDF files, Apache POI for Microsoft Office documents, and python-docx library for DOCX files. Subsequently, the present invention divides the unsupervised text data into suitable blocks, each of which contains a different topic. The text segmentation is mainly achieved by using paragraph delimiters, sentence boundaries or chapter titles. Using a large language model, the present invention generates one or more questions for each paragraph. These questions are intended to test the model's understanding and memory of the content in the corresponding paragraph. In addition, the present invention uses a question seed library to provide examples to guide the model to generate questions that meet the preferences of the present invention.

[0070] Next, the present invention merges the generated questions with the corresponding original document blocks to form coherent prompts that contain rich traffic information. Then, the language model is used to generate answers to each question. The generation of answers takes into account the content of each paragraph to ensure the accuracy and comprehensiveness of the answers. In the process of generating answers, prefixes and suffixes may be used to clearly indicate the beginning and end of questions and answers to help the model better identify and understand. For example, the present invention generates the following question and answer pairs for questions about pipeline types:

[0071] {

[0072] "id": "478",

[0073] "question": "As an expert in master plan design, please answer the following question\nWhat types of pipelines are mainly covered in the "Urban Engineering Pipeline Comprehensive Planning Code GB50289-2016"? ",

[0074] "answer": "The main types of pipelines covered include water supply, drainage, recycled water, electricity, communications, heat, gas, etc."

[0075] },

[0076] The generated question and answer pairs go through a filtering process before being incorporated into the structured text to remove irrelevant or low-quality samples. The filtering criteria include checking whether the model is likely to repeatedly ask similar questions or ask the same questions to similar texts, and whether the model deviates from the instructions in the response. If the model cannot answer based on the provided fragment or the query is beyond the model's understanding, no output is generated. After the filtering process, the present invention integrates the filtered questions and answers into the structured text in JSON format for subsequent model training work.

[0077] Through this process, the present invention can automatically generate about 156,000 structured question-answer pairs of instruction data from unlabeled text data in the form of question-answer pairs for fine-tuning large language models. This method effectively utilizes unlabeled data, reduces the need for manual annotation, and provides the model with rich conversational data, enhancing the model's ability to understand and handle issues related to urban rail transit.

[0078] Step 3: extracting images and texts from the urban rail transit field data, and aligning each image and text to form a multimodal dataset;

[0079] Furthermore, step 3 includes:

[0080] Step 3.1: construct text-image samples based on the aligned images and texts in a preset order;

[0081] Step 3.2: Set corresponding questions for each text-image sample to form a multimodal driving test sample;

[0082] Step 3.3: Convert all text-image samples into Chinese versions to form a general dataset, and use the multimodal driving test samples and the general dataset as a multimodal dataset.

[0083] In practical applications, the present invention first needs to extract images and texts from the collected multimodal dataset. Next, the present invention carefully aligns each image and text to construct text-image samples in the required order. Then, the present invention designs a question for each sample. The present invention calls this multimodal traffic dataset MTD (Multimodal Driving Test), which contains 3183 samples for training and 122 samples for testing. The typical samples constructed are as follows:

[0084] {

[0085] "id": 1,

[0086] "question": "The figure shows a pV curve of a pre-drilled lateral pressure test. In the figure, a and b are the starting point and end point of the straight line segment in the curve, respectively. Point c is the intersection of the extended line of ab and the V axis. The straight line passing through point c and parallel to the p axis intersects the lateral pressure curve at point d. The rightmost dotted line is the asymptote of the lateral pressure curve. According to the characteristic values ​​given in the figure, which of the following options is the correct calculation formula for determining the bearing capacity (fak) of the foundation soil using the plastic load method?\nA:fak=pb-pd\nB:fak=pb-pa\nC:fak=(1 / 2)pb\nD:fak=(1 / 3)pb",

[0087] "answer": "A:fak=pb-pd",

[0088] "imageName": "C2.jpg"

[0089] }

[0090] In addition, the present invention is based on the aligned dataset and converts it into a Chinese version using Google Translate, which the present invention calls CCAC (Common Dataset), containing 3439 samples.

[0091] Through the above steps, the present invention constructs a unimodal and multimodal dataset containing rich knowledge in the field of transportation, providing a solid foundation for the training and optimization of the UrbanTrackAI model. These datasets not only cover multiple aspects of urban rail transit, but also enhance the model's ability to understand complex traffic scenarios through multimodal data.

[0092] Step 4: Input the unimodal dataset into the ChatGLM2-6B model for training to form the UrbanTrackAI-SM model;

[0093] Furthermore, step 4 includes:

[0094] The initial value of the learning rate is set to 1e-4, the batch size is set to 32, and the ChatGLM2-6B model is fine-tuned for 3 epochs using the single-modal dataset to form the UrbanTrackAI-SM model.

[0095] like Figure 2 As shown in the figure, ChatGLM2-6B is a large pre-trained language model based on the General Language Model (GLM) architecture with 6.2 billion parameters and supports Chinese and English. It uses the Transformer structure, including multi-head self-attention mechanism and position encoding technology. The following are its main components:

[0096] 1. Embedding layer: responsible for converting the input text into a digital vector that the model can process, including word embedding, position embedding, and type embedding.

[0097] 2. Multi-head self-attention mechanism: By calculating the correlation scores between different positions in the input sequence, a weighted context vector is generated for each position.

[0098] 3. Feedforward Neural Network: Perform nonlinear transformation on the output of the multi-head self-attention mechanism to generate the final output of the model.

[0099] The core of ChatGLM2-6B is GLMBlock, and models of different sizes are realized by adjusting the number of GLMBlocks. For example, the 6B model has 28 GLMBlocks.

[0100] Step 5: Input the multimodal dataset into the VisualGLM-6B model for training to form the UrbanTrackAI-MM model;

[0101] Wherein, step 5 comprises:

[0102] Step 5.1: Set the initial value of the learning rate to 1e-4, the lora level to 32, the batch size to 32, and use the multimodal dataset to fine-tune the VisualGLM-6B model for 300 epochs using the Lora technology to form a one-stage optimization model;

[0103] Step 5.2: Use the common dataset to train the first-stage optimization model for 120 epochs to form the UrbanTrackAI-MM model.

[0104] It should be noted that for the training process of UrbanTrackAI-MM, the present invention proposes a two-stage fine-tuning method to enable the VisualGLM-6B model to learn domain-specific knowledge while retaining general capabilities. In the first stage, the present invention fine-tunes the base model on a combined dataset of domain-specific MTD and general CCAC datasets. For each sample, the image and question are used as input, and the answer is used as the golden output. The present invention uses the LoRa technology to fine-tune the model for 300 epochs, with a learning rate of 1e-4, a LoRa level of 32, and a batch size of 32. The present invention also uses an adaptive learning rate optimization algorithm and a cross-entropy loss function. In addition, since the ratio of professional samples and general samples is basically the same. Therefore, after this stage of fine-tuning, the model is preheated, that is, a certain amount of domain-specific knowledge is learned while maintaining general capabilities. In the second stage, the present invention fine-tunes the model on the domain-specific MTD to further acquire more knowledge about the transportation field. In addition, in order to prevent catastrophic forgetting of general knowledge, 32 general samples are added. Similar to the first stage, the present invention uses LORA technology and performs 120 epochs of training. After fine-tuning in this stage, the present invention obtains UrbanTrackAI-MM.

[0105] like Figure 3 As shown in the figure, VisualGLM-6B is a multimodal conversational language model based on ChatGLM-6B. It integrates vision and language models, enabling the language model to understand image information and seamlessly combine it with text conversations. The following are its main components:

[0106] 1. ViT (Vision Transformer): used to extract visual features of images.

[0107] 2. Q-Former: As an intermediate module, it transforms the visual features extracted by ViT into representations that can be understood by the language model.

[0108] 3. ChatGLM-6B: Language model part, used to generate text or answer questions.

[0109] During training, the parameters of ViT and ChatGLM-6B are almost frozen, and Q-Former is mainly fine-tuned. This is done to prevent "catastrophic forgetting", that is, fine-tuning the model on a new task will lead to the loss of its original ability.

[0110] In the process of training the ChatGLM2-6B model or the VisualGLM-6B model, the cross entropy loss function is used to measure the difference between the predicted value and the actual value; wherein the cross entropy loss function is:

[0111]

[0112] in, is the number of categories, is the true label distribution, The ChatGLM2-6B model or VisualGLM-6B model predicts Probability distribution of the class.

[0113] When training the ChatGLM2-6B model or the VisualGLM-6B model, the formula is used:

[0114]

[0115]

[0116]

[0117] Update the learning rate of the ChatGLM2-6B model or the VisualGLM-6B model; is the first-order moment estimate, is the second-order moment estimate, is the gradient, and is a hyperparameter, is the learning rate, is a constant, is the time step The first moment estimate of is the time step The second moment estimate of is the bias-corrected value of the first-order moment estimate at time step t, is a hyperparameter of The power is used to adjust the smoothness of the first-order moment estimate. For hyperparameters The power is used to adjust the smoothness of the second-order moment estimate. is the time step The model parameters when is the time step The model parameters at .

[0118] In the method of the present invention, Lora (Low-Rank Adaptation) technology is used to efficiently fine-tune large pre-trained language models. Lora technology adjusts the weights of the model by introducing a low-rank matrix, thereby achieving fine-tuning for specific tasks without significantly increasing the number of model parameters. The following is a detailed introduction to how the Lora technology is used in the method of the present invention.

[0119] 1. Initialize Lora parameters

[0120] In the model initialization stage, the present invention replaces the linear layer of the model with the Lora layer through the `LoraMixin` class and sets the rank of the low-rank matrix. The specific steps are as follows:

[0121] If using LoRA:

[0122] Set the parameters of the LoraMixin layer: the number of model layers num_layers, the LoRA rank lora_rank, and the layer range layer_range to which the LoRA technology needs to be applied.

[0123] Specific implementation of Lora: The LoRA layer contains two low-rank matrices matrix_A and matrix_B, as well as forward propagation and parameter merging methods.

[0124] classLoraMixin:

[0125] Initialization method:

[0126] Initialize num_layers, lora_rank, layer_range and other parameters

[0127] Initialize low-rank matrices matrix_A and matrix_B, and perform weight initialization

[0128] Forward propagation method forward(x):

[0129] Returns x+(x*matrix_A)*matrix_B

[0130] Merge LoRA parameter method merge_lora():

[0131] Merge LoRA parameters into original weights

[0132] Model parameter setting: Set various parameters of the model through command line parameters:

[0133] Maximum source length: 64

[0134] Maximum target length: 256

[0135] LoRA rank: 32

[0136] Application layer range: 0 to 27

[0137] Pre-sequence length: 4

[0138] 2. Forward Propagation

[0139] During the forward propagation of the model, the updated low-rank matrix is ​​used to calculate the new weight matrix and applied to the calculation of the model. The specific steps are as follows:

[0140] Calculate the new weight matrix: During the forward propagation process, the Lora layer uses the low-rank matrices A and B to calculate the new weight matrix W.

[0141]

[0142] in, is the original weight matrix, A and are low-rank matrices.

[0143] Apply new weight matrix: Apply the calculated new weight matrix to the calculation process of the model.

[0144] 3. Back Propagation and Update

[0145] Calculate loss: During the model training process, the present invention uses the cross entropy loss function to measure the difference between the model's prediction results and the true label. The specific steps are as follows:

[0146] Define the loss function CrossEntropyLoss, ignoring the padding mark

[0147] Get batch data (batchsize is 16), including tokens, labels, image, pre_image

[0148] Calculate logits by forward propagating the model

[0149] Calculating cross entropy loss

[0150] Returns the loss value

[0151] Use the optimizer (Adam) to update the parameters of the low-rank matrices A and B: The present invention uses the Adam optimizer to update the parameters of the model and control the training process of the model by adjusting the learning rate and other hyperparameters. The specific steps are as follows:

[0152] Initialize the Adam optimizer and set the learning rate to 0.0001

[0153] At each training step:

[0154] Zero gradient

[0155] Calculate loss and backpropagate

[0156] Update model parameters

[0157] 4. Merge LoRA parameters

[0158] After fine-tuning is completed, the present invention can choose to merge the parameters of the LoRA part into the original weights to obtain a final model including LoRA adjustments:

[0159] Merge LoRA parameters: Merge the parameters of the LoRA part into the original weights through the merge_lora method.

[0160] Save the merged model: Set the save path and mode to save the merged model as the final model file.

[0161] Through the above steps, the present invention uses Lora technology to efficiently fine-tune a large pre-trained language model. Lora technology adjusts the weight of the model by introducing a low-rank matrix, thereby achieving fine-tuning for a specific task without significantly increasing the number of model parameters. The specific implementation includes steps such as initializing Lora parameters, forward propagation, loss calculation, optimizer update, and merging Lora parameters. Through these steps, the present invention can effectively fine-tune the model and obtain a final model including Lora adjustment.

[0162] The UrbanTrackAI model has demonstrated excellent performance in multiple key tasks in the field of transportation. In order to comprehensively evaluate the performance of the model, the present invention designed a series of benchmark tests and adopted a zero-shot evaluation method. These tests cover multiple aspects such as traffic situation prediction, intelligent assistant, public transportation service, traffic planning, traffic safety education, and accident reporting and analysis. During the evaluation process, the present invention combines the automatic evaluation of the GPT-4 model with manual evaluation to ensure the accuracy and reliability of the results.

[0163] In terms of traffic situation prediction, the UrbanTrackAI-7B model scored 1.33, showing the model's effectiveness in understanding and predicting traffic flow patterns. In the traffic planning test, the model demonstrated its potential for application in traffic network analysis and route planning with a high score of 9.95. In addition, in the field of traffic safety education, the UrbanTrackAI-7B model demonstrated its practicality in providing safety education and training information with a score of 9.84.

[0164] To further verify the performance of the model, the present invention conducted unimodal and multimodal experiments on the TransEval evaluation dataset. In the unimodal experiment, the UrbanTrackAI-SM model performed 10.16% better than the baseline model ChatGLM2-6B on the Traffic Engineering Test (TET), and a modest improvement of 1.78% on the Driving License Exam (DLE). These results show that through fine-tuning in specific areas, the model's ability to understand and apply traffic rules and driving skills has been significantly improved.

[0165] In multimodal experiments, the UrbanTrackAI-MM model achieved an accuracy of 67.21% on the MTD test set, a 40.16% improvement over the baseline model VisualGLM-6B's 27.05%. This significant performance improvement demonstrates the model's effectiveness in handling tasks that combine text and visual inputs.

[0166] In addition, an ablation study was performed to evaluate the impact of each component in the data construction process. The results show that the steps of filtering, chunking, and question-answer generation are all critical to improving the zero-shot prompting performance of the model. The interdependence and synergy of these steps together contribute to the overall improvement of the model performance.

[0167] Table 1: Performance comparison of UrbanTrackAI-SM and baseline models on unimodal tasks

[0168] Model C-Eval TET TPT DLE ChatGLM2-6B 51.7 23.98 29.23 22.02 UrbanTrackAI-SM 47.99 34.14 29.24 23.8

[0169] Table 2: Performance comparison of UrbanTrackAI-MM and baseline models on multimodal tasks

[0170] Model Accuracy VisualGLM-6B 27.05 UrbanTrackAI-MM 67.21

[0171] Table 3: Ablation study results

[0172] Method Accuracy All processes 34.14 -filter 30.48 - Text chunking 27.23 - Question and answer generation 23.98

[0173] In summary, the UrbanTrackAI model has demonstrated strong capabilities and potential in multiple key tasks in the transportation field. These evaluation results not only verify the technical advantages of the model, but also provide strong support for its future application in intelligent transportation systems.

[0174] The present invention also provides a training system for a multimodal model of urban rail transit, comprising:

[0175] Data collection module, used to collect data in the field of urban rail transit;

[0176] A unimodal data set construction module, used to extract text data from the urban rail transit field data, and generate structured question-answer pairs based on the text data to form a unimodal data set;

[0177] A multimodal dataset construction module, used to extract images and texts from the urban rail transit field data, and align each image and text to form a multimodal dataset;

[0178] The unimodal model training module is used to input the unimodal dataset into the ChatGLM2-6B model for training to form the UrbanTrackAI-SM model;

[0179] Multimodal model training module, used to input multimodal datasets into the VisualGLM-6B model for training to form the UrbanTrackAI-MM model;

[0180] The model application module is used to answer users' questions in the field of urban rail transit using the UrbanTrackAI-SM model or the UrbanTrackAI-MM model.

[0181] Compared with the prior art, the beneficial effects of a training system for an urban rail transit multimodal model provided by the present invention are the same as the beneficial effects of a training method for an urban rail transit multimodal model described in the above technical solution, which will not be elaborated here.

[0182] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps in the above-mentioned training method for a multimodal model of urban rail transit are implemented. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the training method for a multimodal model of urban rail transit described in the above-mentioned technical solution, which will not be repeated here.

[0183] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technical solution that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A training method for a multimodal model of urban rail transit, characterized in that: include: Step 1: Collect data in the field of urban rail transit; Step 2: extracting text data from the urban rail transit field data, and generating structured question-answer pairs based on the text data to form a unimodal data set; Step 3: extracting images and texts from the urban rail transit field data, and aligning each image and text to form a multimodal dataset; Step 4: Input the unimodal dataset into the ChatGLM2-6B model for training to form the UrbanTrackAI-SM model; Step 5: Input the multimodal dataset into the VisualGLM-6B model for training to form the UrbanTrackAI-MM model; Step 6: Use the UrbanTrackAI-SM model or the UrbanTrackAI-MM model to answer users’ questions in the field of urban rail transit; The step 3: extracting images and texts from the urban rail transit field data, and aligning each image and text to form a multimodal data set, including: Step 3.1: construct text-image samples based on the aligned images and texts in a preset order; Step 3.2: Set corresponding questions for each text-image sample to form a multimodal driving test sample; Step 3.3: Convert all text-image samples into Chinese versions to form a general dataset, and use the multimodal driving test samples and the general dataset as a multimodal dataset; Step 5: Inputting the multimodal data set into the VisualGLM-6B model for training to form the UrbanTrackAI-MM model, including: Step 5.1: Set the initial value of the learning rate to 1e-4, the lora level to 32, the batch size to 32, and use the multimodal dataset to fine-tune the VisualGLM-6B model for 300 epochs using the Lora technology to form a one-stage optimization model; Step 5.2: Use the common dataset to train the first-stage optimization model for 120 epochs to form the UrbanTrackAI-MM model.

2. The training method for a multimodal model of urban rail transit according to claim 1, characterized in that: The step 2: extracting text data from the urban rail transit field data, and generating structured question-answer pairs based on the text data to form a unimodal data set, includes: Step 2.1: Use document processing tools to extract text data from urban rail transit data; Step 2.2: Use paragraph separators, sentence boundaries, or section headings to divide the text data into multiple chunks, and each chunk contains a different topic; Step 2.3: Use the language model to generate one or more questions based on the corresponding paragraphs in each chunk; Step 2.4: Merge the generated questions with the corresponding chunks, and then use the language model to generate answers to each question based on the content of each paragraph in the corresponding chunk to form structured question-answer pairs.

3. The training method for a multimodal model of urban rail transit according to claim 2, characterized in that: Step 4: Inputting the unimodal dataset into the ChatGLM2-6B model for training to form the UrbanTrackAI-SM model, including: The initial value of the learning rate is set to 1e-4, the batch size is set to 32, and the ChatGLM2-6B model is fine-tuned for 3 epochs using the single-modal dataset to form the UrbanTrackAI-SM model.

4. The training method for a multimodal model of urban rail transit according to claim 3, characterized in that: In step 5.1, when fine-tuning the first-stage optimization model, the ratio of the multimodal driving test samples to the general data set is 1:

1.

5. The training method for a multimodal model of urban rail transit according to claim 4, characterized in that: In the process of training the ChatGLM2-6B model or the VisualGLM-6B model, the cross entropy loss function is used to measure the difference between the predicted value and the actual value; wherein the cross entropy loss function is: Where C is the number of categories, y i is the true label distribution, is the probability distribution predicted by the ChatGLM2-6B model or the VisualGLM-6B model for the i-th class.

6. The training method for a multimodal model of urban rail transit according to claim 4, characterized in that: When training the ChatGLM2-6B model or the VisualGLM-6B model, the formula is used: m t =β1m t-1 +(1-β1)g t Update the learning rate of the ChatGLM2-6B model or VisualGLM-6B model; where subscript t represents the current time step, subscript t-1 represents the previous time step, and m t is the first-order moment estimate at time step t, v t is the second-order moment estimate at time step t, g t is the gradient, β1 is the first hyperparameter, β2 is the second hyperparameter, η is the learning rate, ∈ is a constant, m t-1 is the first-order moment estimate at time step t-1, v t-1 is the second-order moment estimate at time step t-1, is the bias-corrected value of the first-order moment estimate at time step t, is the t-th power of the hyperparameter β1, which is used to adjust the smoothness of the first-order moment estimate. is the t-th power of the hyperparameter, which is used to adjust the smoothness of the second-order moment estimate, θ t is the model parameter at time step t, θ t-1 are the model parameters at time step t-1.

7. A training system for a multimodal model of urban rail transit, characterized in that: include: Data collection module, used to collect data in the field of urban rail transit; A unimodal data set construction module, used to extract text data from the urban rail transit field data, and generate structured question-answer pairs based on the text data to form a unimodal data set; A multimodal dataset construction module, used to extract images and texts from the urban rail transit field data, and align each image and text to form a multimodal dataset; The unimodal model training module is used to input the unimodal dataset into the ChatGLM2-6B model for training to form the UrbanTrackAI-SM model; Multimodal model training module, used to input multimodal datasets into the VisualGLM-6B model for training to form the UrbanTrackAI-MM model; Model application module, used to answer users' questions in the field of urban rail transit using the UrbanTrackAI-SM model or the UrbanTrackAI-MM model; Images and texts are extracted from the urban rail transit data, and each image and text is aligned to form a multimodal dataset, including: Constructing text-image samples based on the aligned images and texts in a preset order; Set corresponding questions for each text-image sample to form a multimodal driving test sample; Convert all text-image samples into Chinese versions to form a general dataset, and use the multimodal driving test samples and the general dataset as a multimodal dataset; The multimodal dataset is input into the VisualGLM-6B model for training to form the UrbanTrackAI-MM model, including: The initial value of the learning rate is set to 1e-4, the lora level is set to 32, the batch size is set to 32, and the VisualGLM-6B model is fine-tuned for 300 epochs using the multimodal dataset using the Lora technology to form a one-stage optimization model; The first-stage optimization model is trained for 120 epochs using a common dataset to form the UrbanTrackAI-MM model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the training method for a multimodal model of urban rail transit are implemented as described in any one of claims 1-6.

Images