Multi-modal traffic large model staged training method and system based on WSD learning rate scheduler

Abstract

The invention relates to the technical field of intelligent traffic, in particular to a multi-modal traffic large model staged training method and system based on a WSD learning rate scheduler, and the method comprises the steps: carrying out the enhancement and extension of an original traffic scene data set through a data enhancement method based on a man-machine cooperation program; obtaining a plurality of downstream task data sets suitable for different levels of training targets in the field of traffic violation scene perception and analysis; and combining a WSD learning rate scheduler and task objectives of different downstream tasks to formulate a staged training strategy. According to the method, limited original data in a low-resource scene in the traffic field is expanded into downstream task data sets oriented to different training targets through a data enhancement method of a man-machine collaboration program, and a WSD learning rate scheduler is introduced to split a fine adjustment process of a traditional multi-modal large language model into different training stages; and different downstream task data sets are put in different training stages in a targeted manner, so that a training process which better fits a model learning rule is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a multi-modal traffic large model phased training method and system based on a WSD learning rate scheduler. Background Art

[0002] With the rapid development of multimodal large models, their excellent image and text perception and reasoning capabilities have brought possibilities for the evolution of intelligent travel and provided solutions to the current difficulties. However, how to make multimodal large models better understand traffic, that is, how to train them in a targeted and efficient manner through scientific and effective methods, is the current research focus.

[0003] In some specific fields, in order to fully train the model and explore its potential in specific downstream tasks, researchers usually use carefully prepared vertical field datasets to fine-tune the model. Common fine-tuning steps include: (1) collecting downstream task datasets suitable for different task scenarios; (2) classifying each dataset according to its characteristics and task objectives; (3) systematically training the model using a phased training strategy. This phased training strategy is in line with human learning habits and can guide the model to have a gradual understanding of the target field, from simple conceptual understanding to specific tasks that require professional knowledge support. This idea of ​​phased training was first introduced by the LLaVA model and gained attention in academia. Later, it was widely used in the training of downstream task models such as LLaVA-Med and TransGPT. Taking the LLaVA series of models as an example, they are first pre-trained on a large-scale image-text pair dataset to align the embedding of visual features and language models, and then fine-tuned on a dataset of specific multi-image instructions to further improve the model's instruction-following ability and expand it to the ability to process multiple images, videos, and 3D scenes. Based on the existing single-image stage training, the LLaVA-NeXT model continues to train using high-quality single-image data to maintain single-image performance and improve multi-image processing capabilities. This training strategy not only improves the overall performance of the model, but also enhances its cross-modal understanding ability, which is crucial for achieving complex multimodal interactions. However, in the field of transportation, the phased training strategy of the general multimodal large language model is usually directly applied during the training of large models, without adaptive adjustments to the unique needs of transportation. This is a problem that needs to be solved in current research.

[0004] However, due to the rapid development of large models, there are few works that apply multimodal capabilities and classic large language model capabilities to specific scenarios in the transportation field, and there are still many potential problems that need to be solved. Taking the influential work of migrating basic large models to the transportation field, such as TrafficGPT, TransGPT and other models, as an example, the current problems mainly exist in two aspects: First, the problem of data scarcity. Since the data sets in the transportation field contain sensitive information such as license plates, faces and driving trajectories, the public data sets are single in form and small in number. This low-resource environment is difficult to obtain data that meets the needs of multiple task goals and capability sharpening, and this problem is rarely mentioned in the current work; second, in the domain adaptation training process of the general multimodal large language model, the above model chooses a data-centric phased training strategy similar to the field models such as LLaVA-med. This traditional phased training strategy divides the model training process through human learning experience, and usually presets fixed training stages and parameters, such as changes in learning rate, which may not fully meet the characteristics of all types of data or tasks. This one-size-fits-all approach may lead to insufficient learning or overfitting in some stages. However, this idealized phased training strategy only conforms to the habits or intuition of human learning knowledge, and does not fit the law of parameter changes of the model during fine-tuning. In addition, when transitioning from one training stage to another, the abrupt transformation of the data set may lead to instability in model training, such as performance fluctuations or even degradation. Summary of the invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art and propose a phased training method and system for a multimodal traffic large model based on a WSD learning rate scheduler, which gradually cultivates and improves the ability of traditional multimodal large language models in traffic violation scene perception and analysis scenarios from enhancing traffic concept understanding to strengthening traffic scene reasoning, so that the model can be trained more scientifically and fully.

[0006] To achieve the above purpose, the technical solution adopted is:

[0007] A phased training method for a large multimodal traffic model based on a WSD learning rate scheduler, comprising:

[0008] Collect and preprocess raw traffic scene data, match and format the images and text annotations in the raw data, and generate the raw traffic scene data set;

[0009] The original traffic scene dataset is enhanced and expanded through a data enhancement method based on human-computer collaborative programs, and multiple downstream task datasets suitable for different levels of training objectives in the field of traffic violation scene perception and analysis are obtained;

[0010] A phased training strategy is formulated by combining the WSD learning rate scheduler and the task objectives of different downstream tasks.

[0011] According to the multimodal traffic large model phased training method based on the WSD learning rate scheduler of the present invention, further, multiple downstream task data sets with different training objectives are obtained, including:

[0012] First, a general multimodal large language model is used to generate scene-level image descriptions and extract key object features from them. These generated contents are confirmed by human annotators. Then, a general multimodal large language model is used to generate formatted related questions and initial responses. Finally, these question-answer pairs are verified again by human annotators.

[0013] According to the multimodal traffic large model phased training method based on the WSD learning rate scheduler of the present invention, further, the downstream task data sets are classified as follows according to the task objectives: the downstream task data sets for the training objectives of supporting the traffic concept understanding of the basic large model include image subtitle data sets, object recognition data sets and license plate number recognition data sets; the downstream task data sets for the training objectives of supporting the traffic scene reasoning of the basic large model include visual reasoning data sets, spatial reasoning data sets and traffic violation recognition data sets.

[0014] According to the multimodal traffic large model phased training method based on WSD learning rate scheduler of the present invention, further, the downstream task dataset is defined as Among them, i n Indicates the instruction passed to the basic large model to be fine-tuned, v n Represents a visual sample, x n Represents the true value that should be output based on the given instructions and visual samples.

[0015] According to the multimodal traffic large model phased training method based on the WSD learning rate scheduler of the present invention, further, a phased training strategy is formulated by combining the WSD learning rate scheduler and the task objectives of different downstream tasks. The strategy consists of two parts: the complete training process is divided by the WSD learning rate scheduler, and data sets with different training objectives are invested according to the characteristics of different learning stages.

[0016] According to the multimodal traffic large model phased training method based on the WSD learning rate scheduler of the present invention, further, the complete training process is segmented by the WSD learning rate scheduler, including:

[0017] After the warm-up phase, the WSD learning rate scheduler divides the training phase into a continuous training phase and a learning rate decay phase. The changes in the learning rates in these three different phases are expressed as follows:

[0018]

[0019] Where W and T represent the number of training steps in the warm-up phase and the continuous training phase, respectively, 0<f(sT)≤1 is a decreasing function of the number of training steps s, and η is the maximum learning rate.

[0020] According to the multimodal traffic large model phased training method based on the WSD learning rate scheduler of the present invention, further, according to the characteristics of different learning stages, data sets with different training objectives are invested, including:

[0021] In the continuous training phase, traffic concept understanding is set as the main training goal of this phase, and a large number of image caption datasets, object recognition datasets, and license plate recognition datasets are used for model training;

[0022] In the learning rate decay phase, traffic scene reasoning is taken as the main training target of this phase, and a large number of visual reasoning datasets, spatial reasoning datasets and traffic violation recognition datasets are used for model training.

[0023] Furthermore, the present invention also proposes a multimodal traffic large model phased training system based on a WSD learning rate scheduler, comprising an original data set construction module, a data enhancement module and a phased training module, wherein:

[0024] The original data set construction module is used for the collection and preprocessing of original traffic scene data, matching and formatting the images and text annotations in the original data, and generating the original traffic scene data set;

[0025] The data enhancement module is used to enhance and expand the original traffic scene dataset through a data enhancement method based on human-computer collaborative programs, and obtain multiple downstream task datasets suitable for different levels of training objectives in the field of traffic violation scene perception and analysis;

[0026] The phased training module is used to formulate a phased training strategy by combining the WSD learning rate scheduler and the task objectives of different downstream tasks.

[0027] The beneficial effects achieved by adopting the above technical solution are:

[0028] 1. The present invention proposes a phased training method for a multimodal traffic large model based on a WSD learning rate scheduler. The data enhancement method can effectively change the low-resource dilemma existing in the original traffic violation scene perception and analysis scene. With the cooperation of a general multimodal large language model and manual annotators, this data enhancement paradigm based on a human-computer collaborative program can fully utilize the excellent natural language processing and formalization capabilities of the general multimodal large language model. At the same time, it can also correct the errors in the content generated by the general multimodal large language model with the help of manual annotators, and generate high-quality data sets more efficiently. This data enhancement method also provides a feasible solution for other low-resource fields.

[0029] 2. The phased training strategy of the present invention combines the WSD learning rate scheduler with the traditional phased training strategy, and realizes the continuous training of the general multimodal large language model in the field of traffic violation scene perception and analysis. Not only can the different downstream tasks in the traffic violation scene perception and analysis scenario be more closely aligned with the parameter change rules in the model training process, but it can also solve the instability of model training caused by the traditional phased training strategy that relies on division of labor to divide the training stages and abruptly switches the data set, such as performance fluctuations or even degradation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings of the embodiments of the present invention, wherein the drawings are only used to illustrate some embodiments of the present invention, but not to limit all embodiments of the present invention thereto.

[0031] Figure 1 It is a flow chart of a multi-modal traffic large model phased training method based on a WSD learning rate scheduler according to an embodiment of the present invention;

[0032] Figure 2 It is a schematic diagram of formulating a phased training strategy for a joint WSD learning rate scheduler and task objectives of different downstream tasks according to an embodiment of the present invention;

[0033] Figure 3 This is the evaluation result of training the GLM-4V model using a WSD learning rate scheduler with 5%, 10%, 15% and 20% of the total training steps as the annealing (learning rate decay) stage step size according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings of specific embodiments of the present invention to clearly and completely describe the exemplary scheme of the embodiment of the present invention. Unless otherwise defined, the technical terms or scientific terms used in the present invention should be the common meanings understood by people with ordinary skills in the field.

[0035] This embodiment discloses a multi-modal traffic large model phased training method based on a WSD learning rate scheduler, such as Figure 1 As shown, it contains the following contents:

[0036] Step S101, collecting and preprocessing raw traffic scene data, matching and formatting the images and text annotations in the raw data, and generating a raw traffic scene data set O that is easy to enhance and expand.

[0037] Step S102: Enhance and expand the limited and single-form original traffic scene dataset through a data enhancement method based on a human-computer collaborative program to obtain multiple high-quality downstream task datasets suitable for different levels of training objectives in the field of traffic violation scene perception and analysis.

[0038] The original traffic scene dataset is enhanced to mine and utilize key information such as object semantics and spatial concepts in the scene, and then multiple targeted downstream task datasets are constructed to enhance the model's deep understanding of traffic violation scenes. In order to efficiently construct these target datasets, a human-machine collaborative program is introduced to expand the original traffic scene dataset. The specific operation is as follows: first, a general multimodal large language model is used to generate detailed scene-level image descriptions, and key object features such as license plate numbers, violation types, vehicle types, and traffic light status are extracted from them. These generated contents are confirmed by human annotators; then a general multimodal large language model (such as GPT-4) is used to generate formatted related questions and initial responses, and finally these question-answer pairs are verified again by human annotators. After such a human-machine collaborative program, the original traffic scene dataset, which was originally single in form and limited in scale, has been enhanced and expanded, and multiple high-quality downstream task datasets suitable for different tasks have been generated.

[0039] The generated traffic dataset mainly serves two training goals in traffic violation scene perception and analysis: traffic concept understanding and traffic scene reasoning. Among them, the image caption dataset, object recognition dataset and license plate number recognition dataset derived from the original traffic scene dataset are used to support the training goal of traffic concept understanding of the basic large model. The visual reasoning dataset (the time of the scene, weather conditions, etc.), spatial reasoning dataset (spatial position between the vehicle and the marking) and traffic violation recognition dataset (inferring whether the vehicle has violated the law through three pictures with temporal continuity) are used to support the training goal of traffic scene reasoning of the basic large model.

[0040] Furthermore, the human-computer collaborative program is used to mine different levels of features in the original data, and the original traffic scene dataset O is organized into the corresponding downstream task dataset Among them, i nrepresents the instruction passed to the base large model to be fine-tuned (such as a question about the status of a traffic light), v n represents a visual sample (such as a picture), x n Represents the true value (response answer) that should be output based on the given instructions and visual samples. and the text encoder f φ Pair the images and texts in the dataset Encode and get an image code and two text encodings This makes these downstream task datasets It can better access subsequent model training.

[0041] Step S103: Combine the WSD learning rate scheduler and the task objectives of different downstream tasks to formulate a phased training strategy. Figure 2 As shown in Figure 1, this step consists of two parts: segmenting the complete training process through the WSD learning rate scheduler, and investing in data sets with different training objectives according to the characteristics of different learning stages.

[0042] (1) The complete training process is split by the WSD learning rate scheduler.

[0043] This embodiment uses the WSD learning rate scheduler (Warmup-Stable-Decay Learning Rate Schedular). This learning rate scheduling strategy is suitable for general multimodal large language models that need to be fine-tuned for domain adaptability on large-scale data sets, and can effectively balance the learning efficiency and stability of model performance. Specifically, it can divide the entire fine-tuning process into different training stages with obvious characteristics and in a continuous training process, so as to ensure the maximization of the training effect.

[0044] Different from the common cosine learning rate scheduler, the WSD learning rate scheduler divides the training phase into a continuous training phase and a learning rate decay phase after the warm-up phase. The changes in the learning rate in these three different phases can be expressed as follows:

[0045]

[0046] Where W and T represent the number of training steps (Global Steps) in the warm-up phase and the continuous training phase, respectively. 0<f(sT)≤1 is a decreasing function of the number of training steps s, and η is the maximum learning rate. It is worth noting that because the WSD learning rate scheduler sets a higher learning rate in the continuous training phase, the model can be updated with a larger number of steps, so this phase is also called the high learning rate training phase.

[0047] (2) Invest in data sets with different training objectives based on the characteristics of different learning stages.

[0048] Different from the training process of a broad traffic basic model, the specific field of traffic violation scene perception and analysis requires the model to not only have basic traffic common sense, but also a certain reasoning ability and professional knowledge for the identification and understanding of traffic violations, so as to achieve deeper reasoning and suggestion generation. Due to this demand, the training goals to be achieved in the entire fine-tuning process are divided into two parts: traffic concept understanding and traffic scene reasoning ability enhancement, and data is arranged according to the characteristics of different learning rate training stages explicitly divided by the WSD learning rate scheduler, so as to achieve continuous and stable training in line with the model learning law.

[0049] Specifically, ① in the continuous training stage, the model undergoes a long period of high learning rate training, during which the gradient norm remains at a high level, which is conducive to the model exploring the global optimal solution in understanding and aligning traffic concepts. Therefore, traffic concept understanding is set as the main training goal of this stage, and a large number of image caption datasets, object recognition datasets, and license plate recognition datasets are used for model training, so that the model can effectively transfer the prior knowledge acquired in the warm-up stage to the traffic field. ② In the learning rate decay stage, as the learning rate decreases rapidly, the loss of the model also drops significantly. In this stage, the continuous reduction of the gradient norm is a key dynamic phenomenon, which helps the model lock a more accurate local optimal solution. Therefore, the enhancement of traffic scene reasoning ability is set as the training goal of this stage, and a large number of visual reasoning datasets, spatial reasoning datasets, and traffic violation recognition datasets are used for model training to encourage the model to effectively apply the previously learned traffic common sense to the in-depth reasoning of traffic scenes, thereby significantly improving the performance of the model.

[0050] In general, the method of the present invention clearly divides the various stages of training by introducing the WSD learning rate scheduler, and guides the model into different learning states by setting different learning rates. According to the characteristics of these learning states, data sets for traffic concept understanding and traffic scene reasoning are selected and used in a targeted manner. This method combines the focus on data with the need for continuous training, effectively implements a phased training strategy, and ensures the comprehensive training and excellent performance of the model when dealing with various downstream tasks in the traffic field. The present invention realizes a phased continuous training process that does not require pauses and restarts through the coordination of the WSD learning rate scheduler and downstream task data sets for different training objectives, and efficiently and stably transfers the capabilities of the general multimodal large language model to the specific application scenario of traffic violation scene perception and analysis.

[0051] Corresponding to the above method, this embodiment also proposes a multimodal traffic large model phased training system based on a WSD learning rate scheduler, including an original data set construction module, a data enhancement module and a phased training module, wherein:

[0052] The original data set construction module is used for collecting and preprocessing original traffic scene data, matching and formatting the images and text annotations in the original data, and generating the original traffic scene data set.

[0053] The data enhancement module is used to enhance and expand the original traffic scene dataset through a data enhancement method based on human-computer collaborative programs, and obtain multiple downstream task datasets suitable for different levels of training objectives in the field of traffic violation scene perception and analysis.

[0054] The phased training module is used to formulate a phased training strategy by combining the WSD learning rate scheduler and the task objectives of different downstream tasks.

[0055] In order to verify the effectiveness of this solution, further explanation is given below in combination with experimental data.

[0056] In order to test the proposed method more fairly, in addition to the downstream task datasets constructed by the human-computer collaborative program, the public CCPD (Chinese City Parking Dataset) and Microsoft COCO (Common Objects in COntext) datasets are also introduced. Evaluation indicators such as BLEU, Rough-L and BERTScore are used for evaluation.

[0057] GLM-4V, a large language model that supports multimodality and multilingualism, is selected as the basic model, and the phased training strategy based on the cosine learning rate scheduler is used as the baseline method. In addition, since the proportion of the global steps used for annealing in the WSD learning rate scheduler may affect the training effect of the model, we set the WSD learning rate scheduler with 5%, 10%, 15% and 20% of the total training steps as the annealing (learning rate decay) stage step size to guide the phased training strategy proposed in this invention, and use it to train the GLM-4V model with the same other variables, in order to prove the advanced nature of the method proposed in this invention. The evaluation results are as follows: Figure 3 shown.

[0058] First, we selected two tasks that are strongly related to traffic scene understanding and reasoning: spatial reasoning task and traffic violation recognition task. Then, we used the GLM-4V model trained by the phased training strategy of the cosine learning rate scheduler as the baseline model, and conducted comparative experiments using the phased training strategy guided by the WSD learning rate scheduler with different annealing steps, and evaluated it on indicators such as BLEU and G-Eval. Finally, in order to more clearly show the impact of different strategies on model training, we used the standard deviation as the weight of each indicator, and used it to normalize the indicator data. Figure 3 It is visualized in . In the spatial reasoning task, the model trained by the WSD learning rate scheduler set to use 10% of the total number of steps for annealing operation achieved the highest scores in all indicators, and in the traffic violation recognition task, 5 indicators achieved the highest scores. This verifies the effectiveness of the multimodal traffic large model phased training method based on the WSD learning rate scheduler and data enhancement proposed in the present invention.

[0059] In summary, the innovation of this invention is that, first, a data enhancement paradigm based on human-computer collaborative program is proposed to solve the problem of insufficient data resources in the current traffic violation scene perception and analysis; second, by combining the WSD learning rate scheduler with the staged training strategy, the model training efficiency is improved while solving the problem of being unable to maintain continuous model training during stage switching.

[0060] Specifically, this data augmentation paradigm based on human-machine collaborative programs can effectively expand the limited raw data in low-resource scenarios such as traffic violation scene perception and analysis, thereby obtaining downstream task data sets suitable for different training objectives. This innovation faces the current low-resource dilemma facing this field at the data level and successfully achieves the expansion of the original data set.

[0061] In addition, the novel phased training strategy proposed by the present invention can accurately combine the dynamic changes of model parameters with the specific needs of the training objectives. By introducing the Warmup-Stable-Decay (WSD) learning rate scheduler, the complete training process is explicitly divided into different learning rate change stages. And according to the model parameter changes caused by different learning rate change stages, different downstream task data sets are used in a targeted manner to support the goals of each training stage. By accurately corresponding the data set to the learning stage, the present invention ensures that the model can make full use of the most relevant data for learning at each stage, and such a method significantly improves the training efficiency and the generalization ability of the model. This fine-tuning strategy that is more in line with the laws of machine learning can make the model more fully utilize the data and achieve better training results. Most importantly, the method uses a softer way to switch downstream task data sets for different training objectives. This method can maintain the continuity of model training and avoid the impact of training interruptions.

[0062] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A multimodal traffic large model phased training method based on WSD learning rate scheduler, characterized in that: Include: Collect and preprocess raw traffic scene data, match and format the images and text annotations in the raw data, and generate the raw traffic scene data set; The original traffic scene dataset is enhanced and expanded through a data enhancement method based on human-computer collaborative programs, and multiple downstream task datasets suitable for different levels of training objectives in the field of traffic violation scene perception and analysis are obtained; A phased training strategy is formulated by combining the WSD learning rate scheduler and the task objectives of different downstream tasks.

2. The multimodal traffic large model phased training method based on WSD learning rate scheduler according to claim 1 is characterized in that: Get multiple downstream task datasets with different training objectives, including: First, a general multimodal large language model is used to generate scene-level image descriptions and extract key object features from them. These generated contents are confirmed by human annotators. Then, a general multimodal large language model is used to generate formatted related questions and initial responses. Finally, these question-answer pairs are verified again by human annotators.

3. The multimodal traffic large model phased training method based on WSD learning rate scheduler according to claim 1 is characterized in that: The downstream task datasets are classified into the following categories according to the task objectives: the downstream task datasets used to support the training objectives of traffic concept understanding of the basic large model include image caption datasets, object recognition datasets, and license plate number recognition datasets; The downstream task datasets used to support the training objectives of traffic scene reasoning of the basic large model include visual reasoning datasets, spatial reasoning datasets, and traffic violation recognition datasets.

4. The multimodal traffic large model phased training method based on WSD learning rate scheduler according to claim 3 is characterized in that: The downstream task dataset is defined as Among them, i n Indicates the instruction passed to the basic large model to be fine-tuned, v n Represents a visual sample, x n Represents the true value that should be output based on the given instructions and visual samples.

5. The multimodal traffic large model phased training method based on WSD learning rate scheduler according to claim 3 is characterized in that: The phased training strategy is formulated by combining the WSD learning rate scheduler and the task objectives of different downstream tasks. It consists of two parts: dividing the complete training process through the WSD learning rate scheduler, and investing data sets with different training objectives according to the characteristics of different learning stages.

6. The multimodal traffic large model phased training method based on WSD learning rate scheduler according to claim 5 is characterized in that: The complete training process is split by the WSD learning rate scheduler, including: After the warm-up phase, the WSD learning rate scheduler divides the training phase into a continuous training phase and a learning rate decay phase. The changes in the learning rates in these three different phases are expressed as follows: Where W and T represent the number of training steps in the warm-up phase and the continuous training phase, respectively, 0<f(sT)≤1 is a decreasing function of the number of training steps s, and η is the maximum learning rate.

7. The multimodal traffic large model phased training method based on WSD learning rate scheduler according to claim 5 is characterized in that: According to the characteristics of different learning stages, data sets with different training objectives are used, including: In the continuous training phase, traffic concept understanding is set as the main training goal of this phase, and a large number of image caption datasets, object recognition datasets, and license plate recognition datasets are used for model training; In the learning rate decay phase, traffic scene reasoning is taken as the main training target of this phase, and a large number of visual reasoning datasets, spatial reasoning datasets and traffic violation recognition datasets are used for model training.

8. A multimodal traffic large model phased training system based on WSD learning rate scheduler, characterized in that: It includes the original dataset construction module, data enhancement module and phased training module, among which: The original data set construction module is used for the collection and preprocessing of original traffic scene data, matching and formatting the images and text annotations in the original data, and generating the original traffic scene data set; The data enhancement module is used to enhance and expand the original traffic scene dataset through a data enhancement method based on human-computer collaborative programs, and obtain multiple downstream task datasets suitable for different levels of training objectives in the field of traffic violation scene perception and analysis; The phased training module is used to formulate a phased training strategy by combining the WSD learning rate scheduler and the task objectives of different downstream tasks.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images