Road safety early warning method and system based on mixed precision quantification visual large model

Abstract

The invention discloses a road safety early warning method and system based on a mixed precision quantification visual large model, and the method comprises the steps: collecting road traffic safety videos and pictures, and carrying out the preprocessing, data enhancement and marking, thereby forming a diversified data set; a pre-trained visual large model is selected as a teacher model, after fine tuning, output layer and middle layer knowledge is extracted, key features are weighted, and meanwhile, a lightweight neural network is taken as a student model, same input is received, and prediction and middle feature maps are output. And inputting data into the two models and the student model to carry out mixing precision quantification forward propagation, constructing a total loss function containing tasks, knowledge distillation and quantification learning loss, and updating parameters through back propagation. And after training is completed, exporting a quantitative model, and deploying the quantitative model to an edge computing platform to realize safety early warning. The lightweight model can realize rapid reasoning on edge equipment such as a vehicle-mounted road side, and the problem that performance and efficiency are difficult to consider in a traditional model compression method is solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent transportation and deep learning, and more particularly relates to a road safety warning method and system based on mixed precision quantization of a visual large model. BACKGROUND

[0002] With the rapid development of modern society, intelligent transportation and autonomous driving technology have made remarkable progress. In this context, real-time and accurate monitoring and warning of road driving safety has become a key link to ensure the efficient operation of the transportation system and reduce accidents. Using real-time video streams obtained by vehicle-mounted or roadside cameras and other visual sensors, timely identification of various disasters and accidents that may affect vehicle driving safety, such as traffic accidents, road obstacles, road slippery or water accumulation due to extreme weather, landslides, and road collapses, is of great significance to improving traffic efficiency and protecting public life and property safety.

[0003] Currently, road safety risk identification and warning technology based on visual information mainly includes traditional computer vision methods and deep learning-based methods. Traditional computer vision methods usually rely on manually designed feature extraction algorithms and machine learning classifiers. For example, researchers often use color, texture, edge, and other low-level visual features in combination with support vector machines (SVM), Adaboost, and other classifiers to identify specific traffic events or obstacles; some methods use scale-invariant feature transform (SIFT), speeded up robust features (SURF), histogram of oriented gradients (HOG), and other feature descriptors to represent image content and identify them through template matching or classification models. However, such methods have many flaws that cannot be ignored: manually designed features are designed for specific scenarios and targets, and in complex and variable road environments, such as lighting conditions (day, night, strong light, backlight), weather conditions (sunny, rainy, snowy, foggy), road conditions (dry, slippery, water accumulation, icy), and the influence of various sudden obstacles and events, the robustness and generalization ability are severely insufficient, and the cost of redesigning and adjusting the feature extraction algorithm is high; some traditional feature extraction algorithms have high computational complexity, making it difficult to meet the real-time processing of video streams, while faster feature expression capabilities are limited; traditional methods lack deep semantic understanding of scene content, making it difficult to accurately distinguish between ordinary garbage and dangerous obstacles on the road and to judge the interaction between vehicles in complex traffic accidents, severely limiting accurate judgment and warning of potential risks.

[0004] To overcome the limitations of traditional methods, deep learning techniques are introduced into the field of road safety risk identification, and convolutional neural networks (CNN) are widely used. Based on deep learning, methods can automatically learn feature representations from large amounts of data, and have made significant progress in tasks such as target detection, image segmentation, and behavior recognition. However, existing general deep learning models still face serious challenges when faced with complex and variable road safety risk identification tasks: the performance of deep learning models is highly dependent on training data, and the collection of road safety scene data is difficult, leading to a decline in model performance in un-covered scenarios or when encountering new risk types, and limited generalization ability; conventional CNN models have limitations in understanding complex scene semantics, target relationships, and event causal logic; to pursue high accuracy, the complexity of deep learning models is increasing, with huge parameter quantities and computational loads, making real-time deployment on resource-constrained edge devices difficult, and existing model compression and quantization techniques also have difficulty in reducing computational load while effectively preserving key risk information identification capabilities.

[0005] In summary, existing road driving safety risk warning technologies have obvious shortcomings in terms of computational efficiency, generalization ability, semantic understanding, and model deployment, and cannot meet the urgent needs of modern intelligent transportation systems for real-time, accurate, and reliable road driving safety warning. Therefore, an innovative technical solution is needed to effectively overcome the above-mentioned deficiencies and achieve efficient, accurate, and reliable real-time road driving safety warning. SUMMARY

[0006] The present application aims to solve the deficiencies of existing road driving safety risk warning methods. It overcomes the problems of traditional computer vision methods, such as dependence on manual features and poor robustness, improves the generalization ability of general deep learning models and their difficulty in understanding complex scenes, breaks through the difficulties of deploying high-performance models and model compression and performance preservation, utilizes the capabilities of visual large models and solves their deployment problems, and achieves efficient, accurate, robust, and general road driving safety real-time warning to meet the needs of modern intelligent transportation systems.

[0007] To address the above deficiencies or improvement needs of the prior art, as a first aspect of the present application, the present application provides a road safety warning method based on a mixed precision quantization visual large model, comprising:

[0008] S1. Collect road driving safety related videos and pictures, pre-process, data enhance and label, form large-scale diversified dataset;

[0009] S2. Prepare the teacher model and build the knowledge distillation framework, select a pre-trained visual large model as the teacher model, save it after fine-tuning on demand, and during training, extract the output layer and intermediate layer knowledge by forward propagation of the teacher model for each input sample and evaluate the feature importance to weight the key features, while selecting a lightweight neural network architecture as the student model, which receives the same input as the teacher model and outputs its own prediction and intermediate feature map for feature distillation;

[0010] S3. Input the sampled data into the teacher model to obtain knowledge forward propagation with mixed precision quantization operation; build a total loss function L including task loss, knowledge distillation loss, and quantization learning loss total ; task loss supervises the risk identification performance of the student model, knowledge distillation loss transfers the knowledge of the teacher model to the student model, and quantization learning loss punishes high bit cost to drive adaptive quantization strategy; perform back propagation to calculate L total the student model weight W s and the quantization strategy parameters, and use the optimizer to update these parameters;

[0011] S4. After training, export the solidified model, deploy the exported quantized model to the target edge computing platform, and perform road driving safety warning.

[0012] Further, the specific process of collecting road driving safety related videos and pictures in S1, preprocessing, data augmentation and labeling to form a large-scale diversified dataset is as follows:

[0013] Based on the pre-prepared search word library, search for videos and pictures related to road driving safety; the videos and pictures contain diversified road scenes, traffic participants and potential safety risks;

[0014] Preprocess the searched videos and pictures, including video frame extraction, picture cropping, scaling, grayscale, normalization and other operations;

[0015] Label the preprocessed videos and pictures with a sentence describing the content in the video or picture;

[0016] Divide the labeled video and picture dataset into training set, validation set and test set according to the preset proportion;

[0017] Save the training set, validation set and test set as different dataset files.

[0018] Further, the output layer and intermediate layer knowledge extracted in S2 are as follows:

[0019] During the student model training process, a forward propagation through the teacher model is needed for each input training sample to extract the knowledge it contains. These knowledge mainly include two aspects:

[0020] Output layer knowledge z T : the final prediction result of the teacher model for the input image x; for classification tasks, it is the logits vector z T output by the classifier layer

[0021] Intermediate layer feature knowledge The feature vector output by at least one intermediate layer of the teacher model where j is the index of the feature.

[0022] Further, the method for evaluating feature importance in S2 is:

[0023] Gradient-based importance evaluation: calculate the gradient norm of the final task loss of the teacher model with respect to the intermediate feature A larger gradient means that the feature has a greater impact on the final prediction result

[0024] Attention mechanism-based analysis: when the teacher model is based on Transformer or contains attention modules, the internal attention weight map is directly used. High attention score areas / feature channels correspond to information that the model considers important

[0025] Feature activation-based analysis: analyze the feature maps that show stronger activation responses when inputting images containing specific disasters / risk

[0026] Domain knowledge guidance: combine prior knowledge in the field of road safety to determine the types of features that are critical for the warning task.

[0027] Further, the method for calculating the task loss in S3 is:

[0028] Constructing task loss L task : this loss is used to directly supervise the performance of the student model on the final target task, i.e., accurately identifying road safety risks

[0029] For risk classification tasks: use the standard cross-entropy loss:

[0030]

[0031] where C is the total number of classes, y c is the cth element of the label y, and p S，c = Softmax(z s ) cis the probability that the student model predicts that the input belongs to category c;

[0032] Use focal loss to increase attention to difficult samples when the class imbalance problem is serious:

[0033]

[0034] Among them, a c is the class weight, γ is the focusing parameter;

[0035] For risky object detection or segmentation tasks: L task Contains classification loss and regression loss.

[0036] Furthermore, the calculation method of the knowledge distillation loss in S3 is:

[0037] Constructing knowledge distillation loss L KD , L KD Include at least one of the following components:

[0038] Output layer distillation: Let the output probability distribution of the student model mimic the output probability distribution of the teacher model; use KL divergence to measure the difference between the two probability distributions:

[0039]

[0040] Among them, p S (T) = Softmax(z S / T) and p T (T) = Softmax(z T / T) is the softened probability distribution calculated using a distillation temperature T>1; a higher temperature T will make the probability distribution smoother and emphasize the information transfer of negative labels; T 2 Factor is used to maintain the gradient magnitude;

[0041] Intermediate layer feature map distillation: Enforce the intermediate features of the student model Approximate the characteristics corresponding to the teacher model under some metric

[0042]

[0043] Among them, j is a traversal of all intermediate layer pairs involved in distillation; and It is an optional feature conversion function, and sometimes the original features can be directly compared; represents the p-th power of the Lp norm; β j is the weight of the j-th pair of feature distillation;

[0044] The total knowledge distillation loss is the weighted sum of the losses of each distillation method:

[0045] L KD = α soft L KD,soft + α feature L KD,feature +...

[0046] where α soft , α feature ,... are weight coefficients of each distillation method.

[0047] Further, the design method of the S3 quantization learning loss is:

[0048] Construct a quantization learning loss / regularization term L quant for driving the model to learn the optimal bit allocation strategy {b l} in training.

[0049] Define the quantifiable layer l in the model and the set of bit number options allowed by each quantifiable layer B={b1, b2,..., b k};

[0050] Introduce learnable parameters to control the selection of the number of bits, including:

[0051] Assign a learnable parameter including a gating value g l,b , a probability p l,b to each bit option b∈B for each layer l; normalize these parameters through the Softmax function, so that

[0052] In the forward propagation process of training, the actual number of bits b l used is determined according to the learned probability, and the quantization operation Q(x, b l ) is performed; and the non-differentiable problem caused by discrete selection is handled through methods including differentiable sampling and expected quantization from the category distribution;

[0053] Construct the quantization function Q(x, b) through methods including uniform quantization, i.e. linearly map the floating-point number to 2 b discrete integer values;

[0054] Introduce a continuous learnable parameter c l for each layer l, and get the discrete bit number through the mapping function b l = Map(c l ); in order to let the gradient be able to backpropagate, the quantization operation Q itself needs to use a differentiable approximation, which performs the real quantization operation in the forward propagation, but directly passes the gradient to the floating-point value in the backpropagation;

[0055] L quantThe objective is to guide the model to learn the desired quantization strategy; then is designed to penalize the model for choosing higher bits.

[0056] Further, the total loss function L total is calculated as:

[0057] The total loss function L total is the weighted sum of the task loss, the knowledge distillation loss and the quantization learning loss:

[0058]

[0059] where z S is the output of the student model, y is the true label; is the knowledge extracted from the teacher model; is the intermediate feature of the student model;{b l} represents the learned quantization bit strategy of each layer; λ task , λ KD , λ quant are weight hyperparameters of each loss, used to balance the importance between different optimization objectives.

[0060] As a second aspect of the present application, a road safety warning system based on mixed precision quantization visual large model is provided, comprising:

[0061] A data construction unit is configured to collect road driving safety related videos and pictures, pre-process, data enhance and label to form a large-scale diversified data set;

[0062] A model architecture unit is configured to prepare a teacher model and construct a knowledge distillation framework, select a pre-trained visual large model as the teacher model, save after on-demand fine-tuning, and extract the output layer and intermediate layer knowledge by forward propagation of the teacher model for each input sample during training, and evaluate the feature importance to weight the key features, while selecting a lightweight neural network architecture as the student model, which receives the same input as the teacher model and outputs its own prediction and intermediate feature map for feature distillation;

[0063] A training optimization unit is configured to input the sampled data into the teacher model to obtain knowledge and input the student model to perform forward propagation with mixed precision quantization operation; construct a total loss function L total including task loss, knowledge distillation loss, and quantization learning loss; the task loss supervises the risk identification performance of the student model, the knowledge distillation loss transfers the knowledge of the teacher model to the student model to make up for the possible performance loss due to the reduction of model size, and the quantization learning loss punishes high bit cost to drive adaptive quantization strategy; perform back propagation to calculate L total on the student model weight W Sand quantization strategy parameters, and updating the parameters using an optimizer;

[0064] A model deployment unit is configured to export the solidified model after the training is completed, deploy the exported quantized model to a target edge computing platform, and perform road driving safety early warning.

[0065] As a third aspect of the present application, a computer readable storage medium is provided, which stores a computer program executed by a processor to implement any of the above methods.

[0066] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0067] 1. The road safety early warning method based on the mixed precision quantized visual large model of the present application uses a pre-trained visual large model as a knowledge source, breaks through the limitations of traditional methods through cross-model capability transfer, selects a pre-trained model with strong general visual understanding capability as a teacher model, and uses the rich visual patterns and semantic concepts learned by the teacher model on a super large-scale data set to provide a cross-scene generalization knowledge base for road safety risk identification. Through an optional fine-tuning mechanism, the teacher model is further guided to focus on the road safety early warning task, and the sensitivity to risk features in the traffic environment is enhanced. This design enables the model to inherit the semantic understanding advantages brought by large-scale pre-training, significantly improves the analysis capability for risks in complex scenes (such as hidden obstacles and multi-factor related disasters), and makes up for the semantic understanding defects of traditional methods relying on manual features.

[0068] 2. The road safety early warning method based on the mixed precision quantized visual large model of the present application realizes precise transfer of the core capabilities of the visual large model through a hierarchical knowledge distillation and dynamic feature weighting mechanism. By extracting the soft information of the output layer of the teacher model containing the probability distribution between classes and the multi-level abstract expression of the intermediate layer features covering from low-level visual features to high-level semantics, and combining gradient analysis attention mechanism and other methods to evaluate feature importance, the method assigns different distillation weights to different levels of features, and guides the student model to focus on learning semantic information critical to risk identification. This mechanism enables the lightweight student model to effectively inherit the visual understanding capability of the teacher model while significantly reducing the parameter amount, significantly improves the analysis and judgment capability for early disaster signs in complex risk scenes such as multi-vehicle accident related potential congestion environments with high integration degree, and breaks through the limitations of traditional methods in semantic understanding and generalization capability.

[0069] 3. The road safety warning method based on the mixed-precision quantization visual large model of the application, through the innovative adaptive mixed-precision quantization framework, realizes the optimization balance of model efficiency and performance. Through the learning parameter, the model is dynamically allocated with differentiated quantization precision, the core layer processing key semantic information is reserved with higher precision, and the non-key layer is compressed with low precision, and the task performance knowledge distillation and quantization efficiency are optimized in the unified loss function. The framework significantly reduces the model calculation complexity and memory occupation, and maximizes the identification ability of the model to the risk characteristics, so that the lightweight model can realize fast reasoning on the edge device such as vehicle-mounted roadside, meet the strict requirements of real-time warning on computing resources and response speed, and solve the problem that the performance and efficiency are difficult to balance in the traditional model compression method. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The flowchart of the road safety warning method based on the mixed-precision quantization visual large model of the embodiment of the application;

[0071] Figure 2 The overall flowchart of the method of the embodiment of the application.

[0072] Figure 3 The student model architecture diagram of the embodiment of the application

[0073] Figure 4 The road safety sample set part sample diagram of the embodiment of the application

[0074] Figure 5 The diagram of the relationship between the teacher model and the lightweight student model architecture and the loss function of the embodiment of the application;

[0075] Figure 6 The loss function curve diagram of the embodiment of the application;

[0076] Figure 7 The part sample recognition result display diagram of the embodiment of the application;

[0077] Figure 8 The confusion matrix diagram of the recognition result of the embodiment of the application;

[0078] Figure 9 The system unit diagram of the embodiment of the application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0080] Embodiment 1

[0081] For reference Figure 1 The embodiment 1 provides a road safety warning method based on mixed precision quantization of a visual large model, comprising:

[0082] S1. Collecting road driving safety related videos and pictures, pre-processing, data enhancement and labeling to form a large-scale diversified data set;

[0083] S2. Preparing a teacher model and building a knowledge distillation framework, selecting a pre-trained visual large model as the teacher model, saving after fine-tuning as needed, and during training, extracting the output layer and intermediate layer knowledge by forward propagation of the teacher model for each input sample and evaluating the feature importance to weight the key features, while selecting a lightweight neural network architecture as the student model, which receives the same input as the teacher model and outputs its own prediction and intermediate feature map for feature distillation;

[0084] S3. Inputting the sampled data into the teacher model to obtain knowledge Forward propagation with mixed precision quantization operation into the student model; building a total loss function L including task loss, knowledge distillation loss and quantization learning loss total ; Task loss supervises the risk identification performance of the student model, knowledge distillation loss transfers the knowledge of the teacher model to the student model, making up for the possible performance loss due to the reduction of model size, and quantization learning loss punishes high bit cost to drive adaptive quantization strategy; performing back propagation to calculate L total the student model weight W S and the quantization strategy parameters, and updating these parameters using an optimizer;

[0085] S4. After training, export the solidified model, deploy the exported quantized model to the target edge computing platform, and perform road driving safety warning.

[0086] For reference Figures 2-8 The embodiment 1 specifically expands the above method.

[0087] (1) Data construction

[0088] Based on the pre-prepared search word library, a large number of road driving safety related videos and pictures are searched from the Internet. Ensure that it contains diversified road scenes (different weather, light, time, geographical location), traffic participants and various potential safety risks (such as traffic accident scene, litter, potholes, water accumulation, landslide signs, construction area, etc.).

[0089] Preprocess the searched videos and pictures, including video frame extraction, picture cropping, scaling, grayscale, normalization, etc.

[0090] The pre-processed videos and images are annotated with a one-sentence description of the content in the video or image.

[0091] The annotated video and image dataset is divided into training, validation, and test sets in a certain ratio.

[0092] The training, validation, and test sets are saved as different dataset files for subsequent model training, evaluation, and testing. Finally, a large-scale, diverse annotated dataset is formed, containing as many road scene types as possible (urban, highway, rural; day, night, dawn, dusk; sunny, rainy, snowy, foggy, etc.) and various safety risk instances that require early warning (different types of traffic accidents, various sizes and shapes of obstacles / spills, road abnormalities such as potholes / water / ice, landslide or rockfall signs, congestion ahead, abnormal weather effects, etc.).

[0093] In a specific preferred embodiment, the data augmentation techniques and data annotation classification of the road driving safety dataset are as follows:

[0094] Geometric transformation: random cropping, rotation (±15°), horizontal flipping

[0095] Light variation: brightness (±20%), contrast (±20%), saturation adjustment

[0096] Weather simulation: rain, snow, and fog effect synthesis

[0097] Lighting conditions: day-night conversion, shadow addition

[0098] Mixed techniques: CutMix and Mixup methods mix different samples

[0099] The data annotation uses a multi-level classification system, and the main risk categories include: 1. normal; 2. ice and snow; 3. landslide (mudslide, etc.); 4. road obstacles (large obstacles, small spills, etc.); 5. traffic accidents (vehicle collision, fire, rollover, etc.); 6. road abnormalities (potholes, collapse, cracks, water, ice, etc.); 7. weather-induced risks (low visibility, slippery road, etc.); 8. bridge collapse; 9. others

[0100] The final dataset is divided into training set (40,000), validation set (5,000), and test set (5,000) in the ratio of 8:1:1.

[0101] (2) Model architecture

[0102] A pre-trained visual large model (VLM) with strong general visual understanding capability and good generalization on complex scenes is selected as the teacher model. Suitable VLMs can be models based on the Transformer architecture, such as Vision Transformer (ViT) and its variants, or models based on large-scale convolutional networks combined with attention mechanisms, or the visual encoder part of multi-modal pre-training models (such as CLIP, ALIGN). These models are usually pre-trained on super-large datasets (such as ImageNet-21k, JFT-300M, LAION) and learn rich visual patterns and semantic concepts.

[0103] To make the teacher model more suitable for the road safety warning task, optional fine-tuning can be performed.

[0104] The fine-tuned teacher model is saved as a model file for subsequent knowledge distillation and adaptive mixed-precision quantization.

[0105] In a preferred embodiment, the specific method of extracting "knowledge" from the teacher model to guide the student model learning is:

[0106] During the subsequent student model training process, for each input training sample (e.g., an image frame x taken from a real-time video stream), a forward propagation through the teacher model is required to extract the knowledge it contains. These knowledge mainly includes two aspects:

[0107] Output layer knowledge (Soft Targets) z T : The final prediction result of the teacher model for the input image x. For classification tasks (e.g., determining whether there is a risk or the type of risk), this is usually the logits vector z T output by the classifier layer. These logits contain the "soft" information of the probability distribution of different classes of the teacher model, which can better reflect the similarity between classes and the "thinking" process of the teacher model than hard labels (one-hot vector).

[0108] Intermediate layer feature knowledge (Intermediate Features) The feature map or feature vector output by one or more intermediate layers (or modules) inside the teacher model where j is the index of the layer or feature. These intermediate features contain the teacher model's understanding of the input image at different levels of abstraction, for example, low-level features may contain edge, texture information, and high-level features may encode semantic information such as objects, scene layout, etc.

[0109] Key feature identification and weighting. To more effectively guide the student model to learn information crucial for road safety risk identification, it is necessary to identify and focus on those teacher features that are highly relevant to the task objective. According to the importance scores obtained from evaluation, different weights β are assigned to different teacher features j , giving more important features greater weights in subsequent knowledge distillation loss calculation, guiding the student model to prioritize learning these key knowledge. The importance of different features in the teacher model can be evaluated in one or more of the following ways:

[0110] Gradient-based importance evaluation: Calculate the gradient norm of the final task loss of the teacher model with respect to the intermediate features. Features with larger gradients usually mean that the feature has a greater impact on the final prediction result.

[0111] Attention mechanism-based analysis: If the teacher model is based on Transformer or contains attention modules, the internal attention weight map can be directly utilized. Areas or feature channels with high attention scores usually correspond to information that the model considers important.

[0112] Feature activation-based analysis: Analyze which feature maps show stronger activation responses when the input contains images containing specific disasters or risks.

[0113] Domain knowledge guidance: Combine prior knowledge in the field of road safety to determine which types of features (e.g., features related to abnormal objects, road conditions, potential dangers in the distance) are more critical for the early warning task.

[0114] In a preferred embodiment, a specific method for designing a student model that is computationally efficient and suitable for running on edge devices is as follows:

[0115] Select or design a lightweight neural network architecture with fewer parameters and lower computational complexity (low FLOPs) as the basis for the student model M S . Common choices include:

[0116] Efficient convolutional networks: MobileNet series (v1, v2, v3), ShuffleNet series (v1, v2), EfficientNet-Lite series, GhostNet, etc. These networks use techniques such as depthwise separable convolution, group convolution, channel shuffle, inverted residual structure, neural network architecture search (NAS), etc. to reduce computational complexity.

[0117] ​​Lightweight Transformers: Models like MobileViT attempt to combine the global modeling capabilities of Transformers with the efficiency of CNNs, or reduce the complexity of Transformers through optimized attention computation, model structure.

[0118] Customized Architectures: Designing specialized lightweight network structures based on specific hardware platform characteristics and task requirements.

[0119] Balancing Model Capacity and Distillation Needs. The capacity of the student model needs to be large enough to effectively absorb and represent the complex knowledge distilled from the teacher model; but at the same time, it needs to be small enough to meet the efficiency requirements of edge deployment. This is a design choice that needs to be balanced and adjusted based on experimental results. The student model receives the same input image x as the teacher model and outputs its own prediction Z S and intermediate feature maps for feature distillation It is necessary to ensure that there are corresponding intermediate features in the student model that can be compared with the selected teacher features

[0120] In a particularly preferred embodiment, the pre-trained visual language large model DeepSeek-VL2 is selected as the teacher model. This model has strong general visual understanding ability and good generalization to complex scenes, and is suitable as the source of knowledge distillation.

[0121] To make the teacher model more suitable for the road safety warning task, we perform two-stage fine-tuning:

[0122] 1. First, fine-tune on general driving scene datasets such as BDD100K and Waymo to enhance its understanding of traffic environment.

[0123] 2. Then, fine-tune on the road safety risk dataset we constructed to focus on identifying specific safety risks.

[0124] The fine-tuning process uses the AdamW optimizer with a learning rate of 1e-5, uses the cosine annealing scheduling strategy, the batch size is 32, and trains for 80 epochs. The fine-tuned DeepSeek-VL2 model has high performance for this task, and its output and intermediate layer features will be used to guide the learning of the student model.

[0125] In a particularly preferred embodiment, the student model is designed as follows:

[0126] ​The student model is designed based on DeepSeek-VL2-tiny. DeepSeek-VL2-tiny is a light version of DeepSeek-VL2, aiming to significantly reduce the model parameter amount and computational complexity, while retaining the key visual understanding capabilities of the original model as much as possible, making it suitable for deployment on edge devices with limited computing resources.

[0127] The architecture of the student model is adapted for the road driving safety warning task, which aims to efficiently output risk category judgments or related features after receiving input images (e.g., 224x224x3 RGB images). Compared to the teacher model DeepSeek-VL2, DeepSeek-VL2-tiny has significantly reduced parameter amount and computational complexity (GFLOPs), thus meeting the needs of real-time processing on edge devices.

[0128] (3) Training optimization

[0129] Sample a mini-batch of input images x and their corresponding labels y from the dataset.

[0130] Input x into the teacher model (usually freeze its parameters, only forward propagation), get teacher knowledge

[0131] Input x into the student model. During forward propagation, perform mixed-precision quantization operations (e.g., using Gumbel-Softmax sampling or expected quantization) on the weights and / or activations of each layer according to the current learned quantization strategy parameters (e.g., p l，b ). Get the output z S and intermediate features

[0132] According to z S , y, z T , and the learned quantization strategy {p l，b} or {c l}, calculate the total loss L total . A unified loss function is used to optimize the task performance, knowledge absorption, and adaptive mixed-precision quantization strategy of the student model simultaneously.

[0133] Perform backpropagation to calculate the gradient of L total on the student model weights W S and quantization strategy parameters (as well as the learnable parameters of Q). Update these parameters using an optimizer.

[0134] (4) Loss function construction

[0135] In a preferred embodiment, its total loss function Ltotal The construction method is as follows:

[0136] 4.1 Construction of task loss L task

[0137] This loss is used to directly supervise the performance of the student model on the final target task, i.e., accurately identifying road safety risks.

[0138] For risk classification tasks: (e.g., determining whether a risk exists in an image or which risk type it belongs to: accident, obstacle, slippery, collapse, etc.), a standard Cross-Entropy Loss can be used:

[0139]

[0140] where C is the total number of categories, y c is the c-th element of the label y (for one-hot encoding, the true category is 1 and others are 0), p S，c = Softmax(z S ) c is the probability of the student model predicting that the input belongs to category c.

[0141] In cases where the class imbalance problem is severe (some risk types are very rare), Focal Loss can be considered to increase attention to difficult samples:

[0142]

[0143] where α c is the category weight and γ is the focus parameter.

[0144] For risk object detection or segmentation tasks: (e.g., locating accident vehicles, obstacle areas), L task will be more complex, usually containing classification loss (judging which category the detection box or pixel belongs to) and regression loss (e.g., Smooth L1Loss or IoU Loss, used to optimize the accuracy of bounding box coordinates or segmentation mask).

[0145] 4.2 Construction of knowledge distillation loss L KD

[0146] This loss aims to transfer the knowledge of the teacher model to the student model, making up for the performance loss due to model size reduction, and improving the generalization ability of the student model. L KD may contain one or more of the following components:

[0147] Soft Target Distillation: Let the output probability distribution of the student model mimic that of the teacher model. This helps the student learn the teacher's understanding of inter-class relationships. The Kullback-Leibler Divergence is often used to measure the difference between two probability distributions:

[0148]

[0149] where p S (T) = Softmax(z S / T) and p T (T) = Softmax(z T / T) are softened probability distributions computed using a distillation temperature T > 1. Higher temperature T makes the probability distribution smoother, emphasizing the transfer of information for negative labels (non-correct classes). T 2 factors are used to keep the gradient magnitude.

[0150] Feature Map Distillation: Force the student model's intermediate feature to approximate the corresponding feature of the teacher model under some metric. This helps the student learn the teacher's feature representation ability at the intermediate stage.

[0151]

[0152] where j is the index of all pairs of intermediate layers participating in distillation. and are optional feature transformation functions (e.g., using a 1x1 convolutional layer for channel number matching or feature space alignment), sometimes the original features can also be compared directly (if the dimensions are compatible); represents the p-th power of the Lp norm, commonly used are L2 norm (p = 2, Mean Squared Error MSE) or L1 norm (p = 1, Absolute Error); β j is the weight of the j-th pair of feature distillation, which can be set according to the importance of the teacher feature . For example, β j can be proportional to the gradient norm or attention score of , so that the student model focuses on mimicking those teacher features that contribute more to the final task.

[0153] The total knowledge distillation loss is the weighted sum of the losses of each distillation method:

[0154] L KD = α soft L KD,soft + αfeature L KD,feature +...

[0155] where α soft ,α feature ,... are weight coefficients for each distillation method.

[0156] 4.3 Constructing the quantization learning loss / regularizer L quant

[0157] This term is the key to achieve adaptive mixed-precision quantization, which is used to drive the model to learn the optimal bit allocation strategy {b l} during training, i.e., to decide how many bits should be used to quantize each quantizable layer l (including the weights w l and / or activations a l ) in the model. The goal is to reduce the overall computation and storage cost of the model as much as possible (usually by encouraging lower bit numbers) while preserving the performance of the task (constrained by L task and L KD ) and protecting the layers that are crucial to the task performance by assigning them higher precision.

[0158] Quantizable units and bit options: define which layers l in the model are quantizable and the set of bit number options B = {b1, b2,..., b k} that each quantizable layer is allowed to have, e.g., B = {2, 4, 8} bit integers, possibly including 16-bit floating point (FP16) or 32-bit floating point (FP32, i.e., no quantization).

[0159] Quantization strategy parameterization and learning mechanism: introduce learnable parameters to control the selection of bit numbers. Common methods include:

[0160] Gate / probability-based methods: assign a learnable parameter, e.g., a gate value g l，b or a probability p l，b , to each bit option b e B for each layer l. These parameters can be normalized by a Softmax function, so that

[0161] Quantization in forward propagation: during the forward propagation of training, the actual bit number b l to be used needs to be determined according to these learned probabilities and the quantization operation Q(x, b l ) is performed.

[0162] This usually requires dealing with the non-differentiable problem caused by discrete selection. A common technique is Gumbel-Softmax / Concrete Distribution, which allows differentiable sampling from the category distribution to obtain an approximate one-hot vector, thereby selecting a specific number of bits b l To quantify: Another approach is to use expected quantization, which is to perform a weighted average of all possible quantization results according to their probabilities: The latter is usually more stable during training, but still requires a fixed number of bits to be selected during inference.

[0163] Quantization function Q(x,b): refers to the function that quantizes the floating point value x (weight or activation) to b bits. The commonly used one is uniform quantization, which linearly maps the floating point number to 2 b Its parameters (scaling factors scales and zero-point z) can be fixed (determined according to the numerical range) or learnable (optimized together with the model weights). For example:

[0164] Symmetric uniform quantization: Q sym (x, b) = round(clip(x / s, -2 b-1 , 2 b-1 -1)),

[0165] Asymmetric uniform quantization: Q asym (x,b)=round(clip(x / s+z,0,2 b -1)).

[0166] Continuous relaxation-based method: Introduce a continuous learnable parameter c for each layer l l , for example, to represent the desired number of bits or degree of quantization. Then, through a (usually non-differentiable) mapping function b l =Map(c l ) to get the discrete bit number. In order for the gradient to be back-propagated, the quantization operation Q itself needs to use a differentiable approximation, such as the Straight-ThroughEstimator (STE), which performs the actual quantization operation during forward propagation but directly passes the gradient to the floating-point value during back-propagation (ignoring the derivatives of the round and clip operations).

[0167] Design of quantization loss / regularization term: L quant The goal is to guide the model to learn the desired quantization strategy. It is usually designed to penalize the model for choosing a higher number of bits or to penalize the overall complexity of the model.

[0168] 4.4 Overall loss function Ltotal

[0169] An overall loss function L is designed to contain multiple components total to guide the whole training process. The loss function is usually a weighted sum of task loss, knowledge distillation loss and quantization learning loss:

[0170]

[0171] where z S is the output of the student model, y is the ground truth; is the knowledge extracted from the teacher model; is the intermediate feature of the student model;{b l} represents the learned quantization bit strategy of each layer; λ task , λ KD , λ quant are the weight hyperparameters of each loss to balance the importance between different optimization objectives. These weights can need to be carefully tuned or dynamically changed during training.

[0172] In a preferred embodiment, the overall loss L total is trained as follows:

[0173] Staged training: One can first perform pure knowledge distillation ( λ quant = 0) for a while to let the student model learn the teacher knowledge first, and then introduce the quantization learning loss, or gradually increase the weight of λ quant .

[0174] Quantization-aware training: This method is essentially an advanced QAT, because it considers the impact of quantization during training. After training, a quantized model can be directly obtained.

[0175] In a preferred specific embodiment, the specific design of the loss function in the joint optimization training framework is as follows

[0176] Task loss L task : Due to the class imbalance problem (e.g., landslide samples are much less than obstacle samples), focal loss (FocalLoss) is adopted:

[0177]

[0178] where α c is set according to the class frequency, γ = 2.0

[0179] Knowledge distillation loss L KD :

[0180] LKD = 0.5 x KL d iv(z S / T, z T / T) x T 2 + 0.5 x sum(w j ||f Sj -f Tj || 2 )

[0181] where distillation temperature T = 3.0, feature weight w j Dynamically calculated according to teacher model attention map or gradient importance.

[0182] Quantization loss L quant :

[0183] L quant = gamma x sum[(1 - Importance(l)) e ta x sum(p l，b x b)]

[0184] where gamma = 0.01 controls quantization pressure, eta = 2 controls importance modulation degree, Importance(l) is calculated based on gradient size.

[0185] Total loss function:

[0186] L total = lambda task L task + lambda KD L KD + lambda quant L quant

[0187] In a preferred embodiment, the stage-by-stage strategy adopted in the joint optimization training framework is designed as follows:

[0188] First 600 epochs: lambda task = 1.0, lambda KD = 0.8, lambda quant = 0.0, pure knowledge distillation;

[0189] 600-2000 epochs: lambda task = 1.0, lambda KD = 0.8, lambda quant = 0.005, introduce quantization loss;

[0190] 2000-4000 epochs: lambda task = 1.0, lambda KD = 0.5, lambda quant = 0.01, enhance quantization pressure.

[0191] Optimizer: AdamW, initial learning rate 1e-3, cosine annealing schedule, batch size 64, trained on 1 server with RTX 4060Ti for about 48 hours.

[0192] (5) Model deployment unit

[0193] 5.1 Model export and solidification: After training, the quantization strategy (i.e. the number of bits b l used by each layer in the final model) is determined. Usually, the number of bits corresponding to the maximum probability p l，b is selected, or a fixed bit allocation scheme is selected according to the expected cost and other indicators. Then, the student model and its corresponding fixed mixed-precision quantization parameters (the number of bits b l for each layer, and the scale and zero-point of the quantization function Q) can be exported. This exported model is a lightweight model that can be directly used for low-precision inference.

[0194] 5.2 Edge platform deployment: Deploy the exported quantization model to the target edge computing platform. This may include:

[0195] On-board computing unit: integrated into the vehicle's advanced driver assistance system (ADAS) or autonomous driving (AD) system.

[0196] Roadside unit (RSU): installed on the roadside to monitor specific road sections and broadcast warning information to nearby vehicles through vehicle-to-infrastructure (V2X) communication.

[0197] Mobile device: such as a smartphone, used for assisted driving applications.

[0198] When deployed, it may be necessary to use the inference engine and optimization library of the specific hardware platform (such as NVIDIA TensorRT, Qualcomm SNPE, Google TensorFlow Lite, OpenVINO, etc.) to further compile and optimize the model, so as to fully utilize the low-precision computing capabilities of hardware (such as NPU, DSP, GPU), achieve the best inference speed and energy efficiency.

[0199] Real-time warning system operation: after deployment, the system enters real-time operation state:

[0200] 1. Continuously capture road video streams through on-board or roadside cameras.

[0201] 2. Preprocess the video frames (such as scaling, normalization).

[0202] 3. Input the preprocessed image frames into the deployed mixed-precision quantized student model for fast forward inference.

[0203] 4. The model outputs the judgment result (e.g., risk category probability, bounding box position, segmentation mask) of whether there is a driving safety risk in the current scene and the corresponding confidence score.

[0204] 5. According to the output result of the model, combined with the preset threshold and rules, trigger the early warning mechanism.

[0205] Since the model has undergone joint optimization and adaptive mixed precision quantization, it has fast inference speed and low memory occupation, and can meet the real-time (e.g., processing tens of frames per second) warning needs of road scenes.

[0206] In a specific preferred embodiment, a computer program written in Python language is used to complete the specific implementation, and the specific implementation content is as follows:

[0207] First, based on the pre-prepared keyword library (containing 80 keywords such as "road accident", "road waterlogging", "road pothole", "road obstacle", "landslide", etc.), relevant images and videos are scraped from public datasets (BDD100K, Waymo Open Dataset) and Internet resources. A total of about 40,000 images and 1,000 video clips are collected.

[0208] The collected videos are frame-extracted at a frequency of 5 frames per second, and combined with the original image data, after screening and deduplication, about 50,000 image samples are obtained. Then, these samples are preprocessed, including operations such as uniform scaling to 224x224 resolution, standardization, etc. The training set, validation set and test set are divided in the ratio of 8:1:1. The hardware environment here and thereafter is shown in Table 1.

[0209] Table 1 Hardware environment parameters

[0210] Parameter Value CPU Intel(R) Core(TM) i7-13700KF CPU @ 3.40GHz Memory 32 GB Graphics card NVIDIA RTX 4060Ti @ 16 GB OS Windows 11

[0211] The student model adopts the lightweight model architecture designed based on DeepSeek-VL2-tiny described in embodiment four. The training process is implemented based on Python language and deep learning framework (such as PyTorch). The total loss function L total defined in embodiment five is used (which combines the task loss L task for risk classification, the distillation loss L KD for knowledge transfer, and the adaptive quantization loss L quant for model compression).The student model is trained on a server equipped with an NVIDIA RTX 4060Ti GPU using the phase-wise joint optimization training strategy described in Embodiment Six. The training process is conducted for a total of 4000 epochs, using the AdamW optimizer with an initial learning rate of 1e-3 and a cosine annealing learning rate scheduling strategy, and the batch size is set to 64.

[0212] Please refer to the training loss curve shown in Figure 6 , Figure 6 , which shows the changes in the total loss L total during the training process. The curve shows that the total loss steadily decreases with training iterations. After the quantization loss L quant is introduced at the 600th epoch according to the predetermined strategy, the loss curve shows a short and slight fluctuation, after which the model quickly adapts and continues to optimize, eventually converging. This indicates that the joint optimization framework proposed by the present invention can effectively coordinate the three goals of classification task learning, knowledge distillation, and adaptive quantization.

[0213] Please refer to Figure 7 , Figure 7 , which shows examples of risk identification results of some test samples by the model obtained using the method of the present invention. These examples include various risk types under different lighting, weather, and complex backgrounds, such as small road obstacles at a distance, road surface water under night or low light conditions, and slight landslide signs with high integration with the surrounding environment, etc. The results show that the method of the present invention can accurately and effectively identify these risks, demonstrating good robustness and generalization ability for complex scenes.

[0214] Please refer to Figure 8 , Figure 8 , which gives the confusion matrix of the model on the test set. The matrix shows that the model performs relatively evenly on each predefined road risk category, with high values on the diagonal elements, indicating that each risk category is well identified. It is worth noting that even for rare but dangerous risk categories with relatively few samples in the data set, the method maintains a high recognition recall rate, which is crucial for safety warning systems.

[0215] In this embodiment, the bit allocation strategy learned by the adaptive mixed precision quantization mechanism during the training process clearly reflects the intelligent perception and differentiated treatment of the importance of different components of the model. The observed bit allocation patterns are as follows:

[0216] 1. For the shallow convolutional network part responsible for extracting low-level and middle-level visual features of images, most of its layers are automatically allocated lower precision, mainly 4 bits or 8 bits.

[0217] 2. For the Transformer structure in the middle part of the model structure, the attention computation related layers that play a key role are identified and tend to maintain a higher precision (e.g. 16 bits), while other auxiliary layers within the module (such as the MLP layer) are assigned a lower 8-bit precision.

[0218] 3. For the deep network responsible for integrating high-level semantic information and performing final risk classification or detection, as well as the classification / detection head part, the key layers are assigned the highest precision, such as 16 bits or even maintain 32-bit full precision.

[0219] This differentiated and non-uniform bit allocation strategy, which is automatically learned and formed according to the function of the layer and the contribution to the final task performance, verifies the core idea and advantage of the adaptive quantization mechanism in the joint optimization framework of the present application: it can intelligently identify the parts that are crucial to maintaining performance in the model and retain them (assign higher precision), while more aggressively compressing the redundant or secondary parts (assign lower precision). Compared with traditional quantization methods that use fixed bits (such as all INT8) or simple rules (such as by layer type), the adaptive strategy of the present application can maximize model compression rate, improve computational efficiency, while minimizing the damage to model identification accuracy and reliability, thereby achieving a better balance between performance and resource consumption.

[0220] In summary, through detailed experimental settings, process descriptions and result analyses, the present embodiment fully verifies the feasibility, effectiveness and superiority of the "road driving safety real-time warning method based on mixed precision quantization visual large model" proposed by the present application. This method successfully transfers the powerful general visual understanding ability and generalization knowledge of large pre-trained visual models to an extremely lightweight student model, and through innovative adaptive quantization technology, it achieves orders of magnitude compression of model size, computational complexity and memory occupation while maintaining high-level risk identification performance (high accuracy, high mAP, good generalization). This provides a practical and effective technical approach for deploying a high-performance, high-reliability real-time road driving safety warning system on edge devices with limited computing resources.

[0221] Embodiment 2

[0222] Please refer to Figure 9 The present embodiment 2 provides a road safety warning system based on a mixed precision quantization visual large model, comprising:

[0223] A data construction unit for collecting road driving safety related videos and pictures, pre-processing, data enhancement and labeling to form a large-scale diversified data set;

[0224] The model architecture unit is configured to prepare a teacher model and build a knowledge distillation framework, select a pre-trained visual large model as the teacher model, save after on-demand fine-tuning, and extract output layer and intermediate layer knowledge by forward propagation of the teacher model for each input sample during training and evaluate feature importance to weight key features, while selecting a lightweight neural network architecture as a student model, which receives the same input as the teacher model and outputs its own prediction and intermediate feature maps for feature distillation;

[0225] The training optimization unit is configured to input the sampled data into the teacher model to obtain knowledge forward propagation of the student model with mixed precision quantization operation, and build a total loss function L including task loss, knowledge distillation loss, and quantization learning loss total The task loss supervises the risk identification performance of the student model, the knowledge distillation loss transfers the knowledge of the teacher model to the student model to make up for the possible performance loss due to the reduction of the model size, and the quantization learning loss punishes high bit costs to drive an adaptive quantization strategy. Backward propagation is performed to calculate L total the student model weight W S and the quantization strategy parameters, and an optimizer is used to update these parameters.

[0226] The model deployment unit is configured to export a solidified model after training, deploy the exported quantized model to a target edge computing platform, and perform road driving safety warning.

[0227] Embodiment 3

[0228] Embodiment 3 also provides a computer-readable storage medium having a computer program stored thereon, and the computer program can implement any step of the road safety warning method based on the mixed precision quantized visual large model when executed by a processor.

[0229] The computer-readable storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0230] For the computer-readable storage medium provided in the present application, please refer to the above method embodiments, which will not be repeated here.

[0231] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A road safety warning method based on mixed precision quantized visual large model, characterized by ,include: S1. Collect videos and images related to road safety, and form a large-scale and diverse dataset through preprocessing, data enhancement, and annotation. S2. Prepare a teacher model and build a knowledge distillation framework. A pre-trained large visual model is selected as the teacher model, fine-tuned as needed, and saved. During training, each input sample is forward-propagated through the teacher model to extract knowledge from the output and intermediate layers and evaluate feature importance to weight key features. A lightweight neural network architecture is selected as the student model, which receives the same input as the teacher model and outputs its own predictions and intermediate feature maps for feature distillation. S3. Input the sampled data into the teacher model to obtain knowledge Input the student model for forward propagation with mixed precision quantization operation; construct the total loss function L including task loss, knowledge distillation loss, and quantized learning loss total The task loss supervises the risk identification performance of the student model, the knowledge distillation loss transfers the knowledge of the teacher model to the student model to compensate for the performance loss caused by the reduction of the model size, and the quantitative learning loss punishes the high bit cost to drive the adaptive quantization strategy; perform back propagation and calculate L total Weight W for the student model S and quantify strategy parameters and update these parameters using an optimizer; S4. After training is complete, the solidified model is exported and deployed to the target edge computing platform to provide road safety warnings.

2. A road safety warning method based on a mixed precision quantized visual large model according to claim 1, characterized in that ,The specific process of collecting videos and pictures related to road driving ,safety in S1 and forming a large scale and diverse ,dataset through pre-processing, data enhancement and annotation is as follows: Search for videos and images related to road safety based on a pre-prepared search term library; these videos and images contain diverse road scenes, traffic participants, and potential safety risks. Preprocess the searched videos and images, including video frame extraction, image cropping, scaling, grayscale conversion, and normalization. Annotate the pre-processed videos and images, and describe the content of the video or image in one sentence; Divide the annotated video and image datasets into training sets, validation sets, and test sets according to preset ratios; Save the training set, validation set, and test set as different dataset files.

3. A road safety warning method based on a mixed precision quantized visual large model according to claim 1, characterized in that , the output layer and intermediate layer knowledge extracted in S2 are: During the student model training process, each input training sample needs to be forward propagated through the teacher model to extract the knowledge it contains; this knowledge mainly includes two aspects: Output layer knowledge z T : The final prediction result of the teacher model for the input image x; for classification tasks, the performance is the logits vector z output by the classifier layer T ; Intermediate layer feature knowledge The feature vector output by at least one intermediate layer within the teacher model where j is the index of the feature.

4. A road safety warning method based on a mixed precision quantized visual large model according to claim 1, characterized in that ,The method for evaluating the importance of features in S2 is: Gradient-based importance evaluation: Calculate the teacher model's final task loss versus intermediate features The gradient norm of , a feature with a larger gradient means that the feature has a greater impact on the final prediction result; Analysis based on the attention mechanism: When the teacher model is based on the Transformer or contains an attention module, its internal attention weight map is directly used. Regions / feature channels with high attention scores correspond to information that the model considers important. Feature activation-based analysis: The analysis identifies feature maps that exhibit stronger activation responses when input images contain specific hazards / risks. Domain knowledge guidance: Combined with prior knowledge in the field of road safety, determine the feature types that are critical to the warning task.

5. A road safety warning method based on mixed precision quantized visual large model according to claim 1, characterized in that ,The calculation method of the task loss in S3 is: Construct task loss L task : This loss is used to directly supervise the performance of the student model on the final target task, i.e., accurately identifying road safety risks; For the risk classification task, we use the standard cross entropy loss: Where C is the total number of categories, y c is the cth element of label y, p s，c =Softmax(z s ) c is the probability that the student model predicts that the input belongs to category c; Use focal loss to increase attention to difficult samples when the class imbalance problem is serious: Among them, α c is the class weight, γ is the focusing parameter; For risky object detection or segmentation tasks: L task Contains classification loss and regression loss.

6. A road safety warning method based on a mixed precision quantized visual large model according to claim 1, characterized in that ,The calculation method of the knowledge distillation loss in S3 is: Constructing knowledge distillation loss L KD , L KD Include at least one of the following components: Output layer distillation: Let the output probability distribution of the student model mimic the output probability distribution of the teacher model; use KL divergence to measure the difference between the two probability distributions: Among them, p S (T) = Softmax(z S / T) and p T (T) = Softmax(z T / T) is the softened probability distribution calculated using a distillation temperature T>1; a higher temperature T will make the probability distribution smoother and emphasize the information transfer of negative labels; T 2 Factor is used to maintain the gradient magnitude; Intermediate layer feature map distillation: Enforce the intermediate features of the student model Approximate the characteristics corresponding to the teacher model under some metric Among them, j is a traversal of all intermediate layer pairs involved in distillation; and It is an optional feature conversion function, and sometimes the original features can be directly compared; represents the p-th power of the Lp norm; β j is the weight of the j-th pair of feature distillation; The total knowledge distillation loss is the weighted sum of the losses of each distillation method: L KD =a soft L KD,soft +a feature L KD,feature +... Among them, α soft , α feature , ...are the weight coefficients of each distillation method.

7. A road safety warning method based on a mixed precision quantized visual large model according to claim 1, characterized in that ,The design method of quantizing learning loss in S3 is: Constructing quantized learning loss / regularization term L quant Used to drive the model to learn the optimal bit allocation strategy during training {b l }; Define the quantizable layer l in the model and the set of bit options allowed for each quantizable layer B = {b1, b2, ..., b k }; Introducing learnable parameters to control the choice of the number of bits, including: Each bit option b∈B of each layer l is assigned a gating value g l，b , probability p l，b The learnable parameters including ; these parameters are normalized by the Softmax function so that During the forward propagation of training, the number of bits actually used is determined based on the learned probability. l And perform quantization operation Q(x, b l ); and deal with the non-differentiable problems arising from discrete choice through methods including differentiable sampling from categorical distributions and expectation quantization; The quantization function Q(x, b) is constructed by methods including uniform quantization, which linearly maps floating-point numbers to 2 b discrete integer values; Introduce a continuous learnable parameter c for each layer l l , through the mapping function b l =Map(c l ) to obtain the discrete bit number; in order for the gradient to be able to backpropagate, the quantization operation Q itself needs to use a differentiable approximation, which performs the actual quantization operation during forward propagation, but directly passes the gradient to the floating-point value during backpropagation; L quant The goal of is to guide the model to learn the desired quantization strategy; is designed to penalize the model for choosing a higher number of bits.

8. The road safety warning method based on mixed precision quantized visual large model according to claim 1 is characterized in that: The total loss function L in S3 total The calculation method is: Overall loss function L total is the weighted sum of task loss, knowledge distillation loss, and quantized learning loss: Among them, z S is the output of the student model, and y is the true label; is the knowledge extracted from the teacher model; is the intermediate feature of the student model; {b l } represents the learned quantization bit strategy for each layer; λ task ,λ KD ,λ quant It is the weight hyperparameter of each loss, which is used to balance the importance of different optimization objectives.

9. A road safety warning system based on a mixed precision quantized visual large model, characterized by: include: The data construction unit is used to collect videos and images related to road safety, and form a large-scale and diverse dataset through preprocessing, data enhancement and annotation; The model architecture unit is used to prepare the teacher model and build the knowledge distillation framework. A pre-trained large visual model is selected as the teacher model, which is fine-tuned as needed and saved. During training, each input sample is forward-propagated through the teacher model to extract the output layer and intermediate layer knowledge and evaluate the feature importance to weight the key features. At the same time, a lightweight neural network architecture is selected as the student model, which receives the same input as the teacher model and outputs its own predictions and intermediate feature maps for feature distillation. Training optimization unit, used to input sampled data into the teacher model to acquire knowledge Input the student model for forward propagation with mixed precision quantization operation; construct the total loss function L including task loss, knowledge distillation loss, and quantized learning loss total The task loss supervises the risk identification performance of the student model, the knowledge distillation loss transfers the knowledge of the teacher model to the student model to compensate for the performance loss caused by the reduction of the model size, and the quantitative learning loss punishes the high bit cost to drive the adaptive quantization strategy; perform back propagation and calculate L total Weight W for the student model s and quantify strategy parameters and update these parameters using an optimizer; The model deployment unit is used to export the solidified model after training is completed, and deploy the exported quantized model to the target edge computing platform to provide road driving safety warnings.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement a road safety warning method based on a mixed precision quantized visual large model as described in any one of claims 1 to 8.

Images