A lightweight driver dangerous behavior recognition method based on CLIP and knowledge distillation

Abstract

The application discloses a kind of light-weight driver dangerous behavior identification method based on CLIP and knowledge distillation, belong to intelligent traffic and vehicle vision perception field, including: obtaining the driver image collected by vehicle-mounted camera, constructs multi-label dangerous behavior image dataset;Utilize CLIP image encoder to construct teacher model, extract image high-dimensional semantic features and output prediction probability;The student model with light-weight convolutional neural network as backbone, and cascaded with spatial axial attention module and multi-directional local feature fusion module is constructed, and the compact image representation is output;Using feature alignment distillation strategy, the student model representation is mapped to the teacher model feature space by linear transformation, and is discriminated by sharing classification head;Using joint loss function to train student model, obtain the light-weight identification model that can be deployed in vehicle terminal.The application can realize high-precision, high-robust multi-label driving behavior real-time identification under low parameter quantity and low calculation amount.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and vehicle-mounted visual perception technology, and particularly relates to a lightweight driver dangerous behavior recognition method based on CLIP and knowledge distillation. Background Technology

[0002] Currently, driver dangerous behavior recognition, as one of the core technologies of in-vehicle intelligent monitoring systems, has been widely researched and applied in the fields of intelligent transportation and assisted driving. Existing technologies, driver behavior recognition methods based on deep convolutional neural networks (such as ResNet and MobileNet series) or visual Transformers (such as ViT) can automatically detect typical dangerous behaviors such as smoking, using mobile phones, not wearing seat belts, and taking both hands off the steering wheel using images captured by in-vehicle cameras. These methods, by constructing multi-layer nonlinear feature extraction networks, can achieve high recognition accuracy on public datasets or in limited scenarios, effectively improving the automation level of driving safety monitoring. In particular, using pre-trained large models (such as CLIP) for feature extraction can enhance semantic understanding of complex cockpit environments by leveraging their cross-modal general visual knowledge. Furthermore, the introduction of lightweight network structures (such as MobileNet) allows some methods to maintain basic real-time inference capabilities while reducing the number of model parameters and floating-point operations, providing a feasible path for the initial deployment of in-vehicle edge devices.

[0003] However, existing methods for identifying dangerous driver behaviors still have the following shortcomings in practical in-vehicle applications: First, cockpit images are affected by factors such as differences in camera installation angles, drastic changes in lighting, driver posture occlusion, and diverse vehicle models. The feature robustness of traditional convolutional networks or lightweight models is limited, leading to a significant decrease in recognition performance under different viewing angles and operating conditions. Second, large-scale visual models (such as the original CLIP image encoder) used to obtain high semantic representation capabilities have a large number of parameters and high computational complexity, making them difficult to deploy directly on resource-constrained in-vehicle terminals or embedded devices. Third, while existing lightweight models (such as MobileNet) have low computational costs, their feature representation capabilities are insufficient, making it difficult to accurately capture subtle behaviors (such as holding a cigarette or the edge of a seatbelt) and global spatial dependencies, thus making it difficult to achieve an ideal balance between recognition accuracy and deployment efficiency. Therefore, there is an urgent need for a driver dangerous behavior recognition scheme that can combine strong semantic inheritance capabilities and high generalization performance under conditions of low parameter count and low computational overhead. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a lightweight driver dangerous behavior identification method based on CLIP and knowledge distillation, thereby resolving the issues present in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a lightweight driver hazardous behavior identification method based on CLIP and knowledge distillation, comprising: We acquire driver images captured by vehicle-mounted cameras and construct a preprocessed multi-label dangerous behavior image dataset. A teacher model is constructed using the CLIP image encoder. High-dimensional semantic features of the image are extracted through the teacher model, and the predicted probability of each dangerous behavior category is output. A student model is constructed with a lightweight convolutional neural network as the backbone and cascaded with a spatial axial attention module and a multi-directional local feature fusion module to output compact image representations. A feature alignment distillation strategy is adopted to map the representation features of the student model to the feature space of the teacher model through a linear transformation, and then the two are uniformly distinguished by sharing a classification head; The student model is trained using a joint loss function, which includes classification loss, feature numerical distribution constraint loss, and feature semantic direction constraint loss. After training, a lightweight recognition model that can be deployed on an in-vehicle terminal is obtained. The lightweight recognition model outputs the category of dangerous behavior and its confidence level for real-time input single-frame driver images.

[0006] Preferably, the process of constructing a preprocessed multi-labeled hazardous behavior image dataset includes: extracting driver image frames from in-vehicle camera videos of operating vehicles, labeling at least one of the behaviors of smoking, playing with a mobile phone, not wearing a seat belt, and taking both hands off the steering wheel with multiple labels, removing samples with severe occlusion, blurred images, or difficult-to-determine categories, unifying image resolution, and performing at least one of the data augmentation operations of random flipping, brightness perturbation, and random rotation.

[0007] Preferably, the CLIP image encoder of the teacher model adopts the Vision Transformer architecture, and its underlying network parameters are frozen during training, with only the high-level semantically related layers being fine-tuned; the output of the teacher model is connected to a two-layer fully connected classification head, and the Sigmoid activation function is used to output multi-label independent probabilities.

[0008] Preferably, the classification loss used during the training of the teacher model is Sigmoid Focal Loss, which is used to alleviate the class imbalance problem of multi-label samples.

[0009] Preferably, in the student model: The lightweight convolutional neural network is MobileNetV4, used to extract basic visual features of the input image; The spatial axial attention module receives the basic visual features sequentially and performs axial depth convolution and attention weighting in the horizontal and vertical directions respectively, outputting features with global spatial dependency enhancement. The multi-directional local feature fusion module is used to receive the features processed by the spatial axial attention module, extract multi-directional local structural features through parallel or serial combination of main diagonal convolution, anti-diagonal convolution and axial convolution, and then perform residual fusion of the extracted features with the basic visual features and output a compact image representation through global pooling.

[0010] Preferably, the spatial axial attention module performs the following operations on the input features: pooling and concatenating the feature maps along the horizontal and vertical directions respectively; performing orientation-aware modeling on the concatenated features using axial depth convolutions in the channel direction, horizontal direction, and vertical direction; generating spatial attention weights through pointwise convolution and broadcasting mechanisms, and then multiplying the attention weights element-wise with the input features and superimposing residual connections.

[0011] Preferably, in the multi-directional local feature fusion module, the convolution kernels of the main diagonal convolution and the anti-diagonal convolution slide along the main diagonal direction and the anti-diagonal direction, respectively, to extract directionally sensitive local edge and texture features, and then superimpose these features with the output of the axial depth convolution.

[0012] Preferably, the feature alignment distillation strategy includes: extracting the feature vectors of the teacher model and the student model before their respective classifiers; setting a linear projection layer for the student model to map its feature vectors to the same dimension as the teacher model's feature vectors; and the shared classifier simultaneously receiving the teacher model's feature vectors and the mapped student model's feature vectors, and outputting their respective classification prediction results.

[0013] Preferably, in the joint loss function: the classification loss is a Sigmoid Focal Loss, which is applied to the output of the shared classification head; the feature numerical distribution constraint loss is a Smooth L1 loss, which is used to constrain the dimension-wise numerical differences between the mapped student model features and teacher model features; and the feature semantic direction constraint loss is a cosine embedding loss, which is used to constrain the angle between the two feature vectors to be close to zero.

[0014] In a second aspect, the present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention constructs a student model with a lightweight convolutional neural network as its backbone, cascaded with a spatial axial attention module and a multi-directional local feature fusion module. This student model outputs a compact image representation, effectively enhancing the ability to express global spatial dependencies and fine-grained local structural features in cockpit images while keeping the number of parameters and computational costs within acceptable limits for in-vehicle terminals. Simultaneously, the teacher model constructed using the CLIP image encoder possesses powerful high-dimensional semantic extraction capabilities. A feature alignment distillation strategy transfers the semantic knowledge of the teacher model to the student model, enabling the student model to inherit high-level discriminative features even under lightweight conditions. The synergistic effect of these technical features solves the problem of low recognition accuracy caused by insufficient feature representation capabilities in lightweight models, achieving a balance between small model size and high recognition accuracy.

[0016] The student model of this invention is cascaded with a spatial axial attention module and a multi-directional local feature fusion module. The former enhances the adaptability to interference such as viewpoint changes and occlusion through spatial attention modeling in the horizontal and vertical directions, while the latter enhances the sensitivity to key areas such as the driver's hands, cigarettes, and seat belt edges through multi-directional local feature extraction. Furthermore, by employing feature numerical distribution constraint loss and feature semantic direction constraint loss in the joint loss function, the student model is forced to approximate the numerical distribution and semantic direction of the teacher model at the feature level, thereby inheriting the teacher model's robustness to lighting changes, viewpoint differences, and occlusion scenarios. The combined effect of these features ensures that the finally deployed lightweight recognition model maintains stable recognition performance under different vehicle camera viewpoints and complex lighting conditions.

[0017] The student model of this invention uses a lightweight convolutional neural network as its backbone to output compact image representations. The resulting lightweight recognition model can be deployed on in-vehicle terminals, directly outputting recognition results from real-time input single-frame driver images. This solution does not rely on multi-sensor fusion or video stream compression preprocessing, and features small parameter count, low computational cost, and short inference latency. It can meet the engineering requirements of real-time driving behavior monitoring on resource-constrained in-vehicle edge devices.

[0018] This invention utilizes linear transformation mapping and shared classification heads in the feature alignment distillation strategy, along with the role of feature semantic direction constraint loss (such as cosine embedding loss) in the joint loss function. This enables the student model to not only mimic the feature numerical distribution of the teacher model during the learning process but also to be forced to align with its semantic direction. This deep semantic alignment mechanism allows the student model to inherit the general visual knowledge acquired by the teacher model during cross-modal pre-training, thus maintaining good generalization recognition ability when faced with new cockpit perspectives or vehicle models that were not included in the training.

[0019] This invention constructs a preprocessed multi-label hazardous behavior image dataset and trains it using a joint loss function that includes classification loss. This enables the lightweight recognition model to simultaneously output the categories and confidence levels of multiple hazardous behaviors, such as smoking, using a mobile phone, not wearing a seatbelt, and taking both hands off the steering wheel, adapting to the situation of multiple hazardous behaviors occurring concurrently in real driving scenarios. Compared to existing single-label classification or methods that only identify a single behavior, this invention has stronger practical application value. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a lightweight driver dangerous behavior identification method based on CLIP and knowledge distillation according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the construction and preprocessing of driver dangerous behavior image data according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the teacher model structure according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the overall structure of the student model according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the AxialDualLite module structure according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the directional mixing module structure according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the teacher-student feature distillation and shared classification head structure according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the lightweight model deployment process according to an embodiment of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0023] Example 1 like Figure 1As shown, this embodiment provides a lightweight driver hazardous behavior identification method based on CLIP and knowledge distillation, including: Step 1: Acquire driver images captured by vehicle-mounted cameras and construct a preprocessed multi-label hazardous behavior image dataset; Furthermore, the process of constructing a preprocessed multi-labeled hazardous behavior image dataset includes: extracting driver image frames from in-vehicle camera videos of commercial vehicles, labeling at least one of the behaviors of smoking, using a mobile phone, not wearing a seat belt, and taking both hands off the steering wheel with multiple labels, removing samples with severe occlusion, blurred images, or difficult-to-determine categories, unifying image resolution, and performing at least one of the data augmentation operations of random flipping, brightness perturbation, and random rotation.

[0024] Specifically, real-world driving scene image data is collected using in-vehicle cameras in commercial vehicles. Keyframe images are extracted from the raw videos to construct a dataset of driver dangerous behaviors. The collected images are manually labeled, with categories including smoking, using a mobile phone, not wearing a seatbelt, and taking both hands off the steering wheel. After labeling, severely obstructed, low-resolution, and difficult-to-determine behavior samples are removed, and the images are uniformly scaled to a resolution of 224×224.

[0025] To improve the model's generalization ability, data augmentation operations such as random flipping, brightness adjustment, and random rotation are performed on the images during the training phase. Depending on training requirements, the dataset can be divided into training, validation, and test sets, and an additional cross-viewpoint test set can be constructed to verify the model's adaptability in new viewpoint scenes.

[0026] Figure 2 This diagram illustrates the process of constructing and preprocessing image data of dangerous driver behaviors. It shows the data acquisition, dangerous behavior category labeling, sample selection, image size standardization, data augmentation, dataset partitioning, and cross-view test set construction, which are used to form the image data of dangerous driver behaviors required for model training and evaluation.

[0027] Step 2: Construct a teacher model using the CLIP image encoder, extract high-dimensional semantic features of the image through the teacher model, and output the predicted probability of each dangerous behavior category; Furthermore, the CLIP image encoder of the teacher model adopts the Vision Transformer architecture, freezing its underlying network parameters during training and only fine-tuning the high-level semantically related layers; the output of the teacher model is connected to a two-layer fully connected classification head, and uses the Sigmoid activation function to output multi-label independent probabilities.

[0028] Furthermore, the classification loss used during the training of the teacher model is Sigmoid Focal Loss, which is used to alleviate the class imbalance problem in multi-label samples.

[0029] Specifically, the image encoder in the CLIP model is used as the feature extraction network of the teacher model, and the image encoder is a Vision Transformer architecture; the low-level parameters of the image encoder are frozen, and only the high-level semantic feature related layers are fine-tuned; a two-layer fully connected linear classification head is connected to the output of the image encoder, and the linear classification head is used to map the image features to a multi-label behavior category space; the probability of each dangerous behavior category existing independently is obtained through the Sigmoid function.

[0030] Figure 3 This diagram illustrates the structure of the teacher model, which uses the CLIP image encoder as the teacher model. The model includes components such as the input image, ViT feature extraction, parameter freezing and fine-tuning strategy, classification head, and multi-label prediction output.

[0031] In this embodiment, the image encoder preferably adopts the Vision Transformer ViT-B / 32 structure. The input image is first processed by patching and linear projection, and then fed into a multi-layer Transformer encoder to obtain global semantic features.

[0032] The teacher model features corresponding to input image I can be represented as: (1) In the formula, I represents the input driver image. The CLIP image encoder representing the teacher model. This indicates the output feature vector.

[0033] To avoid disrupting the existing general visual knowledge of the pre-trained model, the low-level parameters are frozen during training, and only the high-level semantically relevant layers are fine-tuned. The teacher model's output features are mapped to the dangerous behavior category space via a two-layer fully connected classification head, and the independent probabilities of each category are output through a Sigmoid function. Sigmoid Focal Loss is used during training to alleviate the problem of imbalanced distribution of multi-label samples.

[0034] The corresponding multi-label prediction process can be represented as: (2) In the formula, , , and Indicates the classification header parameters, This represents the activation function. This represents the probability vector of dangerous behaviors output by the teacher model.

[0035] In this embodiment, the teacher model is trained using Sigmoid Focal Loss as the multi-label classification loss function to reduce the impact of class imbalance on model training and improve the ability to identify minority class dangerous behavior samples.

[0036] Step 3: Construct a student model with a lightweight convolutional neural network as the backbone and cascaded with a spatial axis attention module and a multi-directional local feature fusion module to output compact image representations; Furthermore, in the student model: the lightweight convolutional neural network is MobileNetV4, used to extract basic visual features of the input image; the spatial axial attention module receives the basic visual features sequentially, and performs axial depth convolution and attention weighting in the horizontal and vertical directions respectively, outputting features with global spatial dependency enhancement; the multi-directional local feature fusion module receives the features processed by the spatial axial attention module, extracts multi-directional local structural features through parallel or serial combination of main diagonal convolution, anti-diagonal convolution and axial convolution, and performs residual fusion with the extracted features and the basic visual features, and then outputs a compact image representation through global pooling.

[0037] Furthermore, the spatial axial attention module performs the following operations on the input features: pooling and concatenating the feature maps along the horizontal and vertical directions respectively; performing orientation-aware modeling on the concatenated features using axial depth convolutions in the channel direction, horizontal direction, and vertical direction; generating spatial attention weights through pointwise convolution and broadcasting mechanisms, and then multiplying the attention weights element-wise with the input features and superimposing residual connections.

[0038] Furthermore, in the multi-directional local feature fusion module, the convolution kernels of the main diagonal convolution and the anti-diagonal convolution slide along the main diagonal direction and the anti-diagonal direction, respectively, to extract directionally sensitive local edge and texture features, and then superimpose these features with the output of the axial depth convolution.

[0039] Specifically, the student model uses MobileNetV4 as its backbone network to extract basic visual features from driver images. To enhance the lightweight model's ability to express spatial structural information and fine-grained edge features, an AxialDualLite module and a directional fusion module are cascaded at the output of the MobileNetV4 backbone network. The basic visual features are input into the AxialDualLite module, where axial depth convolution and directional fusion are performed in the horizontal and vertical directions to enhance the ability to model long-range spatial dependencies. The features processed by the AxialDualLite module are input into the directional fusion module, which combines diagonal and axial convolutions to extract multi-directional local structural features in the diagonal, horizontal, and vertical directions. The enhanced features output from the directional fusion module are then residually fused with the output of the MobileNetV4 backbone network and fed into a global pooling layer to obtain the student model's feature vector.

[0040] Figure 4 The diagram shows the overall structure of the student model, which consists of a MobileNetV4 backbone network, an AxialDualLite attention module, a directional mixing module, residual fusion, and a global pooling layer. It is used to extract dangerous driver behavior features under lightweight conditions.

[0041] In this embodiment, the AxialDualLite module includes a horizontal axis branch and a vertical axis branch. The input features are concatenated after average pooling and max pooling, then directional modeling is performed using axial depth convolution, and spatial attention weights are generated through pointwise convolution and broadcast fusion. Finally, residual fusion is performed with the original features.

[0042] Figure 5 This is a schematic diagram of the AxialDualLite module structure. The diagram illustrates the dual-axis modeling process of the AxialDualLite module, including channel pooling, horizontal and vertical axis depth convolution, pointwise convolution, spatial attention weight generation, and residual fusion with input features.

[0043] The channel pooling result of the input feature X can be expressed as: (3) The axial feature extraction result can be expressed as: (4) Spatial attention weights can be expressed as: (5) The module output can be represented as: (6) In the formula, This represents depthwise convolution in the channel direction. This represents horizontal axial depth convolution. This represents vertical axial depth convolution. The symbol represents the Sigmoid function, ⊙ represents element-wise multiplication, and X represents the input feature. This represents the descriptor after channel pooling. This represents the axial feature extraction result, where A represents the spatial attention weight. This indicates the output characteristics of the AxialDualLite module.

[0044] In this embodiment, the directional mixing module constructs multi-directional feature extraction paths through main diagonal convolution, anti-diagonal convolution, and axial convolution to enhance the response to local targets such as mobile phones, cigarettes, seat belt boundaries, and hand contours. The features processed by the directional mixing module are residually superimposed with the backbone network output, and global pooling is used to obtain the student model feature vector.

[0045] Figure 6 This is a schematic diagram of the directional blending module structure. The diagram shows that the directional blending module extracts multi-directional local structural features through main diagonal convolution, anti-diagonal convolution, and axial depth convolution branches, and then fuses the results of each branch for output.

[0046] The local orientation modeling process of the orientation blending module can be represented as: (7) (8) (9) In the formula, This represents the main diagonal convolution kernel. Indicates the anti-angle convolution kernel, This indicates an axial depthwise convolution operation. This represents the characteristic response in the main diagonal direction. Indicates the characteristic response in the anti-angle direction. This indicates the directional mixing characteristics after fusion.

[0047] Step 4: Using a feature alignment distillation strategy, the representation features of the student model are mapped to the feature space of the teacher model through a linear transformation, and the two are uniformly distinguished by sharing a classification head; Furthermore, the feature alignment distillation strategy includes: extracting the feature vectors of the teacher model and the student model before their respective classifiers; setting a linear projection layer for the student model to map its feature vectors to the same dimension as the teacher model's feature vectors; and the shared classifier simultaneously receiving the teacher model's feature vectors and the mapped student model's feature vectors, and outputting their respective classification prediction results.

[0048] Specifically, after training the teacher model, the teacher model parameters are fixed, and the teacher model is used to perform knowledge distillation on the student model. The output features of the teacher model are denoted as... The student model output features are denoted as To address the inconsistency in feature dimensions between the teacher and student models, a linear projection layer is added to the feature output of the student model. Mapping to A consistent feature space.

[0049] This mapping process can be represented as: (10) In the formula, and These represent the weight parameters and bias parameters of the linear projection layer, respectively. This represents the features of the mapped student model.

[0050] To improve distillation consistency, the teacher and student models share the same linear classifier head. This shared classifier head maps the teacher model features and the projected student model features to a unified discriminant space, allowing the student model to focus on learning semantic representations consistent with the teacher model, rather than learning new classification boundaries independently.

[0051] Figure 7 This diagram illustrates the structure of teacher-student feature distillation and shared classification head. It shows the structural relationship where teacher model features and student model features are distilled in the same feature space after linear projection, and category prediction is completed through a shared classification head.

[0052] Step 5: Train the student model using a joint loss function, which includes classification loss, feature numerical distribution constraint loss, and feature semantic direction constraint loss. After training, a lightweight recognition model that can be deployed on an in-vehicle terminal is obtained. The lightweight recognition model outputs the category of dangerous behavior and its confidence level for real-time input single-frame driver images.

[0053] Furthermore, in the joint loss function: the classification loss is Sigmoid Focal Loss, which is applied to the output of the shared classification head; the feature numerical distribution constraint loss is Smooth L1 loss, which is used to constrain the dimension-wise numerical differences between the mapped student model features and teacher model features; and the feature semantic direction constraint loss is cosine embedding loss, which is used to constrain the angle between the two feature vectors to be close to zero.

[0054] Specifically, a joint loss function is used during student model training: (11) in, Sigmoid Focal Loss for multi-label dangerous behavior recognition tasks; The Smooth L1 loss between student model features and teacher model features is used to constrain the consistency of numerical distribution. The cosine embedding loss is used to constrain semantic direction consistency between student model features and teacher model features; λ1 and λ2 are empirically set loss weight coefficients.

[0055] The characteristic distillation loss and directional consistency loss can be further expressed as: (12) (13) In the formula, · This represents the inner product of two eigenvectors. This represents the L2 norm. By simultaneously constraining the consistency of feature numerical distribution and semantic direction, the student model can more stably inherit the high-level semantic expressive power of the teacher model under lightweight conditions.

[0056] As a specific implementation, the AdamW optimizer can be used to train the student model. After training, the student model is exported in a model format suitable for edge deployment and deployed to in-vehicle terminals or mobile AI inference platforms. During actual inference, a single frame of driver image is input, the model outputs the probability value of each dangerous behavior category, and determines whether the driver has engaged in the corresponding dangerous behavior based on a threshold.

[0057] Figure 8 This diagram illustrates the deployment process of a lightweight student model, including model training completion, format export, deployment to an in-vehicle or mobile inference platform, receiving single-frame image input, and outputting dangerous behavior categories and confidence levels.

[0058] In one exemplary embodiment, the number of parameters of the obtained student model is about 2.1M and the number of floating-point operations is about 0.2G. While maintaining low computational overhead, it achieves higher recognition accuracy compared with the original lightweight backbone network, and shows good generalization ability in multi-view scenarios, making it suitable for real-time deployment on vehicle-mounted devices.

[0059] Example 2 This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0060] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A lightweight driver hazardous behavior identification method based on CLIP and knowledge distillation, characterized in that, Includes the following steps: We acquire driver images captured by vehicle-mounted cameras and construct a preprocessed multi-label dangerous behavior image dataset. A teacher model is constructed using the CLIP image encoder. High-dimensional semantic features of the image are extracted through the teacher model, and the predicted probability of each dangerous behavior category is output. A student model is constructed with a lightweight convolutional neural network as the backbone and cascaded with a spatial axial attention module and a multi-directional local feature fusion module to output compact image representations. A feature alignment distillation strategy is adopted to map the representation features of the student model to the feature space of the teacher model through a linear transformation, and then the two are uniformly distinguished by sharing a classification head; The student model is trained using a joint loss function, which includes classification loss, feature numerical distribution constraint loss, and feature semantic direction constraint loss. After training, a lightweight recognition model that can be deployed on an in-vehicle terminal is obtained. The lightweight recognition model outputs the category of dangerous behavior and its confidence level for real-time input single-frame driver images.

2. The method according to claim 1, characterized in that, The process of constructing a preprocessed multi-labeled hazardous behavior image dataset includes: extracting driver image frames from in-vehicle camera videos of commercial vehicles, labeling at least one of the behaviors of smoking, using a mobile phone, not wearing a seat belt, and taking both hands off the steering wheel with multiple labels, removing samples with severe occlusion, blurred images, or difficult-to-determine categories, unifying image resolution, and performing at least one of the data augmentation operations of random flipping, brightness perturbation, and random rotation.

3. The method according to claim 1, characterized in that, The CLIP image encoder of the teacher model adopts the Vision Transformer architecture, which freezes its underlying network parameters during training and only fine-tunes the high-level semantically related layers; the output of the teacher model is connected to a two-layer fully connected classification head, and uses the Sigmoid activation function to output multi-label independent probabilities.

4. The method according to claim 3, characterized in that, The classification loss used during the training of the teacher model is Sigmoid Focal Loss, which is used to alleviate the class imbalance problem in multi-label samples.

5. The method according to claim 1, characterized in that, In the student model: The lightweight convolutional neural network is MobileNetV4, used to extract basic visual features of the input image; The spatial axial attention module receives the basic visual features sequentially and performs axial depth convolution and attention weighting in the horizontal and vertical directions respectively, outputting features with global spatial dependency enhancement. The multi-directional local feature fusion module is used to receive the features processed by the spatial axial attention module, extract multi-directional local structural features through parallel or serial combination of main diagonal convolution, anti-diagonal convolution and axial convolution, and then perform residual fusion of the extracted features with the basic visual features and output a compact image representation through global pooling.

6. The method according to claim 5, characterized in that, The spatial axial attention module performs the following operations on the input features: pooling and concatenating the feature maps along the horizontal and vertical directions respectively; performing orientation-aware modeling on the concatenated features using axial depth convolutions in the channel direction, horizontal direction, and vertical direction; generating spatial attention weights through pointwise convolution and broadcasting mechanisms, and then multiplying the attention weights element-wise with the input features and superimposing residual connections.

7. The method according to claim 5, characterized in that, In the multi-directional local feature fusion module, the convolution kernels of the main diagonal convolution and the anti-diagonal convolution slide along the main diagonal direction and the anti-diagonal direction, respectively, to extract directionally sensitive local edge and texture features, and then superimpose these features with the output of the axial depth convolution.

8. The method according to claim 1, characterized in that, The feature alignment distillation strategy includes: extracting the feature vectors of the teacher model and the student model before their respective classifiers; setting a linear projection layer for the student model to map its feature vectors to the same dimension as the feature vectors of the teacher model; and the shared classifier simultaneously receiving the feature vectors of the teacher model and the mapped feature vectors of the student model, and outputting their respective classification prediction results.

9. The method according to claim 1, characterized in that, In the joint loss function: the classification loss is Sigmoid Focal Loss, which is applied to the output of the shared classification head; the feature numerical distribution constraint loss is Smooth L1 loss, which is used to constrain the dimension-wise numerical differences between the mapped student model features and teacher model features; and the feature semantic direction constraint loss is cosine embedding loss, which is used to constrain the angle between the two feature vectors to be close to zero.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-9.

Images