A method for recognizing distracted driving behavior based on contrastive learning and real-time edge computing

Abstract

This invention discloses a method for recognizing distracted driving behavior based on contrastive learning and real-time edge computing, relating to the field of intelligent traffic monitoring technology. The method includes: acquiring a target dataset; constructing a YOLOv5 detection model and pre-training it; converting the pre-trained YOLOv5 detection model into a YOLOv5 classification model and retraining it on the target dataset to obtain a trained PT model; improving the PT model based on contrastive learning; pruning the improved PT model using Batch Normalization (BN) layers, followed by quantization; converting the PT model to an ONNX model and then to an RKNN model and pre-compiling it; finally, using the pre-compiled RKNN model for real-time inference on an edge computing platform. This invention utilizes contrastive learning to enable the model to learn a feature extractor with representational capabilities, effectively improving the accuracy of driver behavior classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent traffic monitoring, more particularly, to a driver distraction driving behavior recognition method based on contrast learning and real-time edge computing. BACKGROUND

[0002] With the rapid development of the transportation industry, the number of motor vehicles and motor vehicle drivers on the road is rapidly increasing. The progress of transportation tools brings convenience to people, but also makes traffic accidents increasingly common, which poses a serious threat to people's life and property safety. Among them, the driver's non-wearing of safety belts, making and receiving calls and other illegal driving behaviors are one of the main reasons for traffic accidents. In this context, in the face of the increasingly severe traffic safety situation, the development of intelligent traffic monitoring systems has attracted the attention of researchers.

[0003] The technical solution in the prior art is as follows: the ResNet50 network structure is improved and used for driver behavior detection, the second layer convolution of the convolution residual module is replaced by a deep separable convolution (DSCB), the first layer and the second layer convolution in the identity residual module are replaced by a deep separable convolution, the third layer is replaced by a grouped convolution (DSIB), and the number of identity residual modules is reduced, the number of neurons in the last fully connected layer is reduced to realize lightweight, and the parameter quantity is reduced. At the same time, channel attention mechanism and spatial attention mechanism modules are embedded between each convolution residual module and identity residual module to improve the detection rate.

[0004] In the step of modifying the number of residual modules, the number of residual blocks in the second stage, the third stage, the fourth stage and the fifth stage of ResNet50 is modified from 3, 4, 6 and 3 to 3, 3, 4 and 3, and the last fully connected layer is removed, and the number of neurons in the fully connected layer is reset to 256; when adding the attention mechanism module, a total of 13 channel attention mechanism modules and spatial attention mechanism modules are added to pay attention to the importance of different feature channels and different positions of the image.

[0005] This method only proposes a model improvement idea and can only detect two types of behaviors: making a phone call and not wearing a safety belt. This method does not truly propose a feasible method that can be implemented, and does not provide a method for deploying the model on an edge computing platform. At the same time, this method does not consider the inference library operator support situation after the model is deployed to the development board and the influence of adding attention mechanism on the computational complexity.

[0006] Therefore, to solve the difficulties existing in the prior art, a driver distraction driving behavior recognition method based on contrast learning and real-time edge computing is proposed, which is a problem that needs to be solved by those skilled in the art. SUMMARY

[0007] In view of this, the present invention provides a method for recognizing distracted driving behavior of drivers based on contrastive learning and real-time edge computing, in order to solve the technical problems existing in the prior art.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for recognizing distracted driving behavior based on contrastive learning and real-time edge computing includes the following steps:

[0010] Obtain the target dataset;

[0011] Construct a Yolov5 detection model and pre-train it.

[0012] The pre-trained Yolov5 detection model is converted into a Yolov5 classification model and retrained on the target dataset to obtain the trained PT model.

[0013] The PT model is improved based on the contrastive learning method;

[0014] The improved PT model is pruned using BN layers, and then the PT model is quantized.

[0015] The PT model is first converted to the ONNX model and then to the RKNN model, and then pre-compiled.

[0016] Finally, a pre-compiled RKNN model was used for real-time inference on an edge computing platform.

[0017] Optionally, the edge computing platform used in the above method is the RV1126 edge computing platform.

[0018] Optionally, the pre-training of the Yolov5 detection model using the above method includes the following:

[0019] The YOLOv5 detection model is used for behavior classification based on driver behavior images. The backbone of the YOLOv5 detection model is used for image feature extraction and classification. The neck and head structures are removed and the last SPFF layer of the YOLOv5 detection model is changed to a classification layer.

[0020] The target dataset is divided into training, validation, and test sets, ensuring that the training, validation, and test sets are independent and randomized. Within each subset, the images are stored in folders according to their categories.

[0021] Optionally, the specific improvements to the PT model based on contrastive learning methods described above include:

[0022] A simulated open set design was implemented, using the target dataset and training with different numbers of distracted driving behaviors in the training set. During recognition, all distracted driving behavior categories were used to obtain categories of distracted driving behaviors not seen in the test set.

[0023] The closed-set classification task is used as the main task to form the classification loss constraint, and a discriminative learning task is added as the auxiliary task to form the contrast loss constraint. The main task and the auxiliary task share a backbone network and use the same coding layer to convert the image into a feature map.

[0024] The model input size is 480×480, the batch size is 256, the SGD optimizer is used, the learning rate is 1e×10^-4, and the temperature coefficient of contrastive loss is 0.25.

[0025] The task is divided into two categories: normal driving and abnormal driving. AUC, accuracy, and F1 score are used as evaluation metrics. The baseline model is Yolov5s-CLS, which only uses the cross-entropy loss function, while SCL refers to the improved version that uses contrastive learning loss.

[0026] Optionally, the classification loss constraint described above can use the cross-entropy loss function:

[0027]

[0028] Where M is the number of categories, N is the batch size, and L is the number of categories. i y is the negative of the sum of the product of the sign function and the logarithm of the probability. ic p is a symbolic function ic Let i be the predicted probability that observed sample i belongs to category c;

[0029] The contrastive loss constraint uses the contrastive loss function, which incorporates sample label information during training. It utilizes supervised learning, where positive samples include those with the same label, and negative samples include those with different labels. Supervised contrastive loss aligns the embedding representation obtained by the feature extraction network with the labels, as shown in the formula below:

[0030]

[0031] Where N is the batch size, L i for P(i) represents the relationship between x and x i For samples with the same label, · represents the inner product. To normalize the feature vectors of positive samples, To normalize the feature vectors of samples with labels different from the positive samples, Normalize the feature vectors of the augmented version of the positive samples, where τ is the contrastive learning temperature parameter, and x k For xi' Different augmented versions, x i' For x i An expanded version.

[0032] Optionally, the improved PT model can be pruned using BN layers as described above.

[0033] First, initialize the network, identify and label the channels with the smallest scaling factor, and apply L1 regularization to each scaling factor in the channel for adjustment. Each scaling factor corresponds to a specific convolutional channel. The mathematical expression for channel pruning is:

[0034]

[0035] Where W is the trainable weight, l is the normal training loss of CNN, f is the training output, x is the training input, y is the training target, λ is the hyperparameter, and g(γ) is the sparsity penalty for the scaling factor, which is chosen here as |γ|, i.e., the L1 norm.

[0036] Perform the following operations during the pruning of the BN layer:

[0037]

[0038] in, For the standardized input, z in For the input of the BN layer, μ β The average value of the input activation. z is the standard deviation of the input activation, ∈ is the bias, and z out γ is the output of the BN layer, β is the affine transformation ratio, and β is the affine transformation bias.

[0039] The coefficient of the hyperparameter λ gradually decreases as the number of training iterations increases. When performing sparse training, the coefficient is set to 0.001, the batch size is set to 128, and the training runs for 50 epochs.

[0040] Prune the PT model according to the scaling factor, and prune the BN layer with the smallest scaling factor.

[0041] The pruned PT model was fine-tuned and trained using the original dataset.

[0042] Optionally, the quantization process of the PT model after pruning, as described above, includes the following:

[0043] After pruning, the PT model is linearly quantized. The formula for linear quantization is as follows:

[0044] r = S(qZ)

[0045]

[0046] Where r is a floating-point real number, q is a fixed-point integer, S is the scaling factor, representing the proportional relationship between the real number and the integer, and Z is the zero point, representing the integer corresponding to 0 of the floating-point real number after mapping, calculated as follows:

[0047]

[0048] Where, r max r min These are the maximum and minimum values ​​of the floating-point real number r and q, respectively. max q min These are the maximum and minimum values ​​of the fixed-point integer q, respectively;

[0049] The experiment compares the accuracy of four different precision parameters: original precision, floating-point half precision, INT8 precision, and UINT8 precision, and analyzes the accuracy loss of different classification models under the four quantization precisions.

[0050] The above method, optionally, involves converting the PT model to an ONNX model and then to an RKNN model, followed by pre-compilation.

[0051] The trained model is saved as a pt model with a network structure, and then converted into an ONNX model for inference using the ONNX library;

[0052] Transform the ONNX model into an RKNN model capable of inference on an edge computing platform;

[0053] Create a dataset for model quantization, and randomly select training set images for fine-tuning the model;

[0054] Finally, the model was converted from the original FP32 precision model to a quantized RKNN model with UINT8 precision;

[0055] By pre-compiling the RKNN model using RKtoolkit, the speed of model loading and switching is improved, thus enhancing real-time performance.

[0056] The above method, optionally, involves using a pre-compiled RKNN model for real-time inference on an edge computing platform, specifically as follows:

[0057] Connect the IMX335 camera to the MIPI-CSI interface, modify the development board's device tree, and enable the node for image stream transmission to transmit image data.

[0058] The image is optimized according to IQ file, ISP, and ISPP, and then sent to the VI node;

[0059] The image is sent to the RGA module in two separate links for format conversion, rotation, and cropping. In the first link, the image is converted from NV12 format to RGB888 format and cropped to 480*480 size. In the second link, the image is converted from NV12 format to RGB888 format, rotated 270°, and cropped to 1280*720 size.

[0060] The inference results are printed to the image frames after RGA cropping using OpenCV, and the corresponding driving behavior is printed according to the sequence number of the inference results.

[0061] The image frame containing the inference results is sent to the display screen of the MIPI-DSI interface for result display.

[0062] The above methods may include, optionally, driving behaviors such as: normal driving, typing with the left hand, typing with the right hand, making a phone call with the left hand, making a phone call with the right hand, operating the center console, drinking water, retrieving items, applying makeup, and talking to passengers.

[0063] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method for recognizing distracted driving behavior of drivers based on contrastive learning and real-time edge computing, the beneficial effects of which are:

[0064] Using contrastive learning enables the model to learn feature extractors with good representational capabilities, thereby effectively improving the accuracy of driver behavior classification and providing greater robustness.

[0065] By fine-tuning the training, the model weights are adapted to the new network model, thereby restoring the original accuracy to some extent.

[0066] The YOLOv5-CLS series of models have a simple structure, without too many branches and with a shallow number of network layers. Therefore, the model accuracy changes little after quantization, which proves the correctness of using the YOLOv5 model for deployment. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0068] Figure 1 A flowchart illustrating a driver distraction behavior recognition method based on contrastive learning and real-time edge computing provided by this invention;

[0069] Figure 2 This is a schematic diagram illustrating the modification of the YOLOv5 detection model provided by the present invention;

[0070] Figure 3 A structural diagram of the comparative learning method provided by this invention;

[0071] Figure 4 This is a structural diagram of the YOLOv5-CLS model provided by the present invention;

[0072] Figure 5 This is a schematic diagram of the pruning process provided by the present invention;

[0073] Figure 6 This invention provides a comparison chart of the number of network channels before and after pruning.

[0074] Figure 7 A structural diagram of the video link provided by this invention is created;

[0075] Figure 8 The overall logic diagram of the reasoning code provided for this invention;

[0076] Figure 9 This is a diagram showing the driver distraction behavior recognition results provided by the present invention;

[0077] Figure 10 The diagram shows the real-time inference results of the system under different behaviors provided by this invention. Detailed Implementation

[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] See Figure 1 As shown, this invention discloses a method for recognizing distracted driving behavior of drivers based on contrastive learning and real-time edge computing, including the following steps:

[0080] Obtain the target dataset;

[0081] Construct a Yolov5 detection model and pre-train it.

[0082] The pre-trained Yolov5 detection model is converted into a Yolov5 classification model and retrained on the target dataset to obtain the trained PT model.

[0083] The PT model is improved based on the contrastive learning method;

[0084] The improved PT model is pruned using BN layers, and then the PT model is quantized.

[0085] The PT model is first converted to the ONNX model and then to the RKNN model, and then pre-compiled.

[0086] Finally, a pre-compiled RKNN model was used for real-time inference on an edge computing platform.

[0087] Furthermore, the edge computing platform is the RV1126 edge computing platform.

[0088] Furthermore, the specific details of pre-training the Yolov5 detection model are as follows:

[0089] See Figure 2 As shown, the YOLOv5 detection model is used for behavior classification based on driver behavior images. The backbone of the YOLOv5 detection model is used for image feature extraction and classification. The neck and head structures are deleted, and the last SPFF layer of the YOLOv5 detection model is changed to a classification layer.

[0090] Specifically, the classification layer consists of a Softmax layer and a Sigmoid layer, used to transform the features into a 10*1 vector, representing the probability distribution of each category. Since the detection device is in a fixed position and the subject occupies most of the image area, object detection is unnecessary, so the object detection part of the model is simplified. Furthermore, to improve adaptability and computation speed on edge computing platforms, the activation function is replaced with ReLU instead of Silu.

[0091] For details, see Figure 4 The diagram shown is a structural diagram of the YOLOv5-CLS model.

[0092] The target dataset is divided into training, validation, and test sets, ensuring that the training, validation, and test sets are independent and randomized. Within each subset, the images are stored in folders according to their categories.

[0093] Further, see Figure 3 As shown, the specific improvements to the PT model based on the contrastive learning method are as follows:

[0094] A simulated open set design was implemented, using the target dataset and training with different numbers of distracted driving behaviors in the training set. During recognition, all distracted driving behavior categories were used to obtain categories of distracted driving behaviors not seen in the test set.

[0095] The closed-set classification task is used as the main task to form the classification loss constraint, and a discriminative learning task is added as the auxiliary task to form the contrast loss constraint. The main task and the auxiliary task share a backbone network and use the same coding layer to convert the image into a feature map.

[0096] The model input size is 480×480, the batch size is 256, the SGD optimizer is used, the learning rate is 1e×10^-4, and the temperature coefficient of contrastive loss is 0.25.

[0097] The task is divided into two categories: normal driving and abnormal driving. AUC, accuracy, and F1 score are used as evaluation metrics. The baseline model is Yolov5s-CLS, which only uses the cross-entropy loss function, while SCL refers to the improved version that uses contrastive learning loss.

[0098] Furthermore, the classification loss constraint uses the cross-entropy loss function:

[0099]

[0100] Where M is the number of categories, N is the batch size, and L is the number of categories. i y is the negative of the sum of the product of the sign function and the logarithm of the probability. ic p is a symbolic function ic Let i be the predicted probability that observed sample i belongs to category c;

[0101] The contrastive loss constraint uses the contrastive loss function, which incorporates sample label information during training. It utilizes supervised learning, where positive samples include those with the same label, and negative samples include those with different labels. Supervised contrastive loss aligns the embedding representation obtained by the feature extraction network with the labels, as shown in the formula below:

[0102]

[0103] Where N is the batch size, L i for P(i) represents the relationship between x and x i For samples with the same label, · represents the inner product. To normalize the feature vectors of positive samples, To normalize the feature vectors of samples with labels different from the positive samples, Normalize the feature vectors of the augmented version of the positive samples, where τ is the contrastive learning temperature parameter, and x k For x i' Different augmented versions, x i' For x i An expanded version.

[0104] Furthermore, the specific details of performing BN layer pruning on the improved PT model are as follows:

[0105] First, initialize the network, identify and label the channels with the smallest scaling factor, and apply L1 regularization to each scaling factor in the channel for adjustment. Each scaling factor corresponds to a specific convolutional channel. The mathematical expression for channel pruning is:

[0106]

[0107] Where W is the trainable weight, l is the normal training loss of CNN, f is the training output, x is the training input, y is the training target, λ is the hyperparameter, and g(γ) is the sparsity penalty for the scaling factor, which is chosen here as |γ|, i.e., the L1 norm.

[0108] Perform the following operations during the pruning of the BN layer:

[0109]

[0110] in, For the standardized input, z in For the input of the BN layer, μ β The average value of the input activation. z is the standard deviation of the input activation, ∈ is the bias, and z out γ is the output of the BN layer, β is the affine transformation ratio, and β is the affine transformation bias.

[0111] The coefficient of the hyperparameter λ gradually decreases as the number of training iterations increases. When performing sparse training, the coefficient is set to 0.001, the batch size is set to 128, and the training runs for 50 epochs.

[0112] Prune the PT model according to the scaling factor, and prune the BN layer with the smallest scaling factor.

[0113] The pruned PT model was fine-tuned and trained using the original dataset.

[0114] Furthermore, the specific details of quantizing the PT model after pruning are as follows:

[0115] After pruning, the PT model is linearly quantized. The formula for linear quantization is as follows:

[0116] r = S(qZ)

[0117]

[0118] Where r is a floating-point real number, q is a fixed-point integer, S is the scaling factor, representing the proportional relationship between the real number and the integer, and Z is the zero point, representing the integer corresponding to 0 of the floating-point real number after mapping, calculated as follows:

[0119]

[0120] Where, r max r min These are the maximum and minimum values ​​of the floating-point real number r and q, respectively. max q min These are the maximum and minimum values ​​of the fixed-point integer q, respectively;

[0121] The experiment compares the accuracy of four different precision parameters: original precision, floating-point half precision, INT8 precision, and UINT8 precision, and analyzes the accuracy loss of different classification models under the four quantization precisions.

[0122] Furthermore, the specific steps for converting the PT model to an ONNX model and then to an RKNN model, followed by pre-compilation, are as follows:

[0123] The trained model is saved as a pt model with a network structure, and then converted into an ONNX model for inference using the ONNX library;

[0124] Transform the ONNX model into an RKNN model capable of inference on an edge computing platform;

[0125] Create a dataset for model quantization, and randomly select training set images for fine-tuning the model;

[0126] Finally, the model was converted from the original FP32 precision model to a quantized RKNN model with UINT8 precision;

[0127] By pre-compiling the RKNN model using RKtoolkit, the speed of model loading and switching is improved, thus enhancing real-time performance.

[0128] Furthermore, the specific details of using the pre-compiled RKNN model for real-time inference on the edge computing platform are as follows:

[0129] Connect the IMX335 camera to the MIPI-CSI interface, modify the development board's device tree, and enable the node for image stream transmission to transmit image data.

[0130] The image is optimized according to IQ file, ISP, and ISPP, and then sent to the VI node;

[0131] The image is sent to the RGA module in two separate links for format conversion, rotation, and cropping. In the first link, the image is converted from NV12 format to RGB888 format and cropped to 480*480 size. In the second link, the image is converted from NV12 format to RGB888 format, rotated 270°, and cropped to 1280*720 size.

[0132] The inference results are printed to the image frames after RGA cropping using OpenCV, and the corresponding driving behavior is printed according to the sequence number of the inference results.

[0133] The image frame containing the inference results is sent to the display screen of the MIPI-DSI interface for result display.

[0134] Furthermore, driving behaviors include: normal driving, typing with the left hand, typing with the right hand, making a phone call with the left hand, making a phone call with the right hand, operating the center console, drinking water, retrieving items, applying makeup, and talking to passengers.

[0135] In one specific embodiment, one possible implementation method uses the StataFarm dataset for training. The dataset includes ten categories of driving behavior images: normal driving, typing with the left hand, typing with the right hand, making a phone call with the left hand, making a phone call with the right hand, operating the center console, drinking water, retrieving items, applying makeup, and conversing with passengers. The dataset is randomly divided into a training set, a test set, and an evaluation set, with no overlap between the three sets. The test and evaluation sets are not visible during the entire training process. The ten behavior categories are then divided into ten folders and placed under the dataset folder.

[0136] 1. The results of different classification networks on the State Farm dataset are shown in Table 1.

[0137] Table 1. Accuracy, number of layers, number of parameters, number of gradients, and computational cost of each classification network.

[0138] Model ACC Layers Parameters Gradients GFLOPs EffecientNet_b0 0.978 337 4020358 4020358 7.3 EffecientNet_b1 0.982 473 6525994 6525994 10.8 EffecientNet_b2 0.979 473 7715084 7715084 12.5 EffecientNet_b3 0.980 533 10711602 10711602 18.1 ResNet-18 0.980 68 11181642 11181642 29.8 ResNet-34 0.977 116 21289802 21289802 60.1 ResNet-50 0.976 151 23528522 23528522 67.5 ResNet-101 0.976 287 42520650 42520650 128.4 YOLOv5n-CLS 0.977 149 1224810 1224810 3.0 YOLOv5s-CLS 0.979 149 4185290 4185290 10.5

[0139] Among these models, EffecientNet_b1 achieved the highest accuracy, but its deep network layers can lead to increased quantization loss during deployment. ResNet-18 had the fewest layers, but its computational cost was high due to the extensive use of residual structures, making it unsuitable for real-time detection on edge computing platforms with limited resources. YOLOv5n-CLS had the fewest GFLOPs, but its accuracy decreased by 0.02 compared to YOLOv5s-CLS. Therefore, considering computational cost, network depth, and accuracy, YOLOv5s-CLS was ultimately chosen as the baseline model for further improvement and deployment.

[0140] 2. Train the modified Yolov5-CLS model, dividing the dataset into training, validation, and test sets. Set the image size to 480*480 pixels, the batch size to 128, and train for 10 epochs.

[0141] The trained model was saved as a pt model with weights and path information. The specific results of the YOLOv5s-CLS model in each classification are shown in Table 2.

[0142] Table 2. Top-1 and Top-5 accuracy of the YOLOv5s-CLS model.

[0143] Class Top1_acc Top5_acc All 0.979 0.999 C0 0.978 0.997 C1 0.995 1 C2 0.994 0.997 C3 0.996 1 C4 0.994 0.998 C5 0.991 1 C6 0.990 1 C7 0.996 0.999 C8 0.993 0.999 C9 0.910 0.999

[0144] Among the ten types of actions, the recognition rate for C9 talking to passengers was the lowest. This may be because the action is similar to the behavior of looking at the right-side rearview mirror in safe driving. This problem will be addressed in subsequent model improvements.

[0145] 3. A contrastive learning method is incorporated into the model during training. A baseline model using an FC layer as the binary classification output layer and trained with a cross-entropy loss function is used for comparison. The baseline model only uses the cross-entropy loss function, while the model proposed in this invention uses a contrastive loss function. To simulate open set recognition, different numbers of distracted driving behaviors are used in the training set for training, and all distracted driving behavior categories are used during recognition, thereby achieving the goal of including previously unseen distracted driving behavior categories in the test set. In this part of the experiment, the accuracy of training with normal driving combined with 2, 4, 6, and all abnormal driving categories is compared. All experiments use the same hyperparameter configuration. In this part, to measure the anomaly detection effect, the results are defined as a binary classification model, dividing the model into only two categories: normal driving and abnormal driving, to compare the results of the two models.

[0146] The model input size is 480×480, the batch size is 256, the SGD optimizer is used, the learning rate is 1e×10^-4, and the temperature coefficient of the contrastive loss is 0.25.

[0147] The baseline model is Yolov5s-CLS, which uses only the cross-entropy loss function, while SCL refers to the improved version that uses contrastive learning loss. The performance of the two models is shown in Table 3.

[0148] Table 3 shows the model performance trained with different numbers of abnormal driving behavior classes.

[0149]

[0150] 4. First, initialize the network, identify and label channels with small scaling factors; then remove these labeled channels and their associated weights; finally, fine-tune the pruned network to optimize performance.

[0151] To maintain network structure consistency, the two batch normalization (BN) layers in the bottleneck involving addition operations, as well as the BN layers in their upstream convolutional modules, should be included in the unpruned list. These three BN layers are cascaded; if the number of channels within them changes, the addition operation will not execute correctly. Taking the first C3 module of the backbone as an example, the unpruned parts are represented in green, while the pruned training parts are represented in orange, such as... Figure 5 As shown.

[0152] The left-hand convolutional module needs pruning because it uses concatenation instead of addition. The coefficient of the hyperparameter γ gradually decreases with the number of training epochs. During sparse training, the coefficient is set to 0.001, the batch size is set to 128 (this remains unchanged in subsequent iterations), and the training runs for 50 epochs.

[0153] The number of channels in the Yolov5s model before and after pruning is as follows: Figure 6 As shown, since the shallow layers of the network have fewer channels, the number of channels in the model before and after pruning is compared intuitively. Figure 6 A logarithmic coordinate axis was used as the vertical axis.

[0154] according to Figure 6 It can be seen that shallower networks undergo less channel pruning, while deeper networks undergo more pruning.

[0155] The accuracy of the pruned model decreased slightly, but this can be partially restored by fine-tuning the training to adapt the model weights to the new network model. The decrease in accuracy may be due to changes in the network structure, but after pruning, redundant structures are removed, and the remaining weights are sufficient for the inference process, allowing the model performance to recover quickly. This process demonstrates that the pruning strategy employed in this study is effective.

[0156] The number of channels, parameters, inference speed, and weight file size of the pruned model are shown in Table 4.

[0157] Table 4 Number of passages before and after pruning

[0158] Model Channel number Parameter quantity Accuracy Inference speed (ms) Weight file size (MB) Before pruning 9504 4185290 97.9 0.6 10.5 After pruning 3565 2487258 97.7 0.7 6.24 After fine-tuning 3565 2487258 98.1 0.7 6.26

[0159] After pruning, the number of channels decreased by 62.49%, and a large number of Batch Normalization (BN) layers in deeper networks were removed, indicating that the deeper parts of the network contained a large number of redundant parameters. The accuracy of the model did not decrease significantly after removing this data. However, the inference speed of the model increased by 0.1ms after pruning, possibly because the number of channels in the pruned model is no longer 2^n. This type of model is not conducive to parallel computing optimization on GPUs. However, on edge inference platforms, NPU processors do not support large-scale parallel computing, so it will not affect the final deployment. After fine-tuning, the model accuracy increased by 0.2 compared to the original model. This is likely because pruning removed redundant parts of the network while retaining the main feature extraction parts. After fine-tuning, these network parameters were further optimized, thus resulting in a slight improvement in model performance.

[0160] When the model is actually deployed, the weights no longer need to be updated. The weights can be converted from FP32 to UINT8 precision via a computational mapping for easier storage and computation. Currently, the quantization method used in this study is linear quantization, which processes the data through a simple linear mapping. The following is the mapping formula between floating-point real numbers (r) and fixed-point integers (q):

[0161] r = S(qZ)

[0162]

[0163] S is the scaling factor, representing the proportional relationship between real numbers and integers; Z is the zero point, representing the integer corresponding to 0 in a floating-point real number after mapping. Their calculation methods are as follows:

[0164]

[0165] Where, r max r min These are the maximum and minimum values ​​of the floating-point real number r and q, respectively. max q min These are the maximum and minimum values ​​of the fixed-point integer q, respectively.

[0166] The experiment compared the accuracy of the model under four different precision parameters: original precision, floating-point half precision, INT8 precision, and UINT8 precision. The specific values ​​are shown in Table 5.

[0167] Table 5. Accuracy of the Driver Behavior Classification Model after Quantization at Different Refinements

[0168] Model FP32-ACC FP16-ACC INT8-ACC UINT8-ACC Effecientnet_b0 0.978 0.973 0.958 0.963 Effecientnet_b1 0.982 0.972 0.962 0.962 Effecientnet_b2 0.979 0.969 0.960 0.961 Effecientnet_b3 0.980 0.971 0.962 0.961 Resnet-18 0.980 0.978 0.971 0.971 ResNet-34 0.977 0.970 0.967 0.965 ResNet-50 0.976 0.966 0.958 0.959 ResNet-101 0.976 0.971 0.968 0.966 YOLOv5n-CLS 0.977 0.976 0.975 0.975 YOLOv5s-CLS 0.979 0.978 0.976 0.977

[0169] As shown in Table 5, quantization of different precisions leads to a decrease in model accuracy. However, for the EfficientNet series, the accuracy generally decreases with quantization precisions ranging from FP32 to FP16, and then to INT8 and UINT8. From EfficientNet_b0 to EfficientNet_b3, accuracy decreases with increasing model complexity, indicating that the impact of quantization may be more significant for more complex models. For ResNet-18 and ResNet-34, accuracy decreases with different quantization precisions, but the overall trend is relatively consistent. This may be because these two models have relatively shallow layers, and the decrease in model accuracy caused by quantization of model weight parameters is more consistent. For ResNet-50 and ResNet-101, accuracy remains stable or slightly decreases with quantization precisions from FP32 to FP16, but decreases significantly with INT8 and UINT8. This indicates that deeper networks are more affected by quantization, possibly because deeper networks have a larger number of parameters, and the error caused by multi-layer quantization accumulates layer by layer, leading to a significant decrease in accuracy. For the YOLOv5 series: the accuracy of the YOLOv5n-CLS and YOLOv5s-CLS models varies relatively little under different precision quantizations, with no significant differences. This may be because the YOLOv5-CLS series models have a relatively simple structure, without too many branches and with a shallow number of network layers. Therefore, the model accuracy changes little after quantization, which also proves the correctness of using the YOLOv5 model for deployment in this invention.

[0170] The edge computing platform used in this invention includes an RV1126 processor, a 5.5-inch touchscreen, and an IMX335 camera. The RV1126 processor is a quad-core ARM Cortex-A7 and RISC-V MCU, containing 2GB DDR4 memory, 8GB eMMC storage, a 2.0 TopsNPU, 14 MISP, and supports simultaneous input of 4K H.264 / H.265 video encoding and decoding from three cameras. The touchscreen is a 5.5-inch MIPI LCD touchscreen, supporting a resolution of 720*1280, and uses a 4-lane MIPIDISI interface for data transmission. The camera is an MCIMX335 camera module, supporting 5MP resolution, 2K resolution, a fixed focal length of 4.4mm, an aperture of f / 2.0, and a field of view of 70°, and uses a 4-lane MIPIDISI interface for data transmission.

[0171] After initializing the ISP service, the RKmedia system was initialized, and video links were created and bound. In this study, two links were created: one for model inference and one for streaming images to the display. See [link to relevant documentation]. Figure 7 As shown.

[0172] When using RKmedia, it's important to ensure that the image width is 16-bit aligned and the height is 8-bit aligned. If misalignment occurs, pad with zeros; otherwise, it may lead to memory misalignment and resulting in image distortion. When using the IMX335 camera, the maximum ISPP resolution is 2592x1944. Furthermore, the data link must satisfy the following condition: Sensor = MIPI_DPHY ≥ ISP input ≥ ISP output.

[0173] To achieve high-performance model inference on edge computing platforms, this part of the inference code is written in C++. Compared to the Python version, the C++ inference code is more efficient and concise. The code logic is as follows: Figure 8 As shown, the specific content is as follows:

[0174] Model loading and initialization:

[0175] The code loads and initializes the model by calling the `rknn_init()` function. This function accepts the model file path and model size as parameters and returns an `rknn_context` object, which manages the model's context information, ensuring that the model can be correctly loaded and used during inference.

[0176] Set input / output information:

[0177] Use the `rknn_query()` function to query the model's input and output information, including details such as the number, size, and data type of the inputs and outputs. This information is crucial for subsequent data preprocessing and postprocessing, ensuring that the input data matches the model's expected format and size, and that the model's output can be correctly parsed.

[0178] Input data preprocessing:

[0179] After acquiring raw image data from the camera, necessary preprocessing operations were performed to meet the model's input requirements. These preprocessing operations included image scaling and format conversion. The images were converted using RGA according to the model's input requirements queried in the previous step, ensuring that the input data matched the model's expected input format and size.

[0180] Model Inference:

[0181] The model inference process is executed by calling the `rknn_run()` function. During inference, preprocessed image data is input into the model, which performs calculations and generates inference results; this step is the core of the entire inference process. In this step, the program creates a new thread for model inference, which runs in parallel with data preprocessing, thereby improving inference efficiency.

[0182] Post-processing of inference results:

[0183] Once the model inference is complete, the code retrieves the inference results from the model output and performs post-processing operations. Post-processing operations may include parsing the classification results, using OpenCV to overlay the inference results onto the final result image, and streaming to the RKmediaVO module to display the results on the screen via VENC.

[0184] Release resources:

[0185] After the entire inference process is completed, the code releases relevant resources, including the model context and image buffer, ensuring the effective use of system resources and avoiding problems such as resource leaks and memory overflows.

[0186] In this section, real-world vehicles were used for simulation testing. Because the experiment included distracted driving behavior, the vehicles were kept stationary to avoid traffic accidents, and recording was performed from a dual-viewpoint, such as... Figure 9 As shown, the left side displays the current image of the driver, while the right side displays the real-time images and inference results obtained by the edge computing platform. The inference results are displayed in the upper left corner of the screen.

[0187] The model performance is shown in Table 6:

[0188] Table 6. Model inference speed before and after pre-compilation.

[0189] Model Before pre-compilation (FPS) After pre-compilation (FPS) Original model 8 14 After pruning and quantization 27 45

[0190] The performance of the pruned and quantized inference model on RV1126 is shown in Table 7.

[0191] Table 7 Model Usage Before and After Optimization

[0192]

[0193] Real-time inference results of the model, such as Figure 10 As shown.

[0194] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0195] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A driver distraction driving behavior recognition method based on contrastive learning and real-time edge computing, characterized in that, The method comprises the following steps: obtaining a target data set; constructing a Yolov5 detection model and pre-training the Yolov5 detection model; changing the pre-trained Yolov5 detection model into a classification model of Yolov5 and re-training the Yolov5 detection model on the target data set to obtain a PT model after training; improving the PT model based on a contrast learning method; performing BN layer pruning on the improved PT model, and performing quantization processing on the PT model after pruning; converting the PT model into an ONNX model and then into an RKNN model and performing pre-compilation; finally, using the pre-compiled RKNN model to perform real-time inference on an edge computing platform; the specific content of pre-training the Yolov5 detection model is: the Yolov5 detection model is used for behavior classification based on driver behavior images, the Backbone of the Yolov5 detection model is used for picture feature extraction and classification, the neck and head structures are deleted, and the last SPFF layer of the Yolov5 detection model is changed into a classification layer; the target data set is divided into a training set, a validation set and a test set, and the training set, the validation set and the test set are independent and random, and under each subset, the pictures are stored in folders according to the categories; the specific content of improving the PT model based on the contrast learning method is: an open set simulation design is performed, the target data set is trained using different numbers of distracted driving behaviors in the training set, all distracted driving behavior categories are used for identification, so that the categories of distracted driving behaviors that have not been seen in the test set are obtained; the closed set classification task is used as a main task to form a classification loss constraint, and a discriminative learning task is added as an auxiliary task to form a contrast loss constraint, the main task and the auxiliary task share a backbone network, and the same encoding layer is used to convert the image into a feature map; The model input size is 480x480, the batch size is 256, the SGD optimizer is used, and the learning rate is 1x10 -4 The temperature coefficient of the contrast loss is 0.

25. two classification tasks of normal driving and abnormal driving are divided, AUC, accuracy and F1 score are used as evaluation indexes, the Baseline model is a Yolov5s-CLS using only a cross-entropy loss function, and SCL refers to a version improved by using a contrast learning loss.

2. The driver distraction driving behavior recognition method based on contrastive learning and real-time edge computing according to claim 1, characterized in that, The edge computing platform is an RV1126 edge computing platform.

3. The driver distraction driving behavior recognition method based on contrastive learning and real-time edge computing according to claim 1, characterized in that, the classification loss constraint uses a cross-entropy loss function: where, is the number of classes, is the batch size, is the negative of the sum of the product of the sign function and the log of the probability, is the sign function, is the predicted probability that observation sample i belongs to class c; the contrast loss constraint uses a contrast loss function, the label information of the sample is used for training, a supervised learning is used, the positive samples include samples with the same label, the negative samples include samples with different labels, the supervised contrast loss can align the embedding representation obtained by the feature extraction network with the label, and the formula is as follows: wherein, is a batch size, is , represents samples with the same label, is an inner product, is a positive example sample feature vector normalization, is a sample feature vector normalization different from the positive example sample, is a positive example sample augmented version feature vector normalization, is a contrastive learning temperature parameter, is a different augmented version from , is an augmented version of 4. The driver distraction driving behavior recognition method based on contrastive learning and real-time edge computing according to claim 1, characterized in that, the specific content of performing BN layer pruning on the improved PT model is: first, initialize the network, identify and mark the channels with the smallest scaling factor, and adjust each scaling factor in the channel by applying L1 regularization, each scaling factor corresponds to a specific convolution channel, and the mathematical expression of channel pruning is: wherein, is a trainable weight, is a normal training loss for the CNN, is a training output, is a training input, is a training target, is a hyperparameter, is a sparsity penalty on the scaling factors, here chosen as | i.e. the L1 norm; the following operations are performed in the pruning operation of the BN layer: wherein, is the standardized input, is the input to the BN layer, is the mean of the input activations, is the standard deviation of the input activations, is the bias, is the output of the BN layer, is the affine change scale, is the affine transform bias; The coefficient of the hyperparameter λ gradually decreases with the increase of the number of training, and when the sparse training is performed, the coefficient is 0.001 and the batch_size is 128, and the running is 50 epochs; According to the scaling factor, the PT model is pruned, and the BN layer with the smallest scaling factor is pruned; The PT model after pruning is fine-tuned using the original data set.

5. The driver distraction driving behavior recognition method based on contrastive learning and real-time edge computing according to claim 1, characterized in that, The specific content of quantizing the PT model after pruning is as follows: After pruning, the PT model is linearly quantized, and the formula of linear quantization is as follows: wherein, is a floating-point real number, is a fixed-point integer, S is a scale, representing the proportional relationship between the real number and the integer, and Z is a zero point, representing the integer corresponding to the 0 of the floating-point real number after mapping, and the calculation method is as follows: wherein are the maximum and minimum values of the floating-point real numbers r respectively, are the maximum and minimum values of the fixed-point integers q; The experiment compares the accuracy of the original accuracy, floating-point half-precision, INT8 accuracy and UINT8 accuracy under four different parameter accuracies, and analyzes the accuracy loss of different classification models under four quantization accuracies.

6. The driver distraction driving behavior recognition method based on contrastive learning and real-time edge computing according to claim 1, characterized in that, The specific content of converting the PT model into an ONNX model and then into an RKNN model and pre-compiling is as follows: Save the trained model as a PT model with network structure, and convert it into an ONNX model for inference through the ONNX library; Convert the ONNX model into an RKNN model that can be used for inference on the edge computing platform; Establish a data set for model quantization, randomly select training set pictures for fine-tuning the model; Finally, the model is converted from the original FP32 precision model to the UINT8 precision quantized RKNN model; Pre-compile the RKNN model through RKtoolkit to improve the model loading and switching speed and improve the real-time performance.

7. The driver distraction driving behavior recognition method based on contrastive learning and real-time edge computing according to claim 1, characterized in that, The specific content of using the pre-compiled RKNN model for real-time inference on the edge computing platform is as follows: Insert the IMX335 camera into the MIPI-CSI interface, modify the device tree of the development board, and turn on the node of the image code stream transmission to perform image data transmission; After optimizing the image according to the IQ file, ISP and ISPP, send it to the VI node; The picture is split into two links to be sent into the RGA module for format conversion, rotation and cropping. In the first link, the picture is converted from NV12 format to RGB888 format and cropped to 480 480size. In the second link, the picture is converted from NV12 format to RGB888 format, rotated by 270° and cropped to 1280 720size. Print the inference result to the RGA cropped picture frame through OpenCV, and print the corresponding driving behavior according to the serial number of the inference result; Send the picture frame with the inference result to the display screen of the MIPI-DSI interface for result display.

8. The driver distraction driving behavior recognition method based on contrastive learning and real-time edge computing according to claim 7, characterized in that, Driving behaviors include: normal driving, left-hand typing, right-hand typing, left-hand calling, right-hand calling, operating the center console, drinking, taking things, making up, and talking with passengers.

Images