Driver distraction driving behavior identification method based on comparative learning and real-time edge calculation

Abstract

The invention discloses a driver distracted driving behavior recognition method based on comparative learning and real-time edge calculation, and relates to the technical field of intelligent traffic monitoring. Comprises: obtaining a target data set; a Yov5 detection model is constructed, and the Yov5 detection model is pre-trained; changing a pre-trained Yov5 detection model into a Yov5 classification model, and performing training again on the target data set to obtain a trained PT model; improving the PT model based on a comparative learning method; performing BN layer pruning on the improved PT model, and performing quantization processing on the PT model after pruning; converting the PT model into an ONNX model, then converting the ONNX model into an RKNN model, and pre-compiling the ONNX model; and finally, performing real-time reasoning on an edge computing platform by using the pre-compiled RKNN model. According to the method, the model learns the feature extractor with the characterization capability by utilizing comparative learning, so that the accuracy of driver behavior classification is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic monitoring technology, and more specifically to a method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing. Background Art

[0002] With the rapid development of the transportation industry, the number of motor vehicles and motor vehicle drivers on the road has increased rapidly. While the advancement of transportation has brought convenience to people, it has also led to an increasing number of traffic accidents, posing a serious threat to people's lives and property safety. Among them, illegal driving behaviors such as drivers not wearing seat belts and making phone calls are one of the main causes of traffic accidents. In this context, facing the increasingly severe traffic safety situation, the development of intelligent traffic monitoring systems has attracted the attention of many researchers.

[0003] The technical solution in the prior art is: improve the ResNet50 network structure for driver behavior detection, replace the second layer of convolution in the convolution residual module with depthwise separable convolution (DSCB), replace the first and second layers of convolution in the identity residual module with depthwise separable convolution, replace the third layer with grouped convolution (DSIB), and reduce the number of identity residual modules and the number of neurons in the last fully connected layer to achieve lightweight and reduce the number of parameters. At the same time, channel attention mechanism and spatial attention mechanism modules are embedded between each convolution residual module and the identity residual module to improve the detection rate.

[0004] In the step of modifying the number of residual modules, the number of residual blocks in the second, third, fourth, and fifth stages of ResNet50 was changed from 3, 4, 6, 3 to 3, 3, 4, 3, and the last fully connected layer was removed, and the number of neurons in the fully connected layer was reset to 256; when adding attention mechanism modules, a total of 13 channel attention mechanism modules and spatial attention mechanism modules were added to focus on the importance of different feature channels and different positions of the image.

[0005] This method only proposes an idea for model improvement and can only detect two types of behaviors: making phone calls and not wearing seat belts. This method does not really propose a feasible method that can be implemented, and does not provide a method for deploying the model on the edge computing platform. At the same time, this method does not consider the support of the inference library operators after the model is deployed to the development board, and the impact of adding the attention mechanism on the computational complexity.

[0006] Therefore, it is an urgent problem for those skilled in the art to propose a method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing to solve the difficulties existing in the prior art. Summary of the invention

[0007] In view of this, the present invention provides a method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing, which is used to solve the technical problems existing in the prior art.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing, comprising the following steps:

[0010] Get the target dataset;

[0011] Build the Yolov5 detection model and pre-train the Yolov5 detection model;

[0012] The pre-trained Yolov5 detection model is changed to the Yolov5 classification model, and it is re-trained on the target dataset to obtain the trained PT model;

[0013] Improve the PT model based on contrastive learning method;

[0014] The improved PT model is pruned on the BN layer, and the PT model is quantized after pruning;

[0015] Convert the PT model to ONNX model first and then to RKNN model and precompile;

[0016] Finally, the precompiled RKNN model is used for real-time reasoning on the edge computing platform.

[0017] In the above method, optionally, the edge computing platform is an RV1126 edge computing platform.

[0018] The above method optionally includes the following steps for pre-training the Yolov5 detection model:

[0019] The Yolov5 detection model is used to classify the driver's behavior based on the driver's behavior image. The Backbone of the Yolov5 detection model is used for image feature extraction and classification. The neck and head structures are deleted and the last SPFF layer of the Yolov5 detection model is changed to a classification layer.

[0020] Divide the target dataset into training set, validation set and test set to ensure that the training set, validation set and test set are independent and random. Under each subset, store the images in folders according to categories.

[0021] The above method, optionally, improves the PT model based on the contrastive learning method as follows:

[0022] Conduct a simulated open set design, use the target dataset and different numbers of distracted driving behaviors in the training set for training, and use all distracted behavior categories during recognition, so as to obtain categories of distracted driving behaviors that have not been seen in the test set;

[0023] The closed-set classification task is used as the main task to form a classification loss constraint, and a discriminative learning task is added as an auxiliary task to form a contrast loss constraint. The main task and the auxiliary task share a backbone network and use the same encoding layer to convert the image into a feature map.

[0024] The model input size is 480×480, the batch size is 256, the SGD optimizer is used, the learning rate is 1e×10^-4, and the temperature coefficient of the contrast loss is 0.25;

[0025] It is divided into two classification tasks: normal driving and abnormal driving. AUC, accuracy and F1 score are used as evaluation indicators. The baseline model is Yolov5s-CLS that only uses the cross entropy loss function, and SCL refers to an improved version using contrastive learning loss.

[0026] The above method optionally uses the cross entropy loss function for the classification loss constraint:

[0027]

[0028] Where M is the number of categories, N is the batch size, and L is the i is the negative of the sum of the products of the sign function and the logarithm of the probability, y ic is the symbolic function, p ic is the predicted probability that the observed sample i belongs to category c;

[0029] Contrastive loss constraint uses contrastive loss function, uses sample label information to participate in training, uses supervised learning, positive samples include samples with the same label, negative samples include samples with different labels, and uses supervised contrastive loss to align the embedding representation obtained by the feature extraction network with the label. The formula is as follows:

[0030]

[0031] Where N is the batch size, L i for P(i) represents the i For samples with the same label, · is the inner product, Normalize the feature vector of the positive sample. Normalize the feature vector of samples with different labels from the positive samples. is the normalized feature vector of the augmented version of the positive sample, τ is the contrastive learning temperature parameter, x k For xi' Different augmented versions, x i' For x i An expanded version of .

[0032] The above method optionally performs BN layer pruning on the improved PT model as follows:

[0033] First, initialize the network, identify and mark the channel with the smallest scaling factor, and apply L1 regularization to each scaling factor in the channel for adjustment. Each scaling factor corresponds to a specific convolution channel. The mathematical expression of channel pruning is:

[0034]

[0035] Where W is the trainable weight, l is the normal training loss of CNN, f is the training output, x is the training input, y is the training target, λ is the hyperparameter, and g(γ) is the sparsification penalty for the scale factor, which is |γ|, i.e., the L1 norm.

[0036] The following operations are performed in the pruning operation of the BN layer:

[0037]

[0038] in, is the normalized input, z in is the input of the BN layer, μ β is the average of the input activations, is the standard deviation of the input activation, ∈ is the bias, z out is the output of the BN layer, γ is the affine change ratio, and β is the affine transformation deviation;

[0039] The coefficient of the hyperparameter λ decreases as the number of training times increases. When performing sparse training, the coefficient is 0.001, the batch_size is 128, and 50 epochs are run;

[0040] Prune the PT model according to the scaling factor and prune the BN layer with the smallest scaling factor;

[0041] The pruned PT model is fine-tuned using the original dataset.

[0042] In the above method, the specific content of quantizing the PT model after pruning is optional:

[0043] After pruning, the PT model is linearly quantized. The formula for linear quantization is as follows:

[0044] r=S(qZ)

[0045]

[0046] Among them, r is a floating point real number, q is a fixed point integer, S is the scaling ratio, which indicates the proportional relationship between real numbers and integers, and Z is the zero point, which indicates the integer corresponding to the floating point real number 0 after mapping. The calculation method is as follows:

[0047]

[0048] Among them, r max , r min are the maximum and minimum values ​​of the floating point real number r, q max ,q min are the maximum and minimum values ​​of the fixed-point integer q respectively;

[0049] The experiment compares the accuracy under four different parameter precisions: original precision, floating-point half precision, INT8 precision, and UINT8 precision, and analyzes the accuracy loss of different classification models under four quantization precisions.

[0050] The above method optionally converts the PT model into an ONNX model and then into an RKNN model and precompiles it as follows:

[0051] Save the trained model as a pt model with a network structure, and convert it into an ONNX model for inference through the ONNX library;

[0052] Convert the ONNX model into an RKNN model that can be inferred on the edge computing platform;

[0053] Create a data set for model quantization and randomly select training set images for fine-tuning the model;

[0054] Finally, the model is converted from the original FP32 precision model to the RKNN model with UINT8 precision quantization.

[0055] Precompile the RKNN model through RKtoolkit to increase the speed of model loading and switching and improve real-time performance.

[0056] The above method is optional. Finally, the specific content of using the pre-compiled RKNN model to perform real-time reasoning on the edge computing platform is:

[0057] Insert the IMX335 camera into the MIPI-CSI interface, modify the development board device tree, turn on the node for image code stream transmission, and transmit image data;

[0058] After optimizing the image according to the IQ file, ISP, and ISPP, it is sent to the VI node;

[0059] The image is divided into two links and sent to the RGA module for format conversion, rotation and cropping. In the first link, the image is converted from NV12 format to RGB888 format and cropped to 480*480 size; in the second link, the image is converted from NV12 format to RGB888 format, rotated 270° and cropped to 1280*720 size;

[0060] Print the inference results to the RGA cropped image frame through OpenCV, and print the corresponding driving behavior according to the sequence number of the inference results;

[0061] The image frame with the inference result is sent to the display screen of the MIPI-DSI interface for result display.

[0062] In the above method, optional driving behaviors include: normal driving, typing with the left hand, typing with the right hand, making a phone call with the left hand, making a phone call with the right hand, operating the center console, drinking water, getting something, putting on makeup, and talking to passengers.

[0063] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing, which has the following beneficial effects:

[0064] Using contrastive learning enables the model to learn a feature extractor with good representation capabilities, thereby effectively improving the accuracy of driver behavior classification and having higher robustness;

[0065] Through fine-tuning training, the model weights are adapted to the new network model, thereby restoring the original accuracy to a certain extent;

[0066] The YOLOv5-CLS series has a simple model structure, without too many branches and with a shallow network layer. Therefore, the model accuracy changes little after quantization, which also proves the correctness of using the YOLOv5 model for deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0068] Figure 1 A flowchart of a method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing provided by the present invention;

[0069] Figure 2 A schematic diagram of modifying the YOLOv5 detection model provided by the present invention;

[0070] Figure 3 A structural diagram of the comparative learning method provided by the present invention;

[0071] Figure 4 The YOLOv5-CLS model structure diagram provided by the present invention;

[0072] Figure 5 A schematic diagram of the pruning situation provided by the present invention;

[0073] Figure 6 A comparison chart of the number of network channels before and after pruning provided by the present invention;

[0074] Figure 7 Creating a structural diagram for the video link provided by the present invention;

[0075] Figure 8 The overall logic diagram of the reasoning code provided by the present invention;

[0076] Fig. 9 A driver distraction behavior recognition result diagram provided by the present invention;

[0077] Fig.10 This is a diagram of the real-time reasoning results of the system under different behaviors provided by the present invention. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] See also Figure 1 As shown, the present invention discloses a method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing, comprising the following steps:

[0080] Get the target dataset;

[0081] Build the Yolov5 detection model and pre-train the Yolov5 detection model;

[0082] The pre-trained Yolov5 detection model is changed to the Yolov5 classification model, and it is re-trained on the target dataset to obtain the trained PT model;

[0083] Improve the PT model based on contrastive learning method;

[0084] The improved PT model is pruned on the BN layer, and the PT model is quantized after pruning;

[0085] Convert the PT model to ONNX model first and then to RKNN model and precompile;

[0086] Finally, the precompiled RKNN model is used for real-time reasoning on the edge computing platform.

[0087] Furthermore, the edge computing platform is an RV1126 edge computing platform.

[0088] Furthermore, the specific content of pre-training the Yolov5 detection model is as follows:

[0089] See also Figure 2 As shown, the Yolov5 detection model is used to classify the behavior based on the driver behavior image, the Backbone of the Yolov5 detection model is used for image feature extraction and classification, the neck and head structures are deleted, and the last SPFF layer of the Yolov5 detection model is changed to a classification layer;

[0090] Specifically, the classification layer consists of a Softmax and a Sigmoid layer, which is used to convert features into a 10*1 vector, that is, the probability distribution of each category. Because the detection device is in a fixed position and the detection subject occupies most of the image area, there is no need to perform target detection, so the target detection model is deleted. In addition, in order to improve the adaptability and calculation speed of the edge computing platform, the activation function is replaced by ReLU from Silu.

[0091] For details, see Figure 4 Shown is the YOLOv5-CLS model structure diagram.

[0092] Divide the target dataset into training set, validation set and test set to ensure that the training set, validation set and test set are independent and random. Under each subset, store the images in folders according to categories.

[0093] For further information, see Figure 3 As shown in the figure, the specific contents of improving the PT model based on the contrastive learning method are as follows:

[0094] Conduct a simulated open set design, use the target dataset and different numbers of distracted driving behaviors in the training set for training, and use all distracted behavior categories during recognition, so as to obtain categories of distracted driving behaviors that have not been seen in the test set;

[0095] The closed-set classification task is used as the main task to form a classification loss constraint, and a discriminative learning task is added as an auxiliary task to form a contrast loss constraint. The main task and the auxiliary task share a backbone network and use the same encoding layer to convert the image into a feature map.

[0096] The model input size is 480×480, the batch size is 256, the SGD optimizer is used, the learning rate is 1e×10^-4, and the temperature coefficient of the contrast loss is 0.25;

[0097] It is divided into two classification tasks: normal driving and abnormal driving. AUC, accuracy and F1 score are used as evaluation indicators. The baseline model is Yolov5s-CLS that only uses the cross entropy loss function, and SCL refers to an improved version using contrastive learning loss.

[0098] Furthermore, the classification loss constraint uses the cross entropy loss function:

[0099]

[0100] Where M is the number of categories, N is the batch size, and L is the i is the negative of the sum of the products of the sign function and the logarithm of the probability, y ic is the symbolic function, p ic is the predicted probability that the observed sample i belongs to category c;

[0101] Contrastive loss constraint uses contrastive loss function, uses sample label information to participate in training, uses supervised learning, positive samples include samples with the same label, negative samples include samples with different labels, and uses supervised contrastive loss to align the embedding representation obtained by the feature extraction network with the label. The formula is as follows:

[0102]

[0103] Where N is the batch size, L i for P(i) represents the i For samples with the same label, · is the inner product, Normalize the feature vector of the positive sample. Normalize the feature vector of samples with different labels from the positive samples. is the normalized feature vector of the augmented version of the positive sample, τ is the contrastive learning temperature parameter, x k For x i' Different augmented versions, x i' For x i An expanded version of .

[0104] Furthermore, the specific contents of BN layer pruning of the improved PT model are as follows:

[0105] First, initialize the network, identify and mark the channel with the smallest scaling factor, and apply L1 regularization to each scaling factor in the channel for adjustment. Each scaling factor corresponds to a specific convolution channel. The mathematical expression of channel pruning is:

[0106]

[0107] Where W is the trainable weight, l is the normal training loss of CNN, f is the training output, x is the training input, y is the training target, λ is the hyperparameter, and g(γ) is the sparsification penalty for the scale factor, which is |γ|, i.e., the L1 norm.

[0108] The following operations are performed in the pruning operation of the BN layer:

[0109]

[0110] in, is the normalized input, z in is the input of the BN layer, μ β is the average of the input activations, is the standard deviation of the input activation, ∈ is the bias, z out is the output of the BN layer, γ is the affine change ratio, and β is the affine transformation deviation;

[0111] The coefficient of the hyperparameter λ decreases as the number of training times increases. When performing sparse training, the coefficient is 0.001, the batch_size is 128, and 50 epochs are run;

[0112] Prune the PT model according to the scaling factor and prune the BN layer with the smallest scaling factor;

[0113] The pruned PT model is fine-tuned using the original dataset.

[0114] Furthermore, the specific contents of the quantization processing of the PT model after pruning are as follows:

[0115] After pruning, the PT model is linearly quantized. The formula for linear quantization is as follows:

[0116] r=S(qZ)

[0117]

[0118] Among them, r is a floating point real number, q is a fixed point integer, S is the scaling ratio, which indicates the proportional relationship between real numbers and integers, and Z is the zero point, which indicates the integer corresponding to the floating point real number 0 after mapping. The calculation method is as follows:

[0119]

[0120] Among them, r max , r min are the maximum and minimum values ​​of the floating point real number r, q max ,q min are the maximum and minimum values ​​of the fixed-point integer q respectively;

[0121] The experiment compares the accuracy under four different parameter precisions: original precision, floating-point half precision, INT8 precision, and UINT8 precision, and analyzes the accuracy loss of different classification models under four quantization precisions.

[0122] Furthermore, the specific contents of converting the PT model into an ONNX model and then into an RKNN model and precompiling are as follows:

[0123] Save the trained model as a pt model with a network structure, and convert it into an ONNX model for inference through the ONNX library;

[0124] Convert the ONNX model into an RKNN model that can be inferred on the edge computing platform;

[0125] Create a data set for model quantization and randomly select training set images for fine-tuning the model;

[0126] Finally, the model is converted from the original FP32 precision model to the RKNN model with UINT8 precision quantization.

[0127] Precompile the RKNN model through RKtoolkit to increase the speed of model loading and switching and improve real-time performance.

[0128] Furthermore, the specific content of using the precompiled RKNN model for real-time reasoning on the edge computing platform is as follows:

[0129] Insert the IMX335 camera into the MIPI-CSI interface, modify the development board device tree, turn on the node for image code stream transmission, and transmit image data;

[0130] After optimizing the image according to the IQ file, ISP, and ISPP, it is sent to the VI node;

[0131] The image is divided into two links and sent to the RGA module for format conversion, rotation and cropping. In the first link, the image is converted from NV12 format to RGB888 format and cropped to 480*480 size; in the second link, the image is converted from NV12 format to RGB888 format, rotated 270° and cropped to 1280*720 size;

[0132] Print the inference results to the RGA cropped image frame through OpenCV, and print the corresponding driving behavior according to the sequence number of the inference results;

[0133] The image frame with the inference result is sent to the display screen of the MIPI-DSI interface for result display.

[0134] Furthermore, driving behaviors include: normal driving, typing with the left hand, typing with the right hand, making a phone call with the left hand, making a phone call with the right hand, operating the center console, drinking water, getting something, putting on makeup, and talking to passengers.

[0135] In a specific embodiment, in a possible implementation method, the StataFarm data set is used for training, and the data set includes ten types of driving behavior pictures, including: normal driving, typing with the left hand, typing with the right hand, making a phone call with the left hand, making a phone call with the right hand, operating the center console, drinking water, taking things, putting on makeup, and talking to passengers. The data set is randomly divided into a training set, a test set, and an evaluation set. The data of the three data sets have no overlapping parts, and the test set and the evaluation set data are invisible throughout the training process. The ten types of behaviors are divided into ten folders according to the categories and placed in the data set folders respectively.

[0136] 1. The results of different classification networks on the State Farm dataset are shown in Table 1.

[0137] Table 1 Accuracy, number of layers, number of parameters, number of gradients, and amount of calculation for each classification network

[0138] Model ACC Layers Parameters Gradients GFLOPs EffecientNet_b0 0.978 337 4020358 4020358 7.3 EffecientNet_b1 0.982 473 6525994 6525994 10.8 EffecientNet_b2 0.979 473 7715084 7715084 12.5 EffecientNet_b3 0.980 533 10711602 10711602 18.1 ResNet-18 0.980 68 11181642 11181642 29.8 ResNet-34 0.977 116 21289802 21289802 60.1 ResNet-50 0.976 151 23528522 23528522 67.5 ResNet-101 0.976 287 42520650 42520650 128.4 YOLOv5n-CLS 0.977 149 1224810 1224810 3.0 YOLOv5s-CLS 0.979 149 4185290 4185290 10.5

[0139] Among these models, the model with the highest accuracy is EffecientNet_b1, but its network layers are relatively deep. A network that is too deep during deployment will lead to increased quantization loss. The model with the fewest layers is ResNet-18, but because the ResNet series networks use a large number of residual structures, the model has a large amount of computation. In edge computing platforms with limited computing power, such networks cannot achieve real-time detection. The network with the smallest GFLOPs is YOLOv5n-CLS, but its accuracy is reduced by 0.02 compared to YOLOv5s-CLS. Therefore, considering the three factors of computational complexity, network depth, and accuracy, the YOLOv5s-CLS model was finally adopted as the baseline model for further improvement and deployment.

[0140] 2. Use the modified Yolov5-CLS model for training, divide the dataset into training set, validation set and test set, set the image size to 480*480 pixels, batch size to 128, and train for 10 epochs.

[0141] The trained model is saved as a pt model with weight and path information. The specific results of the YOLOv5s-CLS model in each category are shown in Table 2.

[0142] Table 2 Top1 and Top5 accuracy of YOLOv5s-CLS model

[0143] category Top1_acc Top5_acc All 0.979 0.999 C0 0.978 0.997 C1 0.995 1 C2 0.994 0.997 C3 0.996 1 C4 0.994 0.998 C5 0.991 1 C6 0.990 1 C7 0.996 0.999 C8 0.993 0.999 C9 0.910 0.999

[0144] Among the ten types of actions, the recognition rate of C9 talking to passengers is the lowest. This may be because the action is similar to the behavior of partially looking at the right rearview mirror during safe driving. This problem will be solved in subsequent model improvements.

[0145] 3. Add contrastive learning method to the model for training, and use the baseline model that uses FC layer as the binary classification output layer and is trained using cross entropy loss function as a comparison. The baseline model only uses the cross entropy loss function, and the model proposed in the present invention uses the contrastive loss function. In order to simulate open set recognition, different numbers of distracted driving behaviors are used for training in the training set, and all distracted behavior categories are used for recognition, so as to achieve the purpose of including distracted driving behavior categories that have never been seen in the test set. In this part of the experiment, the accuracy of combined training using normal driving is compared with 2 categories, 4 categories, 6 categories, and all abnormal driving categories. All experiments use the same hyperparameter configuration. In this part, in order to measure the anomaly detection effect, this part of the results is defined as a binary classification model, which is divided into only two categories: normal driving and abnormal driving to compare the results of the two models.

[0146] The model input size is 480×480, the batch size is 256, the SGD optimizer is used, the learning rate is 1e×10^-4, and the temperature coefficient of the contrast loss is 0.25.

[0147] The baseline model is Yolov5s-CLS which only uses the cross entropy loss function, and SCL refers to an improved version using contrastive learning loss. The performance of the two models is shown in Table 3.

[0148] Table 3. Model performance using different numbers of abnormal driving behavior classes for training

[0149]

[0150] 4. First, initialize the network, identify and mark the channels with smaller scaling factors; then remove these marked channels and their associated weights; finally, fine-tune the pruned network to optimize performance.

[0151] In order to maintain the consistency of the network structure, the two BN layers in the bottleneck involved in the addition operation and the BN layer in the upstream convolution module should be included in the list of unpruned layers. These three BN layers are connected in series. If the number of channels in them changes, the addition operation will not be performed correctly. Taking the first C3 module of the backbone as an example, the unpruned part is represented in green, and the pruned part is represented in orange. Figure 5 shown.

[0152] The left convolution module needs to be pruned because Concat is used instead of addition. The coefficient of the hyperparameter γ decreases as the number of training rounds increases. When performing sparse training, the coefficient is 0.001, the batch_size is 128 (it remains unchanged in the following few times), and 50 epochs are run.

[0153] The number of channels of the Yolov5s model before and after pruning is as follows Figure 6 As shown in the figure, since the number of shallow channels in the network is small, the number of model channels before and after pruning is intuitively compared. Figure 6 A logarithmic coordinate axis is used as the vertical axis.

[0154] according to Figure 6 It can be seen that the channel pruning is less in the shallower layers of the network, and more in the deeper layers.

[0155] The accuracy of the model after pruning is slightly reduced. The model weights can be fine-tuned to adapt to the new network model and thus restore the original accuracy to a certain extent. The decrease in accuracy may be caused by changes in the network structure, but after pruning, the redundant structures are removed and the retained weights are sufficient to meet the reasoning process, so that the model performance is quickly restored. This process shows that the pruning strategy of this study is effective.

[0156] The number of channels, number of parameters, inference speed, and weight file size of the pruned model are shown in Table 4.

[0157] Table 4 Number of channels before and after pruning

[0158] Model Number of channels Parameter quantity Accuracy Inference speed (ms) Weight file size (MB) Before pruning 9504 4185290 97.9 0.6 10.5 After pruning 3565 2487258 97.7 0.7 6.24 After fine-tuning 3565 2487258 98.1 0.7 6.26

[0159] After pruning, the number of channels decreased by 62.49%, and the BN layer of the deeper network was pruned a lot, indicating that the deep part of the network includes a large number of redundant parameters. The accuracy of the model did not decrease significantly after this part of the data was deleted. However, the inference speed of the model increased by 0.1ms after pruning. This may be because the number of channels of the model after pruning is no longer 2^n. This type of model is not conducive to GPU parallel computing optimization, but on the edge inference platform, the NPU processor does not support a large number of parallel computing, so it will not affect the final actual deployment. After fine-tuning training, the accuracy of the model increased by 0.2 compared with the original model accuracy. This may be because after pruning, the redundant parts of the network are removed and the main feature extraction part is retained. After fine-tuning, the parameters of this part of the network are further optimized, so the model performance is slightly improved.

[0160] When the model is actually deployed, the weights no longer need to be updated, and the weights can be converted from FP32 to UINT8 precision through calculation mapping for easy storage and calculation. At present, the quantization method in this study is linear quantization, which processes data through a simple linear mapping. The following is the mapping formula between floating-point real numbers (r) and fixed-point integers (q):

[0161] r=S(qZ)

[0162]

[0163] S is the scaling ratio, which indicates the proportional relationship between real numbers and integers, and Z is the zero point, which indicates the integer corresponding to the floating point real number 0 after mapping. Their calculation methods are as follows:

[0164]

[0165] Among them, r max , r min are the maximum and minimum values ​​of the floating point real number r, q max ,q min are the maximum and minimum values ​​of the fixed-point integer q, respectively.

[0166] The experiment compares the accuracy of the model under four different parameter precisions: original precision, floating point half precision, INT8 precision and UINT8 precision. The specific values ​​are shown in Table 5.

[0167] Table 5 Accuracy of driver behavior classification model after quantization with different precisions

[0168] Model FP32-ACC FP16-ACC INT8-ACC UINT8-ACC Effecientnet_b0 0.978 0.973 0.958 0.963 Effecientnet_b1 0.982 0.972 0.962 0.962 Effecientnet_b2 0.979 0.969 0.960 0.961 Effecientnet_b3 0.980 0.971 0.962 0.961 Resnet-18 0.980 0.978 0.971 0.971 ResNet-34 0.977 0.970 0.967 0.965 ResNet-50 0.976 0.966 0.958 0.959 ResNet-101 0.976 0.971 0.968 0.966 YOLOv5n-CLS 0.977 0.976 0.975 0.975 YOLOv5s-CLS 0.979 0.978 0.976 0.977

[0169] From the data in Table 5, we can see that different precision quantization will lead to reduced model accuracy, but for the EfficientNet series: the accuracy of the model generally decreases under the precision quantization from FP32 to FP16 and then to INT8 and UINT8. From EfficientNet_b0 to EfficientNet_b3, the accuracy decreases as the complexity of the model increases, which indicates that the impact of quantization may be more significant for more complex models. For ResNet-18 and ResNet-34, the accuracy decreases under different precision quantization, but the overall trend is relatively consistent. This may be because the number of layers of these two models is shallow, and the decrease in model accuracy caused by the quantization of model weight parameters is relatively consistent. For ResNet-50 and ResNet-101, the accuracy remains stable or slightly decreases under the precision quantization from FP32 to FP16, but decreases significantly under INT8 and UINT8. This shows that too deep networks are greatly affected by quantization, which may be because the number of parameters of too deep networks is large, and the errors caused by multi-layer quantization accumulate layer by layer, resulting in a significant decrease in accuracy. For the YOLOv5 series: the accuracy of the YOLOv5n-CLS and YOLOv5s-CLS models under different precision quantization changes is relatively small, and the difference is not obvious. This may be because the model structure of the YOLOv5-CLS series is relatively simple, without too many branch structures and the number of network layers is relatively shallow, so the model accuracy changes little after quantization, which also proves the correctness of the present invention using the YOLOv5 model for deployment.

[0170] The edge computing platform used in the present invention includes an RV1126 processor, a 5.5-inch touch screen, and an IMX335 camera, wherein the RV1126 processor is a quad-core ARMCortex-A7 and RISC-V MCU, including 2GBDDR4 memory, 8GBEMMC storage and 2.0TopsNPU, 14MISP, and supports 3 cameras to input 4KH.264 / H.265 video encoding and decoding at the same time. The touch screen is a 5.5-inch MIPI LCD touch screen, supports 720*1280 resolution, and uses a 4lanesMIPI DSI interface for data transmission. The camera is an MCIMX335 camera module, which supports 500W pixels, 2K resolution, fixed focal length 4.4MM, aperture 2.0, field of view angle 70°, and uses a 4lanesMIPI S interface to transmit data.

[0171] After initializing the ISP service, initialize the RKmedia system, create and bind the video link. In this study, two links are created, one for model inference and one for image streaming to the display. Figure 7 shown.

[0172] When using RKmedia, please note that the image width needs to be aligned to 16 bits and the height needs to be aligned to 8 bits. If it is not aligned, use 0 to fill it. Otherwise, it may cause memory misalignment and image distortion. When using the IMX335 camera, the maximum resolution of ISPP is 2592x1944. In addition, the data link needs to satisfy Sensor = MIPI_DPHY ≥ ISP input ≥ ISP output.

[0173] In order to achieve high-performance model reasoning on the edge computing platform, this part of the reasoning code is written in C++. Compared with the Python version, the C++ reasoning code is more efficient and concise. The code logic is as follows Figure 8 The specific contents are as follows:

[0174] Model loading and initialization:

[0175] The code loads and initializes the model by calling the rknn_init() function. This function accepts the model file path and model size as parameters and returns a rknn_context object, which is used to manage the context information of the model, ensuring that the model can be correctly loaded and used during inference.

[0176] Set input and output information:

[0177] Use the rknn_query() function to query the model's input and output information, including the number, size, data type, and other detailed information of the input and output. This information is crucial for subsequent data preprocessing and postprocessing to ensure that the input data is consistent with the format and size expected by the model, and that the output results of the model can be correctly parsed.

[0178] Input data preprocessing:

[0179] After obtaining the raw image data from the camera, necessary preprocessing operations are performed to meet the input requirements of the model. Preprocessing operations include image scaling and format conversion. The image is converted according to the model input requirements queried in the previous step through RGA to ensure that the input data matches the input format and size expected by the model.

[0180] Model Reasoning:

[0181] The model inference process is executed by calling the rknn_run() function. During the inference process, the preprocessed image data is input into the model, and the model performs calculation inference and generates inference results. This step is the core of the entire inference process. In this step, the program creates a new thread for model inference, which is performed in parallel with data preprocessing, thereby improving inference efficiency.

[0182] Post-processing of inference results:

[0183] Once the model inference is complete, the code gets the inference results from the model output and performs post-processing operations. Post-processing operations may include parsing the classification results, overlaying the inference results on the final result image using OpenCV, and streaming to the RKmediaVO module to display the results on the display through VENC.

[0184] Release resources:

[0185] After the entire inference process is completed, the code releases related resources, including model context, image buffer, etc., ensuring the effective use of system resources and avoiding problems such as resource leakage and memory overflow.

[0186] In this part, a real-life vehicle was used for simulation testing. Since the experiment included distracted driving behavior, in order to avoid traffic accidents, the vehicle was simulated in a stationary environment and recorded using dual perspectives, such as Fig. 9 As shown, the left side is the current image of the driver, and the right side is the real-time image and inference result obtained by the edge computing platform. The inference result is displayed in the upper left corner of the display.

[0187] The model performance is shown in Table 6:

[0188] Table 6 Model inference speed before and after precompilation

[0189] Model Before precompilation (FPS) After precompilation (FPS) Original Model 8 14 After pruning and quantization 27 45

[0190] The performance of the inference model after pruning and quantization on RV1126 is shown in Table 7.

[0191] Table 7 Model occupancy before and after optimization

[0192]

[0193] The real-time inference results of the model are as follows: Fig.10 shown.

[0194] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0195] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing, characterized in that: The following steps are involved: Get the target dataset; Build the Yolov5 detection model and pre-train the Yolov5 detection model; The pre-trained Yolov5 detection model is changed to the Yolov5 classification model, and it is re-trained on the target dataset to obtain the trained PT model; Improve the PT model based on contrastive learning method; The improved PT model is pruned on the BN layer, and the PT model is quantized after pruning; Convert the PT model to ONNX model first and then to RKNN model and precompile; Finally, the precompiled RKNN model is used for real-time reasoning on the edge computing platform.

2. The method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing according to claim 1, characterized in that: The edge computing platform is the RV1126 edge computing platform.

3. The method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing according to claim 1, characterized in that: The specific content of pre-training the Yolov5 detection model is: The Yolov5 detection model is used to classify the driver's behavior based on the driver's behavior image. The Backbone of the Yolov5 detection model is used for image feature extraction and classification. The neck and head structures are deleted and the last SPFF layer of the Yolov5 detection model is changed to a classification layer. Divide the target dataset into training set, validation set and test set to ensure that the training set, validation set and test set are independent and random. Under each subset, store the images in folders according to categories.

4. The method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing according to claim 3, characterized in that: The specific contents of improving the PT model based on the contrastive learning method are as follows: Conduct a simulated open set design, use the target dataset and different numbers of distracted driving behaviors in the training set for training, and use all distracted behavior categories during recognition, so as to obtain categories of distracted driving behaviors that have not been seen in the test set; The closed-set classification task is used as the main task to form a classification loss constraint, and a discriminative learning task is added as an auxiliary task to form a contrast loss constraint. The main task and the auxiliary task share a backbone network and use the same encoding layer to convert the image into a feature map. The model input size is 480×480, the batch size is 256, the SGD optimizer is used, the learning rate is 1e×10^-4, and the temperature coefficient of the contrast loss is 0.25; It is divided into two classification tasks: normal driving and abnormal driving. AUC, accuracy and F1 score are used as evaluation indicators. The baseline model is Yolov5s-CLS that only uses the cross entropy loss function, and SCL refers to an improved version using contrastive learning loss.

5. The method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing according to claim 4, characterized in that: The classification loss constraint uses the cross entropy loss function: Where M is the number of categories, N is the batch size, and L is the i is the negative of the sum of the products of the sign function and the logarithm of the probability, y ic is the symbolic function, p ic is the predicted probability that the observed sample i belongs to category c; Contrastive loss constraint uses contrastive loss function, uses sample label information to participate in training, uses supervised learning, positive samples include samples with the same label, negative samples include samples with different labels, and uses supervised contrastive loss to align the embedding representation obtained by the feature extraction network with the label. The formula is as follows: Where N is the batch size, L i for P(i) represents the i For samples with the same label, · is the inner product, Normalize the feature vector of the positive sample. Normalize the feature vector of samples with different labels from the positive samples, is the normalized feature vector of the augmented version of the positive sample, τ is the contrastive learning temperature parameter, x k For x i' Different augmented versions, x i' For x i An expanded version of .

6. The method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing according to claim 1, characterized in that: The specific contents of BN layer pruning of the improved PT model are as follows: First, initialize the network, identify and mark the channel with the smallest scaling factor, and apply L1 regularization to each scaling factor in the channel for adjustment. Each scaling factor corresponds to a specific convolution channel. The mathematical expression of channel pruning is: Where W is the trainable weight, l is the normal training loss of CNN, f is the training output, x is the training input, y is the training target, λ is the hyperparameter, and g(γ) is the sparsification penalty for the scale factor, which is |γ|, i.e., the L1 norm. The following operations are performed in the pruning operation of the BN layer: in, is the normalized input, z in is the input of the BN layer, μ β is the average of the input activations, is the standard deviation of the input activation, ∈ is the bias, z out is the output of the BN layer, γ is the affine change ratio, and β is the affine transformation deviation; The coefficient of the hyperparameter λ decreases as the number of training times increases. When performing sparse training, the coefficient is 0.001, the batch_size is 128, and 50 epochs are run; Prune the PT model according to the scaling factor and prune the BN layer with the smallest scaling factor; The pruned PT model is fine-tuned using the original dataset.

7. The method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing according to claim 1, characterized in that: The specific contents of quantization processing of PT model after pruning are as follows: After pruning, the PT model is linearly quantized. The formula for linear quantization is as follows: r=S(qZ) Among them, r is a floating point real number, q is a fixed point integer, S is the scaling ratio, which indicates the proportional relationship between real numbers and integers, and Z is the zero point, which indicates the integer corresponding to the floating point real number 0 after mapping. The calculation method is as follows: Among them, r max , r min are the maximum and minimum values ​​of the floating point real number r, q max ,q min are the maximum and minimum values ​​of the fixed-point integer q respectively; The experiment compares the accuracy under four different parameter precisions: original precision, floating-point half precision, INT8 precision, and UINT8 precision, and analyzes the accuracy loss of different classification models under four quantization precisions.

8. The method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing according to claim 1, characterized in that: The specific contents of converting the PT model into an ONNX model and then into an RKNN model and precompiling are as follows: Save the trained model as a pt model with a network structure, and convert it into an ONNX model for inference through the ONNX library; Convert the ONNX model into an RKNN model that can be inferred on the edge computing platform; Create a data set for model quantization and randomly select training set images for fine-tuning the model; Finally, the model is converted from the original FP32 precision model to the RKNN model with UINT8 precision quantization. Precompile the RKNN model through RKtoolkit to increase the speed of model loading and switching and improve real-time performance.

9. The method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing according to claim 1, characterized in that: Finally, the specific content of using the precompiled RKNN model for real-time reasoning on the edge computing platform is: Insert the IMX335 camera into the MIPI-CSI interface, modify the development board device tree, turn on the node for image code stream transmission, and transmit image data; After optimizing the image according to the IQ file, ISP, and ISPP, it is sent to the VI node; The image is divided into two links and sent to the RGA module for format conversion, rotation and cropping. In the first link, the image is converted from NV12 format to RGB888 format and cropped to 480*480 size; in the second link, the image is converted from NV12 format to RGB888 format, rotated 270° and cropped to 1280*720 size; Print the inference results to the RGA cropped image frame through OpenCV, and print the corresponding driving behavior according to the sequence number of the inference results; The image frame with the inference result is sent to the display screen of the MIPI-DSI interface for result display.

10. The method for identifying driver distracted driving behavior based on contrastive learning and real-time edge computing according to claim 9, characterized in that: Driving behaviors include: normal driving, typing with the left hand, typing with the right hand, making a phone call with the left hand, making a phone call with the right hand, operating the center console, drinking water, getting something, putting on makeup, and talking to passengers.

Images