Lightweight optimization method for open-pit mine truck driver driving behavior detection model based on OpenVino

Abstract

The present invention relates to a lightweight optimization method for an OpenVino-based opencast truck driver driving behavior detection model, comprising the following steps: S1: constructing an opencast truck driver image database; S2: classifying and calibrating the driver driving behavior images in the opencast truck driver image database to construct a training set; S3: constructing an opencast truck driver driving behavior detection model; S4: lightweight optimization training of the opencast truck driver driving behavior detection model based on the training set using an optimization iterative algorithm and OpenVino tools; and S5: obtaining real-time driving videos of the truck drivers to be tested, feeding them into the detection model for detection, obtaining the types of illegal driving behaviors of the opencast truck drivers, and providing voice and text warnings for illegal driving behaviors. The real-time face target detection model obtained using the present invention has a high volume compression ratio, consumes little physical space and computational memory, has a fast inference speed, high accuracy, is simple to use, and is easy to maintain.

Description

Technical Field

[0001] The present invention belongs to the technical field of open pit truck driving safety monitoring, and in particular relates to a lightweight optimization method for an open pit truck driver driving behavior detection model based on OpenVino. Background Art

[0002] The booming development of facial recognition technology has enabled machine vision to gradually replace traditional manual methods for driver fatigue detection, becoming mainstream. Because it involves real-time image and video processing, it typically requires high-end graphics processing modules (GPUs). Current human image recognition technologies rely on high-end GPU-equipped machines, dedicated computers, or cloud computing. Traditional open-pit mine truck dispatching systems utilize embedded mobile devices. Due to technical and price constraints, they are often unable to incorporate high-speed modules such as GPUs and 5G. Consequently, they rely on hardware expansion interfaces. This results in a complex cab with numerous devices, high costs, significant interference in space and wiring, inconvenience, and complex maintenance. The lightweight integration of image processing technology on embedded mobile terminals is a challenge facing developers and users of intelligent terminals in open-pit mines. OpenVINO is a toolkit released in 2019 by Intel, based on its own hardware platform, that accelerates the development of high-performance computer vision and deep learning vision applications. It supports deep learning on hardware accelerators across various Intel platforms and allows for direct heterogeneous execution. Currently, there is little research on lightweight optimization of real-time target detection. The relevant patent found so far is an embedded gateway and its deployment method for flame detection using YOLOv5 and OpenVINO, which was published on April 30, 2021. The invention includes a CPU, VPU, a host computer communication module, an Ethernet interface, an HDMI / VGA video output, a USB expansion interface, RAM, ROM, a protection circuit module, a real-time clock module and a reset circuit module. It uses YOLOv5 and OpenVINO to optimize the embedded gateway for flame detection. Its characteristics are that the embedded network management is optimized by combining hardware and software. Image processing still uses VPU technology, the CPU still uses 64 operation, and the software optimization is limited. The present invention is aimed at the actual situation that the mobile vehicle terminal has a low CPU and no high-end GPU or VPU graphics card. It mainly performs lightweight optimization of target detection models such as driver fatigue driving from the software aspect. Summary of the Invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a lightweight optimization method for the open pit truck driver driving behavior detection model based on OpenVino.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] The present invention provides a lightweight optimization method for an open pit mine truck driver driving behavior detection model based on OpenVino, which is characterized by comprising the following steps:

[0006] S1: Build an image database of open pit truck drivers;

[0007] S2: Classify and calibrate the driver driving behavior images in the open-pit mine truck driver image database to construct a training set;

[0008] S3: Build a driving behavior detection model for open pit mine truck drivers;

[0009] S4: Based on the training set, we use an optimized iterative algorithm that introduces momentum to accelerate model training, and the OpenVino tool that provides an operating environment for model pruning, merging, and quantization to perform lightweight optimization training on the open-pit mine truck driver driving behavior detection model.

[0010] S5: Obtain real-time driving videos of the open-pit mine truck driver to be tested and feed them into the open-pit mine truck driver driving behavior detection model for detection. The types of illegal driving behaviors of the open-pit mine truck driver are obtained, and voice and text warnings are issued for illegal driving behaviors.

[0011] Preferably, the specific steps of step S2 include:

[0012] S2.1: Obtain driving behavior types of open pit mine truck drivers;

[0013] S2.2: Calibrate the open pit truck driver driving images based on their driving behavior types, and set the calibrated labeled images as the training set;

[0014] S2.3: Perform Mosaic data augmentation on the training set, randomly adding noise to the images, cropping, flipping, rotating, scaling, translating, and adjusting brightness, contrast, saturation, and color.

[0015] Preferably, the open-pit mine truck driver driving behavior detection model constructed in S3 is a Yolo-V5 convolutional neural model.

[0016] Preferably, in step S4, lightweight optimization training of the Yolo-V5 convolutional neural model is performed based on the training set and by utilizing an optimization iterative algorithm that introduces momentum acceleration model training and provides an operating environment for model pruning, merging, and quantization, specifically including:

[0017] S4.1: Feed the image database convolutional neural model into the open pit mine truck driver driving behavior detection model;

[0018] S4.2: Use torch.optim.SGDM() to improve the optimization iterative algorithm that introduces momentum to accelerate model training and perform stochastic gradient descent optimization on the convolutional neural model weights:

[0019] torch.optim.SGDM() algorithm formula:

[0020]

[0021] Among them, Momentum∈[0,1) is momentum, η is the learning step size indicating the magnitude of the weight θ update, The weight gradient represents the direction of weight update, and Δθ is the update amount of the model weight θ at each iteration;

[0022] S4.3: Use the loss function to evaluate the Yolo-V5 convolutional neural network model and adjust the parameters based on the evaluation results to obtain the optimized Yolo-V5 convolutional neural network training model;

[0023] S4.4: Convert the optimized Yolo-V5 convolutional neural network training model weights .pt format to ONNX format and input it into the OpenVino environment;

[0024] S4.5: Prune and merge redundant network layers of the optimized Yolo-V5 convolutional neural network trained for open-pit mine truck driver driving behavior detection in the OpenVino environment, which provides an operating environment for model pruning, merging, and quantization. Prune the Dropout operation in the last Detect output layer of the optimized Yolo-V5 convolutional neural network model and merge the model's Conv convolutional layer, Batch Normalization (BN) layer, and ReLU activation function.

[0025] S4.6: Quantize the optimized Yolo-V5 convolutional neural network training model from FP32 to FP16 to compress the model size. The quantization formula is:

[0026]

[0027] in, To round to the nearest integer, the clamp() function formula is:

[0028] clamp(input,input_low,input_high)=min(max(input,input_low),input_high)))

[0029] Among them, input_low and input_range are adjustable parameters.

[0030] Preferably, the illegal driving behaviors in S5 include C0 driving while fatigued with eyes closed, C1 driving while fatigued with yawning, C2 driving while fatigued with head down, C3 not wearing a mask, C4 smoking, C5 making phone calls and C6 eating and drinking.

[0031] The present invention provides a driving behavior detection system for open pit truck drivers, which is characterized by comprising an open pit truck driver video acquisition module, a driving behavior recognition module, a voice and text warning module, and a violation record archiving module.

[0032] The open pit truck driver picture acquisition module is used to acquire open pit truck driver videos;

[0033] The driving behavior recognition module is used to perform behavior recognition on open-pit mine truck driver videos based on the lightweight optimization method for detecting driving behavior of open-pit mine truck drivers based on OpenVino as described in any one of claims 1 to 4, and obtain the driving behavior type of the open-pit mine truck driver;

[0034] When the judgment result of the voice and text warning module is abnormal driving behavior, the voice and text warning module sends out an alarm text and voice signal;

[0035] The violation record archiving module is used to record and store the open pit truck driver ID, violation driving behavior category, picture log, and time information.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] (1) The lightweight optimization method of the open pit truck driver driving behavior detection model based on OpenVino of the present invention has a high volume compression ratio of the real-time target detection model of the face, small model physical space and operation memory occupation, fast inference speed and high accuracy.

[0038] (2) The lightweight optimization method of the open-pit mine truck driver driving behavior detection model based on OpenVino of the present invention can be directly deployed on the embedded truck scheduling intelligent terminal without the need for a dedicated fatigue driving terminal, thus reducing the space occupied by the cab and being simple to use and easy to maintain. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the lightweight optimization method for the open pit truck driver driving behavior detection model based on OpenVino of the present invention;

[0040] Figure 2 This is a graph of the model training loss function for the lightweight optimization method of the open pit truck driver driving behavior detection model based on OpenVino in the present invention;

[0041] Figure 3 This is the network layer pruning diagram of the lightweight optimization method for the open pit truck driver driving behavior detection model based on OpenVino in the present invention;

[0042] Figure 4 This is a network layer merging diagram of the lightweight optimization method for the open-pit mine truck driver driving behavior detection model based on OpenVino in the present invention. DETAILED DESCRIPTION

[0043] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Note that the following embodiments are merely illustrative in nature and the present invention is not intended to limit its applicable objects or uses, and the present invention is not limited to the following embodiments.

[0044] Example

[0045] like Figures 1-4 As shown, the lightweight optimization method of the open pit mine truck driver driving behavior detection model based on OpenVino of the present invention is characterized by comprising the following steps:

[0046] S1: Build an image database of open pit truck drivers;

[0047] S2: Classify and calibrate the driver driving behavior images in the open-pit mine truck driver image database to build a self-developed training set;

[0048] The specific steps of step S2 include:

[0049] S2.1: Obtain driving behavior types of open pit mine truck drivers;

[0050] S2.2: Calibrate the open pit truck driver driving images based on their driving behavior types, and set the calibrated labeled images as the training set;

[0051] S2.3: Perform Mosaic data augmentation on the training set, randomly adding noise to the images, cropping, flipping, rotating, scaling, translating, and adjusting brightness, contrast, saturation, and color.

[0052] This prevents the model from learning irrelevant features and avoids identifying irrelevant elements as features. After multiple training comparisons, the training results after using Mosaic data augmentation are better.

[0053] The open-pit mine truck driver driving behavior detection model constructed in S3 described in this embodiment is a Yolo-V5 convolutional neural model.

[0054] S4: Based on the training set, we use an optimized iterative algorithm that introduces momentum to accelerate model training, and the OpenVino tool that provides an operating environment for model pruning, merging, and quantization to perform lightweight optimization training on the open-pit mine truck driver driving behavior detection model.

[0055] Specifically include:

[0056] S4.1: Feed the image database convolutional neural model into the open pit mine truck driver driving behavior detection model;

[0057] S4.2: Use torch.optim.SGDM() to improve the optimization iterative algorithm that introduces momentum to accelerate model training and provides an algorithm for model pruning, merging, and quantization operation environment to perform stochastic gradient descent optimization on the convolutional neural model weights:

[0058] torch.optim.SGDM() improves the optimization iterative algorithm formula of introducing momentum acceleration model training:

[0059]

[0060] Among them, Momentum∈[0,1) is momentum, η is the learning step size indicating the magnitude of the weight θ update, The weight gradient represents the direction of weight update, and Δθ is the update amount of the model weight θ at each iteration;

[0061] S4.3: Use the loss function to evaluate the Yolo-V5 convolutional neural network model and adjust the parameters based on the evaluation results to obtain the optimized Yolo-V5 convolutional neural network training model;

[0062] The loss function is used to evaluate the model and the parameters are adjusted according to the evaluation results.

[0063] The model training hyperparameters are as follows:

[0064]

[0065] S4.4: Convert the optimized Yolo-V5 convolutional neural network training model weights in .pt format to ONNX format and input it into the OpenVino environment, which provides an operating environment for model pruning, merging, and quantization. S4.5: Prune and merge redundant network layers of the optimized Yolo-V5 convolutional neural network training open-pit mine truck driver driving behavior detection model in the OpenVino environment. Prune the Dropout operation in the last Detect output layer of the optimized Yolo-V5 convolutional neural network model and merge the model's Conv convolutional layer, Batch Normalization (BN) layer, and ReLU activation function.

[0066] S4.5: Prune and merge redundant network layers of the model in the OpenVino environment;

[0067] The pruning operation is to remove the Dropout operation that is useless for inference operations in the Detect output layer of the last layer of the Yolo-V5 model network. The Dropout operation randomly deletes several neurons to reduce the network size, but this works well in the fully connected layer. It has little effect on the convolution results of the Detect layer, because the Detect convolution includes convolution layer, activation layer and pooling layer. The role of the pooling layer is to establish connections with adjacent neurons for learning. After discarding the Dropout operation, the Detect layer can still learn with the adjacent activation network layer. The merging operation is: merging the Yolo-V5 model Conv convolution layer, BN (Batch Normalization) batch normalization layer, and ReLU activation function. In the original Yolo-V5 model, the BN layer is located after the Conv layer. It has the function of normalizing the feature map to avoid overfitting of the model and prevent gradient explosion and disappearance of the weights. Although the BN layer has a very positive effect on model reasoning, when the data is propagated forward in the network, each additional layer of the network will increase the model operation and reasoning time. Therefore, merging the BN layer with the previous Conv layer is equivalent to modifying the convolution kernel calculation of the Conv layer, but does not increase the amount of convolution operation, and saves the time for the entire BN layer operation; merging the ReLU activation function with these two layers can perform sparse matrix calculations during convolution operations, extract neurons that are representative of reasoning, enhance the generalization ability of the model, improve the operation speed, and reduce the reasoning time;

[0068] S4.6: Quantize the optimized Yolo-V5 convolutional neural network training model for detecting driving behavior of open-pit mine truck drivers from FP32 to FP16 to compress the model size. The quantization formula is:

[0069]

[0070] in, To round to the nearest integer, the clamp() function formula is:

[0071] clamp(input,input_low,input_high)=min(max(input,input_low),input_high)))

[0072] Among them, input_low and input_range are adjustable parameters.

[0073] S5: Obtain real-time driving video of the open-pit mine truck driver to be tested, and send it to the open-pit mine truck driver driving behavior detection model for detection, obtain the type of illegal driving behavior of the open-pit mine truck driver, and issue voice and text warnings for illegal driving behavior.

[0074] In this embodiment, the types of illegal driving behaviors include C0 driving while tired with eyes closed, C1 driving while tired with yawning, C2 driving while tired with head down, C3 not wearing a mask, C4 smoking, C5 making phone calls, and C6 eating and drinking. The classification results are comprehensive and independent.

[0075] The present invention also provides a driving behavior detection system for open pit truck drivers, including an open pit truck driver video acquisition module, a driving behavior recognition module, a voice and text warning module, and a violation record archiving module.

[0076] The open-pit mine truck driver video acquisition module is used to acquire open-pit mine truck driver videos;

[0077] The driving behavior recognition module is used to perform behavior recognition on open-pit mine truck driver videos based on the lightweight optimization method for detecting driving behavior of open-pit mine truck drivers based on OpenVino as described in any one of claims 1 to 4, and obtain the driving behavior type of the open-pit mine truck driver;

[0078] When the judgment result of the voice and text warning module is abnormal driving behavior, the voice and text warning module sends out an alarm text and voice signal;

[0079] The violation record archiving module is used to record and store the open pit truck driver ID, violation driving behavior category, picture log, and time information.

[0080] The above embodiments are merely examples and do not limit the scope of the present invention. These embodiments can be implemented in various other ways, and various omissions, replacements, and changes can be made without departing from the technical concept of the present invention.

Claims

1. A lightweight optimization method for open pit truck driver driving behavior detection model based on OpenVino, characterized by: The following steps are involved: S1: Build an image database of open pit truck drivers; S2: Classify and calibrate the driver driving behavior images in the open-pit mine truck driver image database to construct a training set; The specific steps of step S2 include: S2.1: Obtain driving behavior types of open pit mine truck drivers; S2.2: Calibrate the open pit truck driver driving images based on their driving behavior types, and set the calibrated labeled images as the training set; S2.3: Perform mosaic data augmentation on the training set, randomly add noise to the images, crop, flip, rotate, scale, translate, and adjust brightness, contrast, saturation, and color. S3: Build a driving behavior detection model for open-pit mine truck drivers. S4: Based on the training set, we use an optimized iterative algorithm that introduces momentum to accelerate model training, and the OpenVino tool that provides an operating environment for model pruning, merging, and quantization to perform lightweight optimization training on the open-pit mine truck driver driving behavior detection model. In step S4, lightweight optimization training of the open pit truck driver driving behavior detection model is performed based on the training set and using an improved optimization iterative algorithm that introduces momentum acceleration model training and the OpenVino tool that provides an operating environment for model pruning, merging, and quantization. Specifically, the training includes: S4.1: Feed the image database convolutional neural model into the open-pit mine truck driver driving behavior detection model; S4.2: Use the improved optimization iterative algorithm that introduces momentum acceleration model training, torch.optim.SGDM() algorithm, to perform stochastic gradient descent optimization on the convolutional neural model weights: torch.optim.SGDM() algorithm formula: θ=θ-Δθ·Momentum-η·▽ θ J(θ) Among them, Momentum∈[0,1) is momentum, η is the learning step size indicating the magnitude of the weight θ update, ▽ θ J(θ) is the weight gradient indicating the direction of weight update, and Δθ is the update amount of the model weight θ at each iteration; S4.3: Use the loss function to evaluate the Yolo-V5 convolutional neural network model and adjust the parameters based on the evaluation results to obtain the optimized Yolo-V5 convolutional neural network training model; S4.4: Convert the optimized Yolo-V5 convolutional neural network training model weights .pt format to ONNX format and input it into the OpenVino environment; S4.5: Prune and merge redundant network layers of the optimized Yolo-V5 convolutional neural network trained for the open pit truck driver driving behavior detection model in the OpenVino environment. Prune the Dropout operation in the last Detect output layer of the optimized Yolo-V5 convolutional neural network model and merge the model's Conv convolutional layer, Batch Normalization (BN) layer, and ReLU activation function. S4.6: Quantize the optimized Yolo-V5 convolutional neural network training model from FP32 to FP16 to compress the model size. The quantization formula is: in, To round to the nearest integer, the clamp() function formula is: clamp(input,input_low,input_high)=min(max(input,input_low),input_high))) Among them, input_low is an adjustable parameter; S5: Obtain real-time driving videos of the open-pit mine truck driver to be tested and feed them into the open-pit mine truck driver driving behavior detection model for detection. The types of illegal driving behaviors of the open-pit mine truck driver are obtained, and voice and text warnings are issued for illegal driving behaviors.

2. The lightweight optimization method for open pit truck driver driving behavior detection model based on OpenVino according to claim 1 is characterized in that: The open-pit mine truck driver driving behavior detection model constructed in S3 is a Yolo-V5 convolutional neural network model.

3. The lightweight optimization method for open pit truck driver driving behavior detection model based on OpenVino according to claim 1 is characterized in that: The illegal driving behaviors in S5 include C0 driving while tired with eyes closed, C1 driving while tired with yawning, C2 driving while tired with head down, C3 not wearing a mask, C4 smoking, C5 making phone calls and C6 eating and drinking.

4. A driving behavior detection system for open pit truck drivers, characterized in that: It includes open-pit mine truck driver video acquisition module, driving behavior recognition module, voice and text warning module, and violation record archiving module. The open pit truck driver video acquisition module is used to acquire the open pit truck driver's picture; The driving behavior recognition module is used to perform behavior recognition on the open-pit mine truck driver image based on the lightweight optimization method of the open-pit mine truck driver driving behavior detection model based on OpenVino as described in any one of claims 1 to 3, and obtain the driving behavior type of the open-pit mine truck driver; When the judgment result of the voice and text warning module is abnormal driving behavior, the voice and text warning module sends out an alarm text and voice signal; The violation record archiving module is used to record and store the open pit truck driver ID, violation driving behavior category, picture log, and time information.

Images