A driver gesture recognition method and device, computer equipment and storage medium

Abstract

The application provides a driver gesture recognition method and device, computer equipment and a storage medium. The method comprises the following steps: constructing an improved YOLOv5 network for monitoring the driving state of a locomotive driver, wherein the improved YOLOv5 network comprises the following steps: replacing the C3 module in the original network with a GhostBottleneck module; replacing the convolution operation with a depth separable convolution operation; replacing the three-scale prediction mechanism with a two-scale prediction mechanism; deleting the Focus module; deleting the spatial pyramid pooling module; training the improved YOLOv5 network; and inputting the image of the locomotive driver gesture to be detected into the trained improved YOLOv5 network for driving state monitoring. The application reduces the calculation amount while ensuring the driver gesture recognition performance, thereby meeting the real-time requirement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a driver gesture recognition method and device, computer equipment and storage medium. BACKGROUND

[0002] Driver gesture recognition can be used for traffic signal interaction, for example, the driver indicates stop, left turn, right turn and other actions by gestures, and the system can recognize and make corresponding traffic signals, which helps to reduce the distraction of the driver when operating the vehicle and improve traffic safety. Driver gesture recognition is part of the development of intelligent transportation systems, which can promote the interconnection between vehicles and transportation infrastructure, and through gesture interaction, vehicles can communicate more intelligently with surrounding vehicles, traffic lights, road facilities, etc.

[0003] The YOLO series is the most widely used single-stage target detection algorithm at present, which can be applied to driver gesture recognition. Its model has the characteristics of simplicity and speed, can predict the entire image and give all detection results at once. The target detection model of the YOLO series has become more and more powerful, from YOLOv1 to YOLOv7, among which YOLOv5 is a basic model, which can be more simply and conveniently deployed in the cab of the train to detect the prescribed gestures of the locomotive driver.

[0004] Although YOLOv5 is an advanced target detection algorithm, its large amount of calculation shows certain limitations on some devices with small memory, such as when recognizing driver gestures in real time, the running time becomes longer due to the large amount of calculation, which cannot meet the real-time requirements. SUMMARY

[0005] The present application provides a driver gesture recognition method, device, computer equipment and storage medium, which can solve the technical problem of large amount of calculation when using YOLOv5 to recognize driver gestures in the prior art.

[0006] The present application provides a driver gesture recognition method, comprising:

[0007] An improved YOLOv5 network including a backbone network Backbone, a neck module Neck and a detection module Output connected in sequence is constructed, and the improved YOLOv5 network includes:

[0008] Replace the C3 module in the original YOLOv5 network with the GhostBottleneck module, replace the convolution operation in the original YOLOv5 network with the depth separable convolution operation, delete the focus module and the spatial pyramid pooling module SPP of the backbone network Backbone in the original YOLOv5 network, and replace the three-scale prediction mechanism of the detection module Output in the original YOLOv5 network with a double-scale prediction mechanism;

[0009] The driver gesture to be recognized is input into the improved YOLOv5 network, the backbone network Backbone obtains lightweight features, the neck module Neck fuses the lightweight features to obtain lightweight fusion features of different sizes, and the detection module Output identifies the lightweight fusion features of different sizes to output the recognition result of the driver gesture image to be recognized.

[0010] Further, the GhostBottleneck module includes parallel Ghostconv-DWConv-Ghostconv branches and DWConv-CBS branches.

[0011] The Ghostconv-DWConv-Ghostconv branches and the DWConv-CBS branches are simultaneously connected with the output end of the previous layer, and two new outputs are obtained after the output of the previous layer is processed through the Ghostconv-DWConv-Ghostconv branches and the DWConv-CBS branches, and the two new outputs are fused to serve as the output of the GhostBottleneck layer.

[0012] Further, the depth separable convolution operation includes:

[0013] The channel-by-channel convolution uses a convolution kernel with the same number of channels as the input image to perform a convolution operation, and outputs a feature map with the same number of channels as the input image.

[0014] The point-by-point convolution uses a 1*1*n convolution kernel to perform a convolution operation on the feature map output by the channel-by-channel convolution, where n is the number of channels of the convolution kernel.

[0015] Further, the replacement of the three-scale prediction mechanism with the double-scale prediction mechanism includes:

[0016] The detection layer of the largest scale in the detection module Output of the original YOLOv5 network is deleted, and the upsampling and downsampling operations corresponding to the detection layer of the largest scale are also deleted.

[0017] Further, the improved YOLOv5 network comprises a backbone network Backbone, a neck module Neck and a detection module Output connected in turn, wherein:

[0018] The backbone network Backbone is used for generating lightweight features, comprising an input layer, a CBS layer connected to the output end of the input layer, three stacked DWConv-GhostBottleneck layers connected to the output end of the CBS layer, and a first DWConv layer connected to the output end of the third DWConv-GhostBottleneck.

[0019] The neck module Neck is used for fusing lightweight features of different scales and is an intermediate connection part between the Backbone and the Output, comprising a second DWConv layer connected to the output end of the first DWConv layer, an Upsample up-sampling layer connected to the output end of the second DWConv layer, a first concat layer connected to the output end of the Upsample up-sampling layer and the output end of the third DWConv-GhostBottleneck, a first C3_F layer connected to the output end of the first concat layer, a third DWConv layer connected to the output end of the first C3_F layer, a second concat layer connected to the output end of the third DWConv layer and the output end of the second DWConv layer, and a second C3_F layer connected to the output end of the second concat layer.

[0020] The detection module Output is used for generating a bounding box and a class probability of a driver gesture to be recognized, comprising two output different scale conv2d layers and a YOLO layer connected to the output ends of the two conv2d layers.

[0021] Further, the loss function used in the training process of the improved YOLOv5 network is a CIOU loss function, and the CIOU loss function is represented as:

[0022]

[0023]

[0024] Wherein, A represents the frame corresponding to the prediction set of the loss function, B represents the frame corresponding to the real set, Union(A, B) represents the union set of A and B, Intersection(A, B) represents the intersection set of A and B, IOU represents the intersection union ratio of the prediction set and the real set of the loss function; c is the diagonal distance of the overlapping region of the frame corresponding to the prediction set of the loss function and the frame corresponding to the real set; p 2 (b,b gt ) represents the Euclidean distance between the center points of the frame corresponding to the prediction set of the loss function and the frame corresponding to the real set.

[0025] Further, the number of training times in the improved YOLOv5 network training process is set to 200 epochs, the weight decay weight_decay is set to 0.0005, the momentum momentum is set to 0.937, the initial learning rate is set to 0.01, and the optimizer adopts the stochastic gradient descent method SGD.

[0026] A driver gesture recognition device comprises:

[0027] A model establishing module is configured to:

[0028] An improved YOLOv5 network comprising a backbone network Backbone, a neck module Neck and a detection module Output connected in sequence is constructed, and the improved YOLOv5 network comprises:

[0029] The C3 module in the original YOLOv5 network is replaced by a GhostBottleneck module, the convolution operation in the original YOLOv5 network is replaced by a depth separable convolution operation, the focus module and the spatial pyramid pooling module SPP of the backbone network Backbone in the original YOLOv5 network are deleted, and the three-scale prediction mechanism of the detection module Output in the original YOLOv5 network is replaced by a double-scale prediction mechanism;

[0030] A recognition module is configured to:

[0031] The driver gesture to be recognized is input into the improved YOLOv5 network, the backbone network Backbone obtains lightweight features, the neck module Neck fuses the lightweight features to obtain lightweight fusion features of different sizes, and the detection module Output recognizes the lightweight fusion features of different sizes to output a recognition result of the driver gesture image to be recognized.

[0032] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the driver gesture recognition method.

[0033] A computer device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor implements the driver gesture recognition method when executing the program.

[0034] The present application provides a driver gesture recognition method, device, computer equipment and storage medium, compared with the prior art, the beneficial effects are as follows:

[0035] The present invention simplifies the YOLOv5 network structure by deleting the Focus layer and the spatial pyramid pooling layer SPP in the backbone of the original YOLOv5 network and replacing the three-scale prediction mechanism with a two-scale prediction mechanism, thereby reducing the amount of calculation by reducing the number of parameters; and replaces the C3 layer in the backbone and Neck of the original YOLOv5 network with GhostBottleneck and the convolution operation with a depthwise separable convolution operation to reduce the amount of calculation, thereby reducing not only the number of network parameters but also the influence of feature geometry changes, thereby improving the feature extraction performance of the gesture to be recognized. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0037] In the attached figure:

[0038] Figure 1 This is the original YOLOv5 network structure provided in this manual. (a) is the original YOLOv5 overall structure, (b) is the CBS structure diagram, (c) is the SPP structure diagram, and (d) is the Focus structure diagram.

[0039] Figure 2 This is a schematic diagram of the anchor frames provided in this manual, where A represents the predicted set and B represents the true set;

[0040] Figure 3 This is the Ghost network schematic provided in this manual;

[0041] Figure 4 This is the GhostBottleneck structure diagram provided in this manual;

[0042] Figure 5 This is a modified area of ​​the backbone provided by this manual;

[0043] Figure 6 This is the process diagram of traditional convolution provided in this manual;

[0044] Figure 7 This is a diagram of the depthwise separable convolution process provided in this specification;

[0045] Figure 8 This is the Focus operation diagram provided in this manual;

[0046] Figure 9 This is the SPP structural principle diagram provided in this manual;

[0047] Figure 10is the improved YOLOv5 structure diagram provided in the specification;

[0048] Figure 11 is the comparison diagram of average precision values before and after improvement provided in the specification. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the application will be clearly and completely described in conjunction with the drawings in the embodiments of the application. It should be understood that the protection scope of the application is not limited by the specific embodiments.

[0050] Embodiments

[0051] A driver gesture recognition method specifically includes the following steps:

[0052] Step 1, establishment and data enhancement of locomotive driver gesture dataset

[0053] In the target detection task, the model needs a large number of pictures to train and learn the target features. The samples need to have sufficient complexity and diversity, so that the network has strong generalization ability. Therefore, the number and quality of the dataset play an important role in the target detection task. In the actual data collection process, it is usually difficult to cover all scenarios, and data acquisition requires a large amount of cost. Therefore, for small sample datasets, data enhancement is an indispensable part. Different gesture pictures under different light intensities, different angles and different backgrounds are taken by an image acquisition device, and the pictures are saved as jpg format as the dataset.

[0054] Step 2, lightweight detection algorithm based on L-YOLOv5

[0055] Step 2.1, network structure of YOLOv5

[0056] YOLOv5 algorithm is the fifth YOLO algorithm released by Ultralytics LLC team. YOLOv5 includes four parts of input (Input), backbone (Backbone), neck (Neck) and output (Output). The model structure is as shown in Figure 1 The neck of the feature fusion layer adopts the structure of PAN+FPN, which fuses different detection layers and main feature layers. FPN only enhances the transmission of semantic information, but the ability to transmit low-level positioning information is not strong. PAN has strong positioning information transmission. In the prediction aspect, this module uses GIoU (Generalized Intersection over Union) to calculate the loss value of the boundary box.

[0057] Mosaic data augmentation can randomly scale, crop and arrange four photos at a time. Finally, the processed photos are randomly spliced. This can greatly enrich the number of data sets and make the performance of the entire network better. The previous YOLO series all set the initial anchor box size in advance, and based on the set value, the final prediction box is obtained by correction and adjustment. However, this method has poor applicability and can cause large errors. The YOLOv5 series uses adaptive calculation for anchor boxes. During training, the best anchor box value can be calculated according to the characteristics of the data set. Adaptive image scaling is also one of the training techniques of YOLOv5. The previous network model will scale the image first. When encountering a rectangular image with different length and width, it will be filled with black edges after scaling, and then the size will be unified before being sent to the network model. This will greatly slow down the detection speed. In the YOLOv5 network, the least black edge will be filled according to the scaling of the original picture. This simple improvement can improve the inference speed by about 37%.

[0058] The specific calculation steps are as follows:

[0059] Step 1, calculate the scaling ratio.

[0060]

[0061] Step 2, calculate the size of the scaled image (use the smallest ratio).

[0062]

[0063] Step 3, calculate the black edge padding number (finally divide the padding black edge number by 2, and give it to both ends of the picture)

[0064]

[0065] (1) Backbone: The backbone network part mainly includes Conv structure, C3 structure, Focus module and SPP module. Among them, the Focus module converts the picture from three channels to 12 channels under the premise of not losing picture information, reduces the size of the original image, and increases the number of channels. The C3 module refers to the CSPBottleneck module containing three convolution operations, which solves the problem of excessive calculation in inference from the perspective of model structure design. The SPP module, also known as the spatial pyramid pooling, aims to perform feature fusion, and uses convolution kernels with consistent steps but different sizes to achieve it.

[0066] (2) Neck: The purpose of the Neck module is to better detect targets of different scales. In the field of deep learning, feature layers at different heights contain different information focuses and each has its own advantages. High-level feature layers contain more semantic information but less positional information, while low-level feature layers contain more positional information but less semantic information. In the Neck part, upsampling and downsampling operations are used to cleverly combine high-level and low-level features. The combination of continuous upsampling operations is called the FPN structure. The combination of continuous downsampling structures is called the PAN structure. The combination of the two greatly improves the feature fusion capability of the entire network.

[0067] (3) Output: The most commonly used loss function at the output is the IOU loss function, which is as follows:

[0068]

[0069] Where: IOU represents the intersection-over-union ratio of the loss function prediction set and the true set, Figure 2 This is a diagram of the intersection and union of sets. Box A represents the predicted set, and box B represents the true set. YOLOv5 uses the CIOU loss function, which considers three elements: overlapping area, center point distance, and aspect ratio (length-to-width ratio), and also adds a penalty factor. The calculation formula of the CIOU loss function is as follows:

[0070]

[0071] Where: ρ 2 (b,b gt ) represents the Euclidean distance between the center points of the two frames; c is the diagonal distance of the overlapping area of ​​the two frames.

[0072] The CIOU loss function also takes two additional variables into account: the parameter v representing the similarity between the two boxes, and the parameter a used to balance the ratio, also known as the penalty factor. The calculation formulas for a and v are as follows:

[0073]

[0074]

[0075] Where: w gt 、h gt , w, h represent the length and width of the B frame and the A frame respectively. Under this condition, the loss function of CIOU is defined as follows:

[0076]

[0077] The bounding box regression loss function of the output end adopts CIOU_Loss, which is improved for some occluded and overlapped targets, and greatly improves the speed and accuracy of the predicted box regression. Therefore, the application selects YOLOv5 as the basic algorithm, and optimizes through a series of experiments to obtain a target detection algorithm more suitable for train driver driving state monitoring.

[0078] Step 2.2, improved YOLOv5 network model

[0079] In order to avoid the threat to the life and property safety of passengers, the network model should keep a balance between speed and accuracy. With the rapid development of intelligent transportation system, the software to be carried by the hardware device is more and more, which brings great challenges. Therefore, under the premise of ensuring the accuracy required in the actual scene, the volume of the whole network structure should be reduced as much as possible. The application redesigns YOLOv5, reduces the volume, parameter quantity and calculation quantity of the algorithm by introducing a lightweight algorithm module and other improvement methods, and improves the detection speed under CPU.

[0080] Step 2.2.1, introducing GhostNet network model

[0081] For the research of lightweight network, MobileNetV1, MobileNetV2, ShuffleNet, GhostNet and the like are excellent lightweight network structures. The application selects GhostNet as the benchmark to construct a bottleneck structure "GhostBottleneck". In the lightweight improvement of YOLOv5 network, the GhostBottleneck is used as the backbone network.

[0082] The Ghost module is as shown in Figure 3 The core idea is to first perform a small convolution operation on the input X to obtain the inherent feature map Y. Then Y is linearly transformed to generate the redundant feature map Y'. Y' is the "ghost" of the inherent feature map Y.

[0083] Suppose n feature maps need to be generated, the specific operation can be represented as:

[0084] Y0=X*f+b

[0085] Wherein, the input data X R c×h×w , * is a convolution operation, the convolution kernel is represented as f R c×k×k×n , and b represents the bias term. Because there is a large amount of feature map redundancy, an ordinary convolution operation can be used to generate:

[0086] Y=X*f'

[0087] In the above formula, Y R h'×w'×m is the output of ordinary convolution, f'∈R c×k×k×m The convolution kernel is used to simplify the bias term. We need to get an n-dimensional feature map, but now we only get an m-dimensional one. Therefore, we perform a linear transformation on the existing feature map:

[0088]

[0089] In the above formula y i ' is the i-th feature map in the intrinsic feature map, Φ i,j In which i represents the i-th feature map, j represents the j-th linear transformation function. Finally, in order to retain the existing feature map, y i 'Do Φ i,s Theoretically, the speedup of the Ghost module transformed into a normal convolution is:

[0090]

[0091] In the above formula, one existing feature map corresponds to The convolution kernel size is d×d. d×d and k×k are similar in size, and s is much smaller than c. The parameter compression ratio is:

[0092]

[0093] In summary, the theoretical parameter compression ratio of the Ghost module transformed into a normal convolution is approximately equal to the theoretical speedup ratio. Therefore, the computational cost per channel of the linear operation is much less than that of a normal convolution.

[0094] In the YOLOv5 network, the C3 architecture is responsible for extracting features from the backbone network. This improves feature extraction capabilities by increasing the network's depth and receptive field. This is crucial for computer vision tasks such as object detection, as accurate recognition and localization require excellent feature extraction capabilities. Furthermore, this module maintains algorithmic detection accuracy while achieving lightweight performance, reducing computational and memory costs.

[0095] The C3 structure is also built based on the idea of ​​bottleneck structure. The bottleneck structure refers to the change in the number of channels, that is, the original number of channels is first reduced and then increased to the original size. The word bottleneck is more vivid to describe this structure. This structure can reduce the number of parameters, thereby reducing the amount of calculation, and after dimensionality reduction, data training and feature extraction can be more efficient and intuitive. It can also increase the depth of the neural network, reflecting the good performance of the network.

[0096] In order to eliminate redundant feature maps and reduce the computing pressure of hardware, the Ghost convolution module is introduced. Similarly, GhostBottleneck is constructed as the feature extraction module of the backbone network according to the bottleneck structure idea. The structure is as followsFigure 4 as shown.

[0097] The GhostBottleneck is used as a feature extraction module in the backbone network to replace the original C3 module. The Backbone structure after replacement is as shown. Figure 5 as shown.

[0098] After the backbone network is replaced, not only the parameter quantity of the network is reduced, but also the influence of feature geometric change is reduced, and the feature extraction performance of the target gesture to be detected is improved.

[0099] Step 2.2.2, introducing depth separable convolution

[0100] In order to solve the problem that the deep neural network is difficult to deploy in the embedded mobile terminal due to the huge volume of the network. The second part of the basic YOLOv5 network is improved by replacing the original convolution in the network with depth separable convolution. Depth separable convolution has less parameter quantity and lower operation cost than ordinary convolution, and is a kind of convolution network with superior performance. In order to accurately extract image features, ordinary convolution needs to fuse and map the channel dimension and the spatial dimension. The traditional convolution structure is as shown. Figure 6 as shown.

[0101] Depth separable convolution (Depthwise Separable Convolution) divides the process of ordinary convolution into two steps of channel-by-channel convolution and point convolution, realizing the decoupling of channel dimension and spatial dimension. The structure of depth separable convolution is as shown. Figure 7 as shown.

[0102] When performing convolution operation, the first step is channel-by-channel convolution, and the second step is channel jointing of the results of the first step by point convolution, and finally the image features are extracted. Assuming that 4 feature maps are required. In Figure 6 In traditional convolution operation, a 3-channel image with a size of 4x4 is first input, and then 4 3x3x3 filters are used, and finally 4 feature maps can be obtained. Calculation amount: 4x3x3x3=108. In Figure 7 In depth separable convolution, a 3-channel image with a size of 4x4 is also input, and then convolution is performed on each channel to obtain three feature maps. The calculation amount of this step is 3x3x3=27. Then, point-by-point convolution (convolution kernel size 1x1x3) is performed to obtain four feature maps. The calculation amount of this step is 1x1x3x4=12. The total calculation amount of the whole process is: 27+12=39. Compared with ordinary convolution, the calculation amount is reduced by about 2 / 3. In the network, the replacement of depth separable convolution can greatly reduce the parameter quantity of the unavoidable convolution operation.

[0103] Step 2.2.3, optimization of double-scale prediction mechanism

[0104] Different levels of feature maps contain different information. In the original YOLOv5 network, the network model can not only learn rich position information from low-level features, but also learn a large amount of position information from high-level features. Therefore, in the locomotive driver gesture detection scene applied in the present application, the third improvement is to redesign the multi-scale prediction mechanism of the original YOLOv5 network into a double-scale prediction, which greatly simplifies the network model to a certain extent, facilitating the deployment of embedded mobile terminals in the train cab.

[0105] In the target detection model, the large-scale feature map has more position information and less semantic information. On the contrary, in the small-scale feature map, the semantic information is more and the position information is less. In the actual locomotive cab scene detection, the general position information is relatively accurate, and the main reason for the target mis-detection problem is the category recognition error. Therefore, the multi-scale prediction mechanism of the YOLOv5 network is optimized in the present application, and the largest scale detection layer is deleted, and a double-scale prediction is adjusted. Correspondingly, the once up-sampling and once down-sampling operations are deleted, and the small-scale feature map is directly up-sampled and tensor-spliced with the medium-scale feature layer, which not only meets the requirement of semantic information and position information fusion, but also makes the overall network structure more lightweight, wherein the small-scale feature map is obtained by convolution operation on the medium-scale feature map of 40x40 in the Backbone part.

[0106] The specific implementation is as follows: the original YOLOv5 network structure with twice up-sampling is modified to once, obtaining a medium-scale feature map of 40x40, which is directly fused with the medium-size feature map in the backbone network to form a detection layer Y2(40x40). The feature map sent into the Y2 detection layer is down-sampled again and tensor-spliced with a small-size detection layer obtained by convolution operation (down-sampling) of the feature map sent into Y2, to form a detection layer Y1(20x20).

[0107] Deep feature map after multiple layers of down-sampling affects the detection accuracy of small and medium targets. Therefore, reducing the down-sampling operation can reduce the impact on accuracy and reduce the parameter quantity and calculation quantity of the network. YOLOv5 uses a three-scale prediction mechanism, and the network finally obtains three scale feature maps Y1:(20x20), Y2:(40x40) and Y3:(80x80). In complex scenes, three scales can basically cover objects of different sizes. However, in the case of single scene in the cab and small number of detection targets, in order to make the detection faster, only the information of medium and small size feature maps can complete the task. Considering further optimization of the multi-scale prediction mechanism of YOLOv5, the waste of computing power in the prediction process of large-scale feature maps is avoided. The image acquisition device in the cab of the locomotive is fixed in position, and the size of the gesture target collected is relatively fixed, so in the three detection layers of the head, two are changed to detect medium and small targets. The original network model can be effectively lightened, and the detection speed can be improved.

[0108] Step 2.2.4, improvement of the Focus module

[0109] The Focus module maps the spatial dimension information to the channel dimension without losing image information, which improves the network speed. After multiple slicing operations, the sliced feature layers are spliced together, the feature map becomes smaller, and the channel number becomes 4 times the original. The Focus schematic diagram is shown in FIG. 8. Without GPU and NPU acceleration, such frequent slicing operations will occupy a lot of cache and increase the calculation burden. Therefore, the Focus layer is removed in the present application to facilitate the deployment of the edge device on the train.

[0110] Step 2.2.5, deletion of the spatial pyramid pooling module

[0111] The structure diagram of the spatial pyramid module is shown in FIG. Figure 9 First, the input feature map is subjected to a pooling operation with three different size convolution kernels to extract feature information; second, in order to normalize the size, the three pooled feature maps are padded respectively; finally, the original feature map and the three feature maps are directly combined to obtain a feature map with a size of HxWx4C.

[0112] Parallel operation exists in the SPP structure, which will greatly slow down the overall running speed of the network. In the application scenario without complex target recognition, the SPP module is selected to be deleted. In the railway signal equipment, there is a "fault-oriented safety" principle, that is, when the equipment fails, it can automatically guide safety to ensure train safety. In the process of monitoring the driving state of the train driver in the present application, due to the fast speed of the train and the short reaction time, the process of monitoring the driving state of the train driver should be as fast as possible to detect whether the hand gesture of the train driver in the train cab is correct, and when the hand gesture is incorrect, the device should be given time to react and guide safety.

[0113] Step 2.2.6, network structure model of L-YOLOv5

[0114] After the above modification of the YOLOv5 model, the overall network structure is as follows Figure 10 As shown in the figure, in the backbone network part, the Focus module in the network is replaced by a 3x3 convolution operation and a maximum pooling module, and the parallel operation structure SPP is deleted, and then the GhostNet module is introduced to build a new bottleneck structure to replace the C3 structure of the algorithm backbone. The ordinary convolution in the whole network is replaced by a depth separable convolution to further reduce the parameter quantity and memory pressure. Finally, the three-scale prediction mechanism in the original network is optimized to a double-scale prediction mechanism. The overall structure is more compact than the original network, and the parameter quantity and calculation quantity are greatly reduced, which is a lightweight target detection model suitable for deployment in the cab of a locomotive driver.

[0115] The present specification also provides a driver gesture recognition method experimental verification, comprising the following steps:

[0116] 1. Experimental environment and parameter setting

[0117] As shown in Table 1, the configuration information in the experiment of the present application. The learning framework selects the Pytorch structure, which is an open source deep learning framework based on dynamic graph calculation, can be calculated and run multiple times, and is one of the very young and excellent deep learning frameworks. CUDA is a GPU computing platform that can provide acceleration services in many deep learning computing processes. The environment used and configured in the present application is Linux+Pytorch3.8+CUDA11.4, and the experimental results are completed using a graphics card version number of Tesla V100 SXM2 32GB.

[0118] Table 1 related environment configuration

[0119] Name Related configuration Operating system Linux Memory 32G Graphics card GPU Tesla V100 SXM2 32G CUDA version CUDA11.4 Language Python3.8 Deep learning framework Pytorch

[0120] The training of the model is based on the basic data and the 10000 locomotive driver gesture data sets after enhanced amplification. The number of training generations is set to 200 epochs, the weight decay is set to 0.0005, the momentum is set to 0.937, the initial learning rate is set to 0.01, and the optimizer adopts the stochastic gradient descent method SGD.

[0121] 2. Common evaluation indicators for target detection

[0122] (1) Precision

[0123] In a data set, it is assumed that the target categories to be trained only have positive and negative two categories. The confusion matrix of positive and negative samples is shown in Table 2.

[0124] Table 2 Confusion matrix of positive and negative samples

[0125] Symbol Meaning TP (Ture Positive) Predicted as positive sample, actual as positive sample FP (False Positive) Predicted as positive sample, actual as negative sample FN (False Negative) Predicted as negative sample, actual as positive sample TN (Ture Negative) Predicted as negative sample, actual as negative sample

[0126] Precision, also known as precision, is for a single picture. The calculation method is: the number of actual positive samples in the predicted sample / the number of all positive samples, as shown in the following formula:

[0127]

[0128] (2) Average precision (AP) and average precision mean (mAP)

[0129] Average precision (AP) is the average value of precision calculated by mathematical integration, as shown in the following formula:

[0130]

[0131] The average precision mean (mAP) is the arithmetic mean of the average precision (AP) of each class. The commonly used mAP is mAP@.5 (indicating the threshold is 0.5). The calculation formula is as follows:

[0132]

[0133] Where: N is the total number of categories, and AveP(i) is the average precision of the ith class. This is a comprehensive measure of the performance of all target detection.

[0134] (3) Prediction speed FPS

[0135] FPS (Frames Per Second) refers to frame rate, which represents the total number of images that can be predicted by the target detection model per second. If a target detection network can predict M images per second, then the FPS of the network is M.

[0136] 3. Experimental results and analysis

[0137] The purpose of using the L-YOLOv5 locomotive driver driving state monitoring algorithm is to reduce the size of the entire detection network, so that it can be smoothly mounted on embedded devices. For such devices, both cloud update speed and model calculation speed should be considered. Therefore, in terms of practicality, the reduction of network size can bring better results.

[0138] Figure 11 is the mAP@.5% curve comparison diagram of YOLOv5 and L-YOLOv5, the average precision of YOLOv5 is 95.2%, and the average precision of L-YOLOv5 is 90.8%, which is slightly lower, but also meets the requirements of the network for detection accuracy.

[0139] As can be seen from Table 3, the parameter amount of L-YOLOv5 is 2098k, and the size is 103.7mb, which is about 57% lower than YOLOv5. At the same time, the frame number of its detection on the local device can reach 34, which is about 2.3 times of YOLOv5. In general, the L-YOLOv5 network model meets the function of real-time detection.

[0140] Table 3 Comparison of model performance between YOLOv5 and L-YOLOv5

[0141] Model name mAP@.5% Parameter quantity / k Model size FPS YOLOv5 95.2％ 7276 242.3 15 L-YOLOv5 90.8％ 2098 103.7 34

[0142] (1) Real-time and accuracy detection performance comparison

[0143] Prediction speed (FPS) is an important indicator to measure the real-time performance of the network. The present application calls the notebook camera to perform real-time detection of video.

[0144] Table 4 Comparison of model performance of mainstream algorithms

[0145] Model name mAP@.5% FPS YOLOv3 92.4％ 11 EfficientDet-D0 75.37％ 45 RCNN 77.5％ 5 YOLOv5 95.2％ 15 L-YOLOv5 90.8％ 34

[0146] YOLOv3, EfficientDet-D0, RCNN, YOLOv5 and L-YOLOv5 under the same conditions. As can be seen from the data shown in Table 4: in terms of accuracy, the mAP of YOLOv5 is the highest, reaching 95.2%, but the FPS is relatively low, only 15; in terms of real-time, the FPS of EfficientDet-D0 detection algorithm is the highest, but its mAP is the lowest, only 75.37%. The mAP of L-YOLOv5 is 90.8%, and the FPS can reach 34, which can balance the real-time and detection accuracy.

[0147] (2) Comparison of lightweight effect

[0148] The volume size and parameter amount are two indicators for measuring the lightweight effect. Similarly, the mainstream target detection algorithm is selected, and a comparative experiment is carried out under the same conditions. As can be seen from the data shown in Table 5: the volume and parameter amount of YOLOv5 are the largest, and the volume and parameter amount of EfficientDet-D0 are the smallest. By comprehensively analyzing all the values, it can be concluded that the parameter amount and model volume of L-YOLOv5 are in the third place among the five algorithms listed.

[0149] Table 5 Comparison of model parameter amount and volume of mainstream algorithms

[0150] Model name Parameter quantity / k Model size / mb YOLOv3 5603 218.2 EfficientDet-D0 590 22.6 RCNN 1070 67.5 YOLOv5 7276 242.3 L-YOLOv5 2098 103.7

[0151] 4. Comprehensive analysis

[0152] Through the analysis of the data in the above table, it can be concluded that the parameter amount of YOLOv5 and YOLOv3 is more than 5000k, and the volume size is more than 200mb, which is difficult to achieve the effect of lightweight. The FPS is also lower than 20, which is difficult to meet the requirements of real-time. In addition to L-YOLOv5, the other two models have good effects in real-time and lightweight, but the mAP is below 80%, which is difficult to meet the standard. In summary, under the premise of ensuring detection accuracy, L-YOLOv5 performs best in driver gesture detection, and in terms of performance, L-YOLOv5 can achieve balance compared with other mainstream algorithms. Through the analysis of the experimental results, it can be concluded that the lightweight L-YOLOv5 detection algorithm proposed in the present application meets the experimental expectations, and is more suitable for use in locomotive driver gesture detection, and can be installed in devices with low memory configuration.

[0153] The specific limitations of the driver gesture recognition device can be referred to the limitations of the driver gesture recognition method described above, which will not be repeated here. Each module in the driver gesture recognition device described above can be implemented by software, hardware and their combination in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to each module by the processor.

[0154] The present specification also provides a computer readable storage medium, which stores a computer program, and the computer program can be used to execute the driver gesture recognition method described above.

[0155] The present specification also provides a computer device, which comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory in the hardware layer, and of course can also comprise other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to realize the single-cell RNA-seq data clustering method described above.

[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, storage, database or other medium can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical storage, etc. The volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM) and the like

[0157] The above-described embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present patent should be subject to the appended claims.

Claims

1. A driver gesture recognition method, characterized by, The application relates to an improved YOLOv5 network comprising a backbone network, a neck module and an output module connected in sequence, and the improved YOLOv5 network comprises the following steps: replacing a C3 module in an original YOLOv5 network with a GhostBottleneck module, replacing a convolution operation in the original YOLOv5 network with a depth separable convolution operation, deleting a focus module and a spatial pyramid pooling module SPP of the backbone network in the original YOLOv5 network, and replacing a three-scale prediction mechanism of an output module in the original YOLOv5 network with a two-scale prediction mechanism; the GhostBottleneck module comprises a parallel Ghostconv-DWConv-Ghostconv branch and a DWConv-CBS branch; the Ghostconv-DWConv-Ghostconv branch and the DWConv-CBS branch are simultaneously connected with an output end of a previous layer, two new outputs are obtained after the output of the previous layer is processed through the Ghostconv-DWConv-Ghostconv branch and the DWConv-CBS branch, and the two new outputs are fused to serve as an output of the GhostBottleneck layer; the three-scale prediction mechanism of the output module in the original YOLOv5 network is replaced with the two-scale prediction mechanism, which comprises the following steps: a detection layer of a maximum scale in the output module of the original YOLOv5 network is deleted, and an up-sampling and down-sampling operation corresponding to the detection layer of the maximum scale is deleted; a driver gesture to be recognized is input into the improved YOLOv5 network, a backbone network obtains lightweight features, a neck module fuses the lightweight features to obtain lightweight fusion features of different sizes, and an output module recognizes the lightweight fusion features of different sizes to output a recognition result of a driver gesture image to be recognized. the depth separable convolution operation comprises the following steps:

2. The driver gesture recognition method of claim 1, wherein, channel-wise convolution, the channel-wise convolution uses a convolution kernel with a same number of channels as an input image to perform a convolution operation, and outputs a feature map with a same number of channels as the input image; point-wise convolution, the point-wise convolution uses a convolution kernel of 1*1*n to perform a convolution operation on the feature map output by the channel-wise convolution, wherein n is the number of channels of the convolution kernel. the improved YOLOv5 network comprises a backbone network, a neck module and an output module connected in sequence, wherein:

3. The driver gesture recognition method of claim 1, wherein, the backbone network is used for generating lightweight features and comprises an input layer, a CBS layer connected with an output end of the input layer, three stacked DWConv-GhostBottleneck layers connected with an output end of the CBS layer, and a first DWConv layer connected with an output end of the third DWConv-GhostBottleneck layer; ​ The neck module Neck is used for fusing lightweight features of different scales, and is an intermediate connecting part of the backbone and the output, and comprises a second DWConv layer connected with an output end of a first DWConv layer, an upsample layer connected with an output end of the second DWConv layer, a first cancat layer connected with an output end of the upsample layer and an output end of a third DWConv-GhostBottleneck layer, a first C3_F layer connected with an output end of the first cancat layer, a third DWConv layer connected with an output end of the first C3_F layer, a second cancat layer connected with an output end of the third DWConv layer and an output end of the second DWConv layer, and a second C3_F layer connected with an output end of the second cancat layer. The detection module Output is used for generating a bounding box and a class probability of a driver gesture to be recognized, and comprises two conv2d layers outputting different scales, and a YOLO layer connected with output ends of the two conv2d layers.

4. The driver gesture recognition method of claim 1, wherein, The loss function used in the training process of the improved YOLOv5 network is a CIOU loss function, and the CIOU loss function is represented as: wherein A represents a frame corresponding to a prediction set of the loss function, B represents a frame corresponding to a real set, Union(A, B) represents a union set of A and B, Intersection(A, B) represents an intersection set of A and B, IOU represents an intersection union ratio of the prediction set and the real set of the loss function, c is a diagonal line distance of an overlapping region of the frame corresponding to the prediction set and the frame corresponding to the real set of the loss function, and p2(b, bgt) represents an Euclidean distance between center points of the frame corresponding to the prediction set and the frame corresponding to the real set of the loss function.

5. The driver gesture recognition method according to claim 4, characterized in that, In the training process of the improved YOLOv5 network, the number of training times is set to 200 epochs, the weight decay is set to 0.0005, the momentum is set to 0.937, the initial learning rate is set to 0.01, and the optimizer adopts a stochastic gradient descent method (SGD).

6. A driver gesture recognition apparatus, characterized by, The method comprises the following steps: The model establishment module is configured to: construct an improved YOLOv5 network comprising a backbone, a neck module and an output connected in sequence, wherein the improved YOLOv5 network comprises: the C3 module in the original YOLOv5 network is replaced by a GhostBottleneck module, the convolution operation in the original YOLOv5 network is replaced by a depth separable convolution operation, the focus module and the spatial pyramid pooling module (SPP) of the backbone in the original YOLOv5 network are deleted, and the three-scale prediction mechanism of the output in the original YOLOv5 network is replaced by a double-scale prediction mechanism; the GhostBottleneck module comprises parallel Ghostconv-DWConv-Ghostconv branches and DWConv-CBS branches; and the backbone is configured to: the neck module is configured to: the output is configured to: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method comprises the following steps: the method The Ghostconv-DWConv-Ghostconv branch and the DWConv-CBS branch are connected to the output of the previous layer at the same time. The output of the previous layer is processed by the Ghostconv-DWConv-Ghostconv branch and the DWConv-CBS branch to obtain two new outputs. The two new outputs are fused as the output of the GhostBottleneck layer; The model building module is further used to delete the largest-scale detection layer in the detection module Output of the original YOLOv5 network, and delete the upsampling and downsampling operations corresponding to the largest-scale detection layer; Identification module for: The driver's gesture to be recognized is input into the improved YOLOv5 network. The backbone network Backbone obtains lightweight features. The neck module Neck fuses the lightweight features to obtain lightweight fused features of different sizes. The detection module Output recognizes the lightweight fused features of different sizes and outputs the recognition result of the driver's gesture image to be recognized.

7. A computer readable storage medium characterized by The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer device, comprising: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

Images