A distracted driving behavior recognition method based on improved YoloV5 and Openpose

Abstract

A distracted driving behavior recognition method based on improved YoloV5 and Openpose includes: improving YoloV5: adding a prediction head to the shallow network of YoloV5 and optimizing the loss function of YoloV5; improving Openpose: drawing on the idea of MobileNet network, using the working principle of point-by-point convolution to increase dimensionality, solving the problem of information loss of depthwise separable convolution in low-dimensional space, applying it to the backbone network of lightweight Openpose, and introducing an attention mechanism of CA structure into the network; combining the improved YoloV5 with the improved Openpose, designing a new two-stage behavior recognition network method, using distance features and angle features as the judgment conditions of the recognition results, so that the entire distracted driving recognition process can not only ensure detection speed, but also further improve detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a distracted driving behavior recognition method based on improved YoloV5 and Openpose. Background Art

[0002] Artificial intelligence (AI) refers to machines created by humans that can display a certain level of intelligence. The concept was first proposed by Stanford University Professor John McCarthy. With the rapid development of information technology in the 21st century, AI has entered a golden age. AI has gradually become integrated into human society, meeting diverse needs. Consequently, AI has spawned the following technological fields: deep learning, computer vision, intelligent robotics, natural language processing, recommendation engines, data mining, and speech recognition.

[0003] Computer vision refers to the ability to use computers to identify, track, and locate objects, replacing the human eye. From facial recognition at supermarket checkouts to fault diagnosis in aerospace, its significance to human development is evident. Computer vision encompasses the following technologies: image classification, object detection, semantic segmentation, human pose estimation, and object tracking.

[0004] Distracted driving means voluntarily or involuntarily diverting attention from the main driving task in the current driving state (without the influence of alcohol or drugs, and without fatigue driving behavior), resulting in the vehicle being in a semi-controlled or uncontrolled state. At present, there are two types of distracted driving behavior identification solutions at home and abroad: one is to judge based on changes in the vehicle's driving trajectory, and the other is to judge based on the driver's external physiological manifestations. Although the indirect monitoring of changes in the vehicle's driving trajectory does not cause interference to the driver, factors such as vehicle differences and driver driving habits pose challenges to the identification of distracted driving. In addition, the direct monitoring of physiological indicators such as the driver's heart rate, blood pressure, and EEG signals often requires the wearing of cumbersome detection instruments, which can cause discomfort to the driver if worn for a long time. Summary of the Invention

[0005] In order to solve the technical problems raised by the background technology, the present invention provides a distracted driving behavior recognition method based on improved YoloV5 and Openpose. YoloV5 and Openpose are improved, and the two models are spliced into a two-stage behavior recognition network, so that the network model improves detection accuracy while ensuring detection speed.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A distracted driving behavior recognition method based on improved YoloV5 and Openpose includes the following steps:

[0008] Step 1: Image preprocessing: Capture an image through an imaging device. After the image is input, preprocess the input image to suppress noise, improve image clarity, and deal with image defects.

[0009] Step 2: Improve YoloV5: To mitigate the negative impact of drastic changes in target size, a prediction head is added to the original YoloV5 shallow network, allowing the entire target detection network to capture more detailed information. At the same time, the loss function of YoloV5 is optimized. This optimization enables it to better train on mutually exclusive category datasets and improves the convergence of the model.

[0010] Step 3: Improve Openpose: Based on the concept of the MobileNet network, we apply the principle of point-by-point convolution to the lightweight Openpose backbone network to solve the problem of information loss in low-dimensional space caused by depthwise separable convolution. At the same time, to make computing resources more reasonably allocated, we also introduce the attention mechanism of the CA structure into the network to avoid insufficient computing resources.

[0011] Step 4: Combine the improved YoloV5 from step 2 with the improved Openpose from step 3 to design a new two-stage behavior recognition network method. This method uses distance and angle features as the criteria for identifying distracted driving, ensuring both detection speed and accuracy.

[0012] Furthermore, in the step 1, the noise signal is suppressed by performing convolution on the Gaussian kernel and the image.

[0013] Furthermore, in step 2, in order to make full use of the feature information extracted by Backbone at different scales and enable the improved target detection network to better adapt to the multi-scale small target dataset, the YoloV5 prediction head was improved, and a P2 prediction head was added to the shallow network. When the overlapping area of the receptive field corresponding to each pixel point in the feature map is small, the network is ensured to capture more detailed information, so that the network can more accurately detect small-sized targets, thereby alleviating the negative impact of drastic changes in target size and detecting the presence of cigarettes and telephones in the dataset images.

[0014] Furthermore, in the step 2, the loss function of YoloV5 is also optimized, and the Softmax function with probability and output of 1 is used to optimize it, converting all scalars into probability distribution problems, rather than simply judging which one is larger and which one is smaller, so that it is more targeted at the corresponding category mutually exclusive data set scenario;

[0015] For any real vector of length T, Softmax compresses it into a real vector of length T with values in the interval (0,1), and the sum of the elements in the vector is 1. The Softmax function formula is mathematically defined as:

[0016]

[0017] For the e in the formula x , when x is large, memory overflow will occur during the code implementation process; in order to make Softmax more stable in terms of value, we process the values of various indicators at the same level. First, multiply the numerator and denominator of the Softmax function by a constant W at the same time, and then use the log function to normalize it. The formula is as follows:

[0018]

[0019] Among them, z is the output vector, z i and z t is an element of it, representing the values of the i-th and t-th output categories in the z vector, respectively. The final result is obtained by multiplying the vector output by the Softmax function with the target label vector. Optimizing the loss function in this way can better adapt to the application scenario and effectively improve the convergence of the overall model.

[0020] Furthermore, in the step three, in order to solve the problem of information loss in the low-dimensional space of the lightweight Openpose backbone network, improvements are made to the depthwise separable convolution; in order to enable the depthwise convolution to extract the manifold of interest in a higher dimension, a 1×1 point-by-point convolution is used before the depthwise convolution to perform a dimensionality increase operation, so that the 3×3 depthwise convolution can extract features in a higher dimensional space with better results; after feature extraction, 1×1 point-by-point convolution is used to reduce the dimensionality and restore the number of channels to reduce the amount of calculation; by drawing on the structure of ShortCut in the residual network, the input features and the output features are added as a unit, which can not only improve the network's propagation ability, but also train the network more effectively.

[0021] Furthermore, in step three, in order to reduce the high computational cost brought by the pose estimation network, the following improvements are made to Openpose: the spatial attention mechanism of the CA, i.e., CoordinateAttention structure, is introduced after the Block4, Block5, and Block6 modules respectively; the CA structure has two core operations: coordinate information embedding operation and coordinate attention generation operation.

[0022] Furthermore, in step 4, by analyzing the consistency of the two behaviors of smoking and making phone calls, the improved YoloV5 in step 2 and the improved Openpose in step 3 are combined to design a new two-stage behavior recognition network method, including the following:

[0023] Phase 1:

[0024] First, when smoking and making phone calls, the distance between the target object and the nose and ear is the shortest. Second, the elbows form a certain angle feature, and this angle is always less than 90°. By calculating their distance and angle features, we can determine which behavior the person is exhibiting.

[0025] Therefore, we first use the YoloV5 framework to obtain the coordinates of the center point M of the target object on the image. The expression is as follows:

[0026] M=(x i ,y i )

[0027] Among them, x i Represents the horizontal coordinate of the center point, y i Represents the vertical coordinate of the center point;

[0028] Phase 2:

[0029] Then, using the improved lightweight Openpose framework, we extract the position information of 18 skeletal key points of the human body posture, and construct the human skeleton through the heat map of the skeletal key points and the correspondence between the points. We also record the position coordinates of the nose key point, the right ear key point, and the left ear key point N coordinates. The expressions are as follows:

[0030] N=(x j ,y j )

[0031] Among them, x j Represents the horizontal coordinate of the key point, y j Represents the vertical coordinate of the key point;

[0032] The Euclidean distance between the phone and the left or right ear is used to determine whether the person in the image is making or receiving a phone call. Similarly, the Euclidean distance between the cigarette and the nose is used to determine whether the person in the image is smoking. The Euclidean distance d is calculated as follows:

[0033]

[0034] By calculating the Euclidean distance, when the result exceeds a certain threshold, we can determine whether the person is smoking or making a phone call. After completing the distance feature judgment, we also need to judge the angle feature. When a person smokes or makes a phone call, the elbow is bent at a certain angle. We make this judgment by calculating the angle θ between the wrist, elbow, and shoulder. Through the posture estimation framework, we obtain the coordinates of the key point position A on one side of the wrist, the coordinates of the key point position B on one side of the elbow, and the coordinates of the key point position C on one side of the shoulder, which are recorded as the formulas:

[0035] A=(x4,y4)

[0036] B=(x3,y3)

[0037] C=(x²,y²)

[0038] Calculate the lengths of each side based on the coordinate information of the three points, which are:

[0039] c=|AB

[0040] a=|BC

[0041] b=|AC

[0042] Using the above side lengths a, b, and c, calculate the elbow joint angle using the cosine theorem formula:

[0043]

[0044] Since the distance between a person and the camera can affect the distance between two key points, the angle feature θ is introduced here as a constraint on the distance feature.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1) This paper investigates the prediction head for object detection. The original YoloV5 prediction head has a minimum size of 8×8 pixels, which is insufficient for detecting cigarettes and phones in our dataset images. To mitigate the negative impact of drastic changes in object size, we added a prediction head to the shallow network, enabling the entire object detection network to capture more detailed information. We also optimized the YoloV5 loss function to better train on our mutually exclusive category dataset, improving model convergence.

[0047] 2) Drawing on the principles of the MobileNet network, this paper first utilizes the principle of point-by-point convolution to increase dimensionality, addressing the information loss problem of depthwise separable convolution in low-dimensional space. This approach is then applied to the lightweight Openpose backbone network. Furthermore, pose estimation networks are computationally expensive. To optimize resource allocation, we also introduce an attention mechanism within the CA architecture to mitigate issues such as underutilization of computational resources.

[0048] 3) This paper combines the improved YoloV5 with Openpose to design a new two-stage behavior recognition network, trains and optimizes its own dataset. The proposed two-stage distracted driving behavior recognition model achieves optimal recognition performance and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the overall workflow diagram of the present invention.

[0050] Figure 2 This is a comparison chart of the image effects before and after the Gaussian filtering used in the present invention.

[0051] Figure 3 This is a comparison diagram of the working principle of histogram equalization used in the present invention.

[0052] Figure 4 This is the working principle of YoloV5 adopted in the present invention.

[0053] Figure 5 This is the prediction head structure diagram proposed in the present invention.

[0054] Figure 6 This is the lightweight Openpose working principle adopted in the present invention.

[0055] Figure 7 This is the human body posture estimation network structure diagram proposed in this invention.

[0056] Figure 8 This is the structural diagram of the spatial attention mechanism proposed in this invention. DETAILED DESCRIPTION

[0057] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0058] This paper proposes a two-stage distracted driving behavior recognition method by combining an improved YoloV5 object detection network and an improved Openpose human pose estimation network. The method includes the following steps:

[0059] Step 1: After the image is input, it is preprocessed using methods such as Gaussian filtering and histogram equalization. Since the images we see are captured by various imaging devices, they may be affected by factors such as sensor materials, ambient air conditions, and circuit structures, generating uncontrollable noise signals. This noise signal is often irrelevant to the object of study and is considered useless information. Ideally, it would be desirable to remove it and create a smoother image. Therefore, preprocessing of the input image is necessary to suppress noise, improve image clarity, and address image defects.

[0060] Step 2: Through research and study of the target detection prediction head, we found that the original YoloV5 prediction head can detect a minimum target size of 8×8 pixels, which is not sufficient to detect the presence of cigarettes and phones in the dataset images. To mitigate the negative impact of drastic changes in target size, this paper adds a prediction head to the shallow network, allowing the entire target detection network to capture more detailed information. In addition, this paper optimizes the YoloV5 loss function, enabling it to better train on mutually exclusive category datasets and improve model convergence.

[0061] Step 3: Drawing on the principles of the MobileNet network and utilizing the principle of point-by-point convolution to increase dimensionality, this approach addresses the information loss issue associated with depthwise separable convolution in low-dimensional space and is applied to the lightweight Openpose backbone network. Furthermore, pose estimation networks are computationally expensive. To optimize resource allocation, this paper introduces a CA-structured attention mechanism to mitigate issues such as insufficient resource utilization.

[0062] Step 4: Combining the improved YoloV5 from Step 2 with the improved Openpose from Step 3, we designed a new two-stage behavior recognition network, using distance and angle features as criteria for recognition results. This ensures that the entire distracted driving recognition process maintains detection speed while further improving detection accuracy.

[0063] The specific contents are as follows:

[0064] 1) In step 1, the present invention uses a Gaussian kernel to perform convolution with the image to suppress the noise signal. The two-dimensional Gaussian distribution formula is defined as:

[0065]

[0066] Where x, y represent the two-dimensional coordinates of any point in the Gaussian filter, ux, uy represent the coordinates of the center point in the Gaussian filter, and σ represents the standard deviation in each direction.

[0067] Before suppressing noise, the weight of the Gaussian kernel needs to be determined. First, create a 3×3 Gaussian filter template and sample it with the center of the template as the coordinate origin. When the σ value is set to 1.5, the weight value of each position is calculated using the two-dimensional Gaussian distribution formula. The calculation process is as follows:

[0068]

[0069] To make the sum of the weights equal to 1, divide each of these nine numbers by their weighted average. After redistribution, the new weight matrix is obtained, which is the Gaussian kernel.

[0070]

[0071] Next, use the Gaussian kernel to perform Gaussian smoothing on the entire image. The calculation process is as follows:

[0072]

[0073] The left side of the convolution represents a random 3×3 pixel on the image. The convolution operation represents element-wise multiplication. The right side of the convolution represents a 3×3 Gaussian kernel. Convolving the 3×3 pixels with the Gaussian kernel produces the value of the new 3×3 pixels. Sliding the Gaussian kernel horizontally across the three channels of the RGB image and repeating the convolution operation yields a new Gaussian-smoothed image.

[0074] 2) In the step 2, in order to make full use of the feature information extracted by Backbone at different scales and enable the improved target detection network to better adapt to multi-scale small target data sets, the present invention has made improvements to the prediction head. The original YoloV5 only used 3 detection heads. The detection layer size corresponding to the P3 prediction head is 80×80, which can be used to detect targets larger than 8×8; the detection layer size corresponding to the p4 prediction head is 40×40, which can be used to detect targets larger than 16×16; the detection layer size corresponding to the P5 prediction head is 20×20, which can be used to detect targets larger than 32×32. The present invention adds a P2 prediction head to the shallow network to ensure that the network can capture more detailed information when the overlapping area of the receptive field corresponding to each pixel point in the feature map is small, so that our network can more accurately detect small-sized targets to alleviate the negative impact caused by drastic changes in target size.

[0075] In addition, the original YoloV5 uses the BCEWithLogitsLoss function as the binary cross entropy loss function to calculate the loss scores of classification and confidence, but because the BCEWithLogitsLoss function is used to include relationship categories, a target can belong to one or several categories. Therefore, the present invention uses the Softmax function with a probability and output of 1 to optimize it. It can convert all scalars into probability distribution problems instead of simply judging which one is larger and which one is smaller, making it more targeted at the category-exclusive dataset scenario corresponding to the present invention. For any real number vector of length T, Softmax can compress it into a real number vector of length T and a value in the interval (0,1), and the sum of each element in the vector is 1. The Softmax function formula can be mathematically defined as:

[0076]

[0077] For the e in the formula x , when x is large, memory overflow may occur during code implementation. To make Softmax more stable numerically, we process the values of various indicators at the same level. First, we multiply the numerator and denominator of the Softmax function by a constant W, and then use the log function to normalize them. The formula is as follows:

[0078]

[0079] Finally, the vector output by the Softmax function is multiplied by the target label vector to obtain the final result. Optimizing the loss function in this way can be better applied to the application scenario of the present invention and effectively improve the convergence effect of the overall model.

[0080] 3) In the step three, in order to solve the problem of information loss in the low-dimensional space of the lightweight Openpose backbone network, the present invention has made improvements on the depth-separable convolution. In order to enable the depth convolution to extract the manifold of interest in a higher dimension, the present invention uses a 1×1 point-by-point convolution to perform a dimensionality increase operation before the depth convolution, so that the 3×3 depth convolution can extract features in a higher dimensional space with better results. After feature extraction, 1×1 point-by-point convolution is used to reduce the dimension and restore the number of channels to reduce the amount of calculation. By drawing on the structure of ShortCut in the residual network, the input features and the output features are added as a unit, which can not only improve the network's propagation ability, but also train the network more effectively.

[0081] In addition, to reduce the high computational cost of the pose estimation network, the present invention introduces a spatial attention mechanism called a CA (Coordinate Attention) structure after the Block 4, Block 5, and Block 6 modules. The CA structure has two core operations: coordinate information embedding and coordinate attention generation.

[0082] In the coordinate information embedding operation, the global pooling is decomposed into a one-dimensional feature encoding operation. For the input feature C×H×W, each channel is first encoded along the horizontal coordinate direction and the vertical coordinate direction using pooling kernels of size (H, 1) and (1, W). The output formulas for X average pooling and Y average pooling are expressed as follows:

[0083]

[0084]

[0085] In the coordinate attention generation operation, in order to better utilize the feature map with precise location information generated by the coordinate information embedding operation, a shared 1×1 convolution is used to transform F1 to generate f∈R C / r×(H+W) , the formula is as follows:

[0086] f=δ(F1([z h ,z w ]))

[0087] Next, split f into two independent tensors f along the spatial dimension h ∈R C / r×H and f w ∈R C / r×W . Respectively f h and f w Use 1×1 convolution to restore the input features Figure 1 The number of channels is as follows:

[0088] g h =σ(F h (f h ))

[0089] g w =σ(F w (f w ))

[0090] The final result is obtained by normalizing and weighting the two:

[0091]

[0092] When computing resources are limited, the attention mechanism can focus on local information more effectively and provide a more reasonable resource allocation plan for important tasks, thereby reducing the attention paid to other information and improving the efficiency and accuracy of information processing.

[0093] 4) In step 4, the present invention designs a combined method for analyzing the consistency of smoking and making phone calls. During both smoking and making phone calls, first, the distance between the target object and the nose and ears is closest; second, the elbow forms a certain angle characteristic, which is always less than 90°. By calculating these distance and angle characteristics, the behavior of the person can be determined.

[0094] Therefore, we first use the YoloV5 framework to obtain the center coordinates of the target object on the image. The expression is as follows:

[0095] M=(x i ,y i )

[0096] Among them, x i Represents the horizontal coordinate of the center point, y i Represents the vertical coordinate of the center point.

[0097] The improved lightweight Openpose framework is then used to extract the position information of 18 skeletal key points of the human body posture. The human skeleton is constructed by using the heat map of the skeletal key points and the correspondence between the points. The position coordinates of the nose key point, the right ear key point position coordinates, and the left ear key point position coordinates are recorded. The expression is as follows:

[0098] N=(x j ,y j )

[0099] Among them, x j Represents the horizontal coordinate of the key point, y j Represents the vertical coordinate of the key point.

[0100] The Euclidean distance between the phone and the left or right ear is used to determine whether the person in the image is making or receiving a phone call. Similarly, the Euclidean distance between the cigarette and the nose is used to determine whether the person in the image is smoking. The Euclidean distance calculation formula is as follows:

[0101]

[0102] By calculating the Euclidean distance, when the result exceeds a certain threshold, we can determine whether the person is smoking or making a phone call. After completing the distance feature judgment, we also need to judge the angle feature. When a person smokes or makes a phone call, the elbow is bent at a certain angle. We make this judgment by calculating the angle θ between the wrist, elbow, and shoulder. Taking the right half of the human body as an example, the posture estimation framework can obtain the coordinates of the key points of the right wrist, right elbow, and right shoulder, which are respectively expressed as the formula:

[0103] A=(x4,y4)

[0104] B=(x3,y3

[0105] C=(x2,y2

[0106] Calculate the lengths of each side based on the coordinate information of the three points, which are:

[0107] c=|AB

[0108] a=|BC

[0109] b=|AC

[0110] Then calculate the radian value using the law of cosines:

[0111]

[0112] The reason we use angle features as an auxiliary judgment factor is that when a person is farther away from the camera, they shrink, and the distance between the two key points decreases. Similarly, when a person is closer to the camera, they zoom out, and the distance between the two key points increases. Therefore, relying solely on distance features for judgment is far from sufficient, so we need to introduce angle features as a constraint on the distance feature.

[0113] The specific implementation process is as follows:

[0114] Figure 1 This is the overall workflow diagram of the present invention. The specific implementation steps of a distracted driving behavior recognition research method based on improved YoloV5 and Openpose are as follows:

[0115] Step 1: First, preprocess the input image using Gaussian filtering and histogram equalization.

[0116] Step 2: Figure 2This is a comparison of the image effects before and after using Gaussian filtering in this invention. First, in a 3×3 Gaussian filter template, a Gaussian kernel is obtained by redistributing the two-dimensional Gaussian distribution. Then, the Gaussian kernel is slid horizontally on the three channels of the RGB image and the convolution operation process is repeated to obtain a new image after Gaussian smoothing. Figure 2 (a) represents the image before Gaussian filtering, Figure 2 (b) shows the image after Gaussian filtering.

[0117] Step 3: Figure 3 This is a comparison chart of the working principle of histogram equalization adopted by the present invention. In real scenes, it is inevitable that the captured image will be too dark or too bright. The target to be detected appears in these areas, which has a great impact on the inference of the results. Therefore, histogram equalization is used to evenly distribute the grayscale levels, so that the image has a higher contrast. Figure 3 (a) Histogram of the original image, where most pixels are concentrated in the brightness distribution range of 0-100; Figure 3 (b) represents the histogram after equalization. After processing, the image brightness distribution is more uniform; Figure 3 (c) represents the original image. When equalization is not performed, the original image scene is too dark and the visual effect is poor; Figure 3 (d) represents the image after equalization. Compared with Figure (c), it can be seen that the image is clearer and has richer details than the original image.

[0118] Step 4: After pre-processing the image through the previous steps, it will then formally enter the deep learning network model. Figure 4 The working principle of YoloV5 adopted in this invention is to give it a set of known questions and answers for simulation training. The model will summarize the rules based on the training content and form its own set of detection methodologies. After forming its own theory, the problem to be solved is given to the model, thereby solving the problems of regression and classification. Figure 5 This is the prediction head structure diagram proposed in this invention. A P2 prediction head is added to the shallow network. In this way, when the overlapping area of the receptive field corresponding to each pixel point of the feature map is small, the network can capture more detailed information, detect small-sized targets more accurately, and alleviate the negative impact caused by drastic changes in target size.

[0119] Step 5: After collecting information from target detection, we use the lightweight Openpose to estimate the human body posture. Figure 6This paper adopts the lightweight Openpose working principle. Drawing on the concepts of the single-person pose estimation network (CPM), using MobileNetV1 as the backbone, it constructs two branches to perform inference estimation on the PCM heatmap of key points and the partial affinity vector field (PAF). Depthwise separable convolution is used to reduce computational complexity, while dilated convolution is used to increase the receptive field, enabling the model to achieve optimal results.

[0120] Step 6: Based on the lightweight Openpose network model, the present invention proposes two improved methods. Figure 7 Figure 1 shows the human pose estimation network architecture proposed in this paper. Deep convolution suffers from severe information loss in low-dimensional space. To address this issue, this paper uses a point-by-point convolution before deep convolution to perform a dimensionality-increasing operation. This allows deep convolution to extract the manifold of interest in a higher dimension, giving it more opportunities to acquire valid information, thus resolving the information loss issue. Figure 8 This is the architecture of the spatial attention mechanism proposed in this paper. Pose estimation networks often come with high computational costs, which can lead to a shortage of computing resources. When computing resources are limited, the attention mechanism can more effectively focus on local information, providing a more reasonable resource allocation solution for important tasks, thereby reducing the focus on other information and improving the efficiency and accuracy of information processing.

[0121] Step 7: After the target detection and human posture estimation reasoning are completed, the reasoning results are further judged. Using the Euclidean distance formula, the distance between the cigarette and the nose and the distance between the mobile phone and the ears are calculated. When a certain threshold is exceeded, it can be determined as distracted driving behavior such as smoking or making phone calls. In order to improve the accuracy of the model result judgment, the present invention adds angle feature auxiliary judgment conditions. When smoking and making phone calls occur, the angles between the wrist, elbow and shoulder joints will be less than 90 degrees. The angle feature judgment of the position coordinates of the three is performed through human posture estimation, thereby greatly improving the accuracy of the judgment results.

[0122] The present invention used two datasets for training: the object detection network was trained on the invention's homemade cigarette and phone datasets, achieving a mAP of 77.22%, a 9.02 percentage point improvement compared to the original training set. The pose estimation network was trained on the COCO dataset and achieved an AP of 64.8% on the COCO dataset, a 6.1 percentage point improvement compared to the original training set.

[0123] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0128] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A distracted driving behavior recognition method based on improved YoloV5 and Openpose, characterized by: The steps include: Step 1: Image preprocessing: Capture an image through an imaging device. After the image is input, preprocess the input image to suppress noise, improve image clarity, and deal with image defects. Step 2: Improve YoloV5: To mitigate the negative impact of drastic changes in target size, a prediction head is added to the original YoloV5 shallow network, allowing the entire target detection network to capture more detailed information. At the same time, the loss function of YoloV5 is optimized. This optimization enables it to better train on mutually exclusive category datasets and improves the convergence of the model. Step 3: Improve Openpose: Based on the concept of the MobileNet network, we apply the principle of point-by-point convolution to the lightweight Openpose backbone network to solve the problem of information loss in low-dimensional space caused by depthwise separable convolution. We also introduce the attention mechanism of the CA structure into the network to avoid insufficient utilization of computing resources. Step 4: Combine the improved YoloV5 from Step 2 with the improved Openpose from Step 3 to design a new two-stage behavior recognition network method. This method uses distance and angle features as criteria for the recognition results. This ensures that the entire distracted driving recognition process not only maintains detection speed but also further improves detection accuracy. In step 2, in order to make full use of the feature information extracted by Backbone at different scales and enable the improved target detection network to better adapt to the multi-scale small target dataset, the YoloV5 prediction head is improved and a P2 prediction head is added to the shallow network. When the overlapping area of the receptive field corresponding to each pixel point in the feature map is small, the network is guaranteed to capture more detailed information, so that the network can more accurately detect small-sized targets, thereby alleviating the negative impact of drastic changes in target size and detecting the presence of cigarettes and phones in the dataset images; In the step 2, the loss function of YoloV5 is also optimized, and the Softmax function with probability and output of 1 is used to optimize it, converting all scalars into probability distribution problems, rather than simply judging which one is larger and which one is smaller, so that it is more targeted at the corresponding category mutually exclusive data set scenario; For any real vector of length T, Softmax compresses it into a real vector of length T with values in the interval (0,1), and the sum of the elements in the vector is 1. The Softmax function formula is mathematically defined as: For the e in the formula x , when x is large, memory overflow will occur during the code implementation process; in order to make Softmax more stable in terms of value, we process the values of various indicators at the same level. First, multiply the numerator and denominator of the Softmax function by a constant W at the same time, and then use the log function to normalize it. The formula is as follows: Among them, z is the output vector, z i and z t is one of the elements, representing the values of the i-th and t-th output categories in the z vector respectively; the final result is obtained by multiplying the vector output by the Softmax function with the target label vector. Optimizing the loss function in this way can be better applied to application scenarios and effectively improve the convergence effect of the overall model; In step 4, by analyzing the consistency of the two behaviors of smoking and making phone calls, the improved YoloV5 from step 2 and the improved Openpose from step 3 were combined to design a new two-stage behavior recognition network method, including the following: Phase 1: First, when smoking and making phone calls, the distance between the target object and the nose and ear is the shortest. Second, the elbows can form a certain angle feature, and this angle is less than 90°. By calculating their distance and angle features, we can determine which behavior the person is exhibiting. Therefore, we first use the YoloV5 framework to obtain the coordinates of the center point M of the target object on the image. The expression is as follows: M=(x i ,y i ) Among them, x i Represents the horizontal coordinate of the center point, y i Represents the vertical coordinate of the center point; Phase 2: Then, using the improved lightweight Openpose framework, we extract the position information of 18 skeletal key points of the human body posture, and construct the human skeleton through the heat map of the skeletal key points and the correspondence between the points. We also record the position coordinates of the nose key point, the right ear key point, and the left ear key point N coordinates. The expressions are as follows: N=(x j ,and j ) Among them, x j Represents the horizontal coordinate of the key point, y j Represents the vertical coordinate of the key point; The Euclidean distance between the phone and the left or right ear is used to determine whether the person in the image is making or receiving a phone call. Similarly, the Euclidean distance between the cigarette and the nose is used to determine whether the person in the image is smoking. The Euclidean distance d is calculated as follows: By calculating the Euclidean distance, when the result exceeds a certain threshold, we can determine whether the person is smoking or making a phone call. After completing the distance feature judgment, we also need to judge the angle feature. When a person smokes or makes a phone call, the elbow is bent at a certain angle. We make this judgment by calculating the angle θ between the wrist, elbow, and shoulder. Through the posture estimation framework, we obtain the coordinates of the key point position A on one side of the wrist, the coordinates of the key point position B on one side of the elbow, and the coordinates of the key point position C on one side of the shoulder, which are recorded as the formulas: A=(x4,y4) B=(x3,y3) C=(x²,y²) Calculate the lengths of each side based on the coordinate information of the three points, which are: c=|AB| a=|BC| b=|AC| Using the above side lengths a, b, and c, calculate the elbow joint angle using the cosine theorem formula: Since the distance between a person and the camera can affect the distance between two key points, the angle feature θ is introduced here as a constraint on the distance feature.

2. The distracted driving behavior recognition method based on improved YoloV5 and Openpose according to claim 1 is characterized in that: In the step 1, the noise signal is suppressed by performing convolution on the Gaussian kernel and the image.

3. The distracted driving behavior recognition method based on improved YoloV5 and Openpose according to claim 1 is characterized in that: In the step three, in order to solve the problem of information loss in the low-dimensional space of the lightweight Openpose backbone network, improvements are made to the depthwise separable convolution; in order to enable the depthwise convolution to extract the manifold of interest in a higher dimension, a 1×1 point-by-point convolution is used before the depthwise convolution to perform a dimensionality increase operation, so that the 3×3 depthwise convolution can extract features in a higher dimensional space with better results; after feature extraction, 1×1 point-by-point convolution is used to reduce the dimensionality and restore the number of channels to reduce the amount of calculation; by drawing on the structure of ShortCut in the residual network, the input features and the output features are added as a unit, which can not only improve the network's propagation ability, but also train the network more effectively.

4. The distracted driving behavior recognition method based on improved YoloV5 and Openpose according to claim 1 is characterized in that: In step three, in order to reduce the high computational cost brought by the pose estimation network, the following improvements are made to Openpose: the spatial attention mechanism of the CA (CoordinateAttention) structure is introduced after the Block4, Block5 and Block6 modules respectively; the CA structure has two core operations: coordinate information embedding operation and coordinate attention generation operation.

Images