Vision-based hand-off detection and judgment method and electronic equipment

Abstract

This invention provides a vision-based method and electronic device for detecting hand detachment from the steering wheel, comprising: acquiring image data containing only the steering wheel image to obtain a steering wheel mask region; acquiring in-vehicle scene image data including the steering wheel and the driver / passenger in real time, and obtaining the driver's position data through a target driver detection neural network model; inputting the driver's position data into a target skeletal keypoint regression neural network model to regress skeletal keypoints and extracting the position coordinates of the left and right wrist points, and expanding them to obtain left and right hand expanded region images; inputting the left and right hand expanded region images into a target hand region segmentation network model to obtain the left and right hand mask regions, calculating the mask IOU of the steering wheel mask region and the left and right hand mask regions respectively, and if it is greater than a first threshold, determining that the corresponding hand is on the steering wheel; otherwise, determining that the hand is detached from the steering wheel. This invention is applicable to assisting in judging whether the driving behavior of candidates is standardized during driving tests.

Description

Technical Field

[0001] This invention belongs to the field of driving safety detection technology, specifically relating to a vision-based method and electronic device for detecting and judging hand detachment from steering wheel. Background Technology

[0002] With the continuous development of the driving test field, the regulations on drivers' driving behavior are becoming increasingly strict. In driving test training and examinations, determining whether hands have left the steering wheel is a crucial indicator. Driving with both hands off the steering wheel or prolonged one-handed driving results in a failing grade.

[0003] Steering wheel hands-off detection is a crucial part of driving safety, as it's key to ensuring drivers maintain focus and can alleviate traffic congestion and reduce accidents to some extent. Currently, most hands-off detection devices on the market rely on hardware sensors. These typically require sensors installed in the steering wheel and connected to control circuitry. The sensors detect when no hands are on the steering wheel, sending a signal that triggers an alarm. However, this method has limitations, including complex installation, high hardware costs, and the need to modify the vehicle's hardware, all of which require improvement. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a vision-based method and system for detecting hand detachment from the steering wheel. By combining machine vision with deep learning algorithms, a method for detecting hand detachment from the steering wheel is obtained, which is applicable to assisting in judging whether a candidate's driving behavior is standardized during driving tests.

[0005] The specific technical solution of the present invention includes:

[0006] Solution 1 discloses a vision-based method for detecting hand detachment from the steering wheel, which includes:

[0007] Step 1: Acquire image data that includes only the steering wheel view, and use markers to delineate the specific position of the steering wheel in the real-time image to obtain the mask area of ​​the steering wheel and the coordinates of the markers;

[0008] Step 2: Acquire in-vehicle scene image data including the steering wheel and passengers in real time, and obtain the driver's position data through a target driver detection neural network model;

[0009] Step 3: Input the driver's position data into the target skeletal key point regression neural network model, regress to obtain the position data of the skeletal key points, extract the position coordinates of the left and right wrist points from the skeletal key points, and expand outwards according to preset requirements with the position coordinates of the left and right wrist points as the center to obtain the images of the outward expansion areas of the left and right hands.

[0010] Step 4: Input the images of the extended regions of the left and right hands into the target hand region segmentation network model to obtain the mask regions of the left and right hands. Map the mask regions of the left and right hands onto the original image of the mask region with the labeled steering wheel. Calculate the mask iou of the steering wheel mask region and the mask regions of the left and right hands respectively. If the mask iou is greater than the first threshold, it is determined that the corresponding hand is on the steering wheel; otherwise, it is determined that the hand is off the steering wheel.

[0011] Optionally, step 1 specifically includes:

[0012] Step 1.1: Acquire image data containing the steering wheel view using a camera fixed inside the driving test vehicle;

[0013] Step 1.2: Use the marking points to outline the specific location of the entire steering wheel in the image, and obtain the mask area of ​​the steering wheel and the coordinates of the marking points.

[0014] Optionally, in step 2, obtaining the driver's location data through a target driver detection neural network model specifically includes:

[0015] Several rectangular bounding boxes are used as detection frames to determine the position of the driver and passengers in the in-vehicle scene image, and the position data, confidence score and classification result corresponding to the detection frames are recorded; the classification result includes two categories: with driver and passengers and without driver and passengers.

[0016] Filter out invalid detection boxes with a confidence level below the second threshold;

[0017] If the number of detection boxes is greater than 1, select the rightmost detection box in the image as the driver detection box and output the position data corresponding to the driver detection box.

[0018] Optionally, the detection box is a rectangular bounding box, the position of which is defined by the coordinates of the upper left and lower right corners of the rectangular bounding box, or by the coordinates of the upper left corner of the rectangular bounding box and the length and width of the rectangle.

[0019] Optionally, in step 2, the target driver detection neural network model is a YOLOv5 neural network model, which is composed of a CSP+Focus+SPP structure; wherein: CSP includes a residual structure of convolutional layers, BN layers, and activation layers; Focus includes four slicing layers for fusing multiple scales of the input image; SPP includes three max pooling layers, which output features at different scales respectively.

[0020] The training process of the target driver detection neural network model includes:

[0021] Acquire multiple sets of in-vehicle scene image data, including images of the steering wheel and driver / passengers;

[0022] In the in-vehicle scene image, a detection box is drawn to define the position of the driver and passengers. The position coordinates and classification of the detection box are recorded to form the label corresponding to the detection box. The in-vehicle scene image and the corresponding label data are used together to form a training dataset.

[0023] The YOLOv5 neural network model is trained using the training dataset until it converges after the total loss of the YOLOv5 neural network model stabilizes, thus obtaining the target driver detection neural network model.

[0024] The total loss function of the YOLOv5 neural network model is:

[0025] Loss=λ1L cls +λ2L obj +λ3L loc

[0026] In the formula: Loss represents the total loss; λ1, λ2, λ3 represent the balance coefficients; L cls The classification loss is represented by two categories: driver and non-driver. The BCE loss is used to calculate the classification loss for positive samples, where the positive samples are detection boxes with an IOU greater than a threshold; L obj The target loss is represented by BCE loss, which is used to calculate the Interchange of Values ​​(IOU) between the predicted bounding boxes and the ground truth bounding boxes of the neural network model; L loc This represents the localization loss, calculated using IOU loss for positive samples.

[0027] Optionally, step 3 specifically includes:

[0028] Step 3.1: Input the driver's position data into the target skeletal key point regression neural network model to obtain the position coordinates of 14 skeletal key points and the confidence of the corresponding coordinates, and extract the position coordinates of the skeletal key points that may belong to the left and right wrist points;

[0029] Step 3.2: Filter the position coordinates with confidence levels below the third threshold using the maximum suppression method to obtain the position coordinates of the left and right wrist points;

[0030] Step 3.3: Expand outward by 100 pixels from the position coordinates of the left and right wrists to obtain the images of the expanded areas of the left and right hands.

[0031] Optionally, in step 3, the target skeleton keypoint regression neural network model includes a ResNet50 network and an SSTN symmetric spatial transformation network; the parameters and computational complexity of the ResNet50 network include 50 convolutional layers, 50 activation layers, 1 pooling layer, and 1 fully connected layer; the SSTN symmetric spatial transformation network is used to perform spatial transformation on the driver detection box so that the driver is in the center of the detection box;

[0032] The training process of the target skeletal key point regression neural network model includes:

[0033] In the in-vehicle scene image containing the steering wheel and the driver and passengers, the positions of 14 skeletal key points of the driver are marked, the coordinates of these positions and the corresponding skeletal key point categories are recorded and training labels are formed. These training labels and image data are used together as the training dataset for the skeletal key point regression neural network model.

[0034] The ResNet50 network and SSTN symmetric space transformation network, constructed by training the training set, are trained until the sum of the loss functions tends to stabilize and converge, thus obtaining the target skeleton key point regression neural network model.

[0035] The loss function is the distance loss function L1 loss, and the sum of the loss functions is the sum of the distances between the 14 skeletal key points and the driver's center point.

[0036] Optionally, step 4 specifically includes:

[0037] Step 4.1: Obtain the extended regions of the left and right hands, and resize the extended regions of the left and right hands to the input size of the target hand region segmentation network model;

[0038] Step 4.2: Input the resized extended regions of the left and right hands into the target hand region segmentation network model, and obtain the mask regions of the left and right hands through the target hand region segmentation network model;

[0039] Step 4.3: Construct a binary array on the original image with steering wheel markings obtained in Step 1, convert the coordinates of the steering wheel markings into a binary array, and map the masks of the left and right hands into the binary array;

[0040] Step 4.4: Calculate the mask iou of the steering wheel mask area and the mask areas of the left and right hands respectively. If the mask iou is greater than the first threshold, it is determined that the corresponding hand is on the steering wheel; otherwise, it is determined that the hand is off the steering wheel. The formula for calculating mask iou is as follows:

[0041] mask iou=(mask_hand∩maks_pan) / (mask_hand∪msk_pan)

[0042] In the formula, mask_hand represents the mask area of ​​the hand, and mask_pan represents the mask area of ​​the steering wheel.

[0043] Optionally, the target hand region segmentation network model includes a ResNet18 network model and an attention mechanism network layer; the parameters of the ResNet18 network model include 18 convolutional layers, 18 activation layers, 1 pooling layer, and 1 fully connected layer; the attention mechanism network layer is used to encode contextual information, reweight the contextual information, and fuse feature information;

[0044] The training process of the target hand region segmentation network model includes:

[0045] Use annotation tools to annotate the left and right hand regions in the original image with driver markings, and generate binarized annotated images as training labels;

[0046] The training labels are input into the constructed hand region segmentation network model for training until the total loss function value tends to stabilize and converges, thus obtaining the target hand region segmentation network model.

[0047] The total loss includes pixel category loss and auxiliary loss, which are ultimately output by the softmax loss function, showing the loss value of each sample pixel in the training set.

[0048]

[0049] In the formula, loss represents the total loss, N represents the number of samples, and i and j both represent samples, i = 1, 2, ..., N, j = 1, 2, ..., N, L i This represents the result of the softmax loss function at each pixel, and p represents the network's predicted output.

[0050]

[0051] In the formula, L represents the sample loss, α is used to balance the pixel category loss and auxiliary loss, and takes a value between 0 and 1. p Indicates pixel category loss, l i Let X represent the auxiliary loss, W represent the pixel predicted by the network, i represent the pixel of the actual label, and i represent the number of pixels starting from the i-th pixel, i = 1, 2, ..., K.

[0052] Option 2 discloses an electronic device, which mainly includes a processor and a memory. The processor is used to call a computer program stored in the memory to execute the vision-based hand-off-steering-wheel detection method described in Option 1 and its alternatives. This invention is based on machine vision technology, can be applied to any complex scenario, and involves only hardware devices including cameras and electronic devices such as computers and tablets. It requires no modification to the vehicle body, greatly reducing hardware costs and power consumption. Furthermore, it exhibits high robustness to different drivers and vehicle models, possessing high practical value and broad application prospects. Attached Figure Description

[0053] Figure 1 Diagram illustrating the marking process for the steering wheel;

[0054] Figure 2 A diagram illustrating the key points of the human skeleton;

[0055] Figure 3 This is a schematic diagram of the STN network structure;

[0056] Figure 4 The images show the actual application effect, among which, Figure 4 'a' represents a gesture indicating both hands are gripping the steering wheel; Figure 4 b is an illustration of holding the steering wheel with one hand. Detailed Implementation

[0057] To further understand the present invention, the technical solution of the present invention will be further explained and described below with reference to specific embodiments and accompanying drawings.

[0058] This embodiment discloses a vision-based method for detecting hand detachment from the steering wheel, which mainly includes the following four steps, as detailed below:

[0059] Step 1: Obtain image data including only the steering wheel view, mark the specific position of the steering wheel in the image, and obtain the mask area of ​​the steering wheel and the coordinates of the marking point.

[0060] The specific process of step 1 is as follows:

[0061] Step 1.1: Acquire image data containing the steering wheel view using a camera. The camera is fixed in a specific location within the driving test vehicle, where there are no drivers or passengers. The fixed camera records real-time footage containing the entire steering wheel.

[0062] Step 1.2: Using the labelme tool, mark the entire steering wheel area in the image using several marking points, outlining the steering wheel mask area. The marking result is as follows: Figure 1 As shown, labelme can save the marking information (i.e., the coordinates of each marking point) to a JSON file.

[0063] Step 1.3: Further, the marking information of the steering wheel in the JSON file can be read to generate a configuration file and configured into the corresponding algorithm, as in step 4.

[0064] Step 2: Acquire in real-time in-vehicle scene image data including the steering wheel and occupants, and detect the driver's position data through a target driver detection neural network model.

[0065] The specific process of step 2 is as follows:

[0066] Step 2.1: Acquire real-time in-vehicle scene image data, including the steering wheel and occupants, using a camera. During driver training, the occupants typically include the safety officer and the examinee in the driving test vehicle, with the examinee acting as the driver. The image data acquired at this stage reflects a normal driving test, where there is at least one driver in the vehicle.

[0067] Step 2.2: Input the acquired in-vehicle scene image data into the trained target driver detection neural network model to detect the driver's position data.

[0068] Specifically, multiple rectangular bounding boxes (also called "detection boxes") of varying sizes can be used to define the object. The position of the rectangular bounding box is generally defined by four integers, representing the coordinates of the top-left and bottom-right corners, or the coordinates of the top-left corner, the length, and the width of the rectangle. In addition to the position data, each detection box also has a corresponding confidence score and classification result. The classification result includes two categories: those with occupants and those without occupants. Invalid detection boxes with confidence scores below a preset threshold are filtered out, and the remaining detection boxes are considered valid. Then, the rightmost detection box in the image is selected as the driver detection box, thus obtaining relevant information about the driver detection box, including its position data, confidence score, and classification result.

[0069] The target driver detection neural network model can be obtained by training a YOLOv5 neural network model. After constructing the initial YOLOv5 neural network model, the model is trained until the total loss function value (Loss) of the neural network tends to converge and stabilize, thus obtaining the target YOLOv5 neural network model, i.e., the target driver detection neural network model.

[0070] The YOLOv5 neural network model uses a CSP+Focus+SPP structure. CSP includes a residual structure of convolutional layers, BN layers, and activation layers. Focus contains four slice layers, which mainly fuse multi-scale input images. Multi-scale means that during training, the network clusters anchors of different scales as sliding boxes. SPP contains three max pooling layers, which output features at different scales, mainly including feature maps of scales of 80*80, 40*40, and 20*20.

[0071] The loss function of the YOLOv5 neural network model mainly consists of three parts: classification loss (L... cls The classification loss for positive samples is calculated using BCE (Binary Cross-Entropy Loss), where positive samples represent bounding boxes with an IOU (Intersection over Union) greater than a threshold of 0.5. The classification loss is mainly divided into two categories: driver and non-driver. The target loss (Lo) is calculated using BCE loss. bj The BCE loss is used to calculate the IOU between the predicted detection boxes and the actual detection boxes; the localization loss (L) is used. loc The positional loss of positive samples is calculated using IOUloss.

[0072] The total loss function is:

[0073] Loss=λ1L cls +λ2L obj +λ3L loc

[0074] In the formula: Loss represents the total loss; λ1, λ2, and λ3 represent the balance coefficients, with values ​​of 0.4, 0.4, and 0.2 respectively, which can be adjusted according to the actual situation; L cls Represents classification loss; Lo bj Indicates the target loss; L l o c This indicates positioning loss.

[0075] The training process for the target YOLOv5 neural network model is as follows:

[0076] Step a1: Acquire multiple sets of in-vehicle scene image data, including images of the steering wheel and driver / passengers;

[0077] Step a2: Use the labelme annotation tool to draw detection boxes in the in-vehicle scene image to define the positions of drivers and passengers by marking points, and record the position coordinates and classification results of the detection boxes to form the labels corresponding to the detection boxes. The in-vehicle scene image and the corresponding labels together constitute the training data.

[0078] Step a3: Input the training data into the YOLOv5 neural network model for training. Through continuous backpropagation and data iteration, the model's total loss tends to stabilize and converge, resulting in the target driver detection neural network model, also known as the "target YOLOv5 neural network model". The backpropagation process involves using matrix multiplication to calculate the gradient derivative of the parameters of each batch of samples in each convolutional layer, summing the derivatives, and then weighting the results from the previous layer with the results from the next layer to obtain the final feature values.

[0079] Step 3: Input the driver detection box information obtained into the target skeletal key point regression neural network model to obtain the position data of 14 skeletal key points. Select the positions of the left and right wrist points and expand outward with the wrist points as the center to obtain the images of the outward expansion areas of the left and right hands.

[0080] The specific process of step 3 is as follows:

[0081] Step 3.1: Input the driver position data obtained in Step 2 into the target skeletal keypoint regression neural network model to obtain the position coordinates (referred to as "skeletal keypoint coordinates") and the confidence of the corresponding coordinates of 14 skeletal keypoints, and find the position coordinates that may belong to the left and right wrist points (referred to as "left and right wrist point coordinates").

[0082] The location distribution of skeletal key points is as follows Figure 3 As shown. It should be noted that the skeletal keypoint coordinates regressed by the network model are usually arranged in a specific order. Each skeletal keypoint includes at least one set of coordinates. This specific arrangement order can be used to determine which coordinates are likely to be the left and right wrist points.

[0083] Step 3.2: After obtaining at least one set of wrist point position coordinates, filter the position coordinates with confidence scores below the threshold of 0.5 using the maximum suppression method to obtain the position coordinates of the left and right wrist points.

[0084] Step 3.3: Based on the position coordinates of the left and right wrist points, expand outward by 100 pixels from the left and right wrist points as the center points to obtain the images of the expanded areas of the left and right hands.

[0085] The target skeletal keypoint regression neural network model here mainly includes the ResNet50 network and the SSTN symmetric spatial transformation network. The ResNet50 network has the following parameters and computational cost: 50 convolutional layers, 50 activation layers, 1 pooling layer, and 1 fully connected layer, serving as a pre-trained model for training the skeletal keypoint network. The network parameters of ResNet50 are shown in Table 1. As can be seen from Table 1, with an input driver image size of 224*224*3, the FLOPs (floating-point numbers representing the number of parameters) of ResNet50 are 39*10^6. 8 The number of parameters is relatively small.

[0086] Table 1

[0087]

[0088] The SSTN (Symmetric Spatial Transform Network) primarily performs spatial transformation on the driver detection box, centering the driver within the box. It mainly comprises the STN (Spatial Transform Network), SPPE (Single Person Pose Estimation) branch, and SDTN (Inverse Spatial Transform Network). The input consists of an image containing the driver and the driver detection box. Typically, the distance between the center point of the driver detection box (the center of the rectangular bounding box) and the center point of the image is initially determined. A large distance indicates a large error value, while a smaller distance indicates a smaller error value. Through continuous error optimization, the center point of the driver detection box gradually approaches the center point after the STN transformation, thus optimizing the STN and enabling it to extract a more accurate driver detection box. SPPE primarily regresses the coordinates of 14 skeletal points. SDTN maps the obtained skeletal points back to the input image using the Inverse Spatial Transform Network. The SSTN network structure is as follows: Figure 3 As shown.

[0089] The training methods for target skeletal keypoint regression neural network models mainly include:

[0090] Based on the location data in the driver detection box information obtained in step 2, the location of each skeletal point of the driver is manually labeled as a training label. The SSTN network obtains the location of the driver's body skeletal key points in the training label, including the location of 14 skeletal key points such as the head, left and right shoulders, left and right elbows and left and right wrists, records the coordinates and categories of these locations (which skeletal key point they belong to) and forms labels. These labels and image data are used together as the training dataset for the skeletal key point regression neural network model.

[0091] Training data is extracted from the training dataset and input into the skeletal keypoint regression neural network model for training. Through backpropagation using the loss function, the sum of the loss functions continuously decreases and tends to stabilize, at which point the model training ends, thus obtaining the target skeletal keypoint regression neural network model. Here, the loss function is the distance loss function L1 loss, and the sum of the loss functions is the sum of the distances between the 14 skeletal keypoints and the driver's center point.

[0092] Step 4: Input the images of the extended regions of the left and right hands into the target hand region segmentation network model to obtain the mask regions of the left and right hands. Calculate the mask iou of the mask regions of the left and right hands and the mask region of the steering wheel. If the mask iou is greater than the set threshold, it is determined that the hand is on the steering wheel; otherwise, it is determined that the hand is off the steering wheel.

[0093] Step 4 specifically includes:

[0094] Step 4.1: Obtain the images of the left and right hand outward expansion regions output in Step 3, and resize the images to the input size (256*256) of the target hand region segmentation network model.

[0095] Step 4.2: Input the resized extended regions of the left and right hands into the target hand region segmentation network model, and obtain the mask regions of the left and right hands through the target hand region segmentation network model.

[0096] Step 4.3: Calculate maskou for the mask areas of the left and right hands and the mask area of ​​the steering wheel obtained in Step 1. If the mask iou is greater than the set threshold, the corresponding hand is considered to be on the steering wheel; otherwise, the hand is considered to be off the steering wheel.

[0097] The specific implementation method for step 4.3 is as follows:

[0098] Step b1: Obtain the image size of the steering wheel mask area obtained in Step 1, construct a binary array of the same size, and map the segmented left and right hand masks into the binary array;

[0099] Step b2: Convert the coordinates of the steering wheel marking points annotated in Step 1 into a binary array using OpenCV methods;

[0100] Step b3: Calculate the mask iou of the mask area of ​​the steering wheel and the mask areas of the left and right hands respectively. If the mask iou is greater than the set threshold, this algorithm sets it to 0.3, which means the hands are considered to be on the steering wheel; otherwise, the hands are considered to be off the steering wheel.

[0101] The formula for calculating mask IOU is as follows:

[0102] mask iou=(mask_hand∩maks_pan) / (mask_hand∪msk_pan)

[0103] In the formula, mask_hand represents the mask area of ​​the hand, and mask_pan represents the mask area of ​​the steering wheel.

[0104] This method can clearly determine which hand the driver took off the steering wheel, and can accurately determine whether the driver's driving behavior constitutes a violation during the test. For example... Figure 4 The diagram illustrates the correct way to hold the steering wheel with both hands and the incorrect way to hold it with only one hand.

[0105] The target hand region segmentation network model mainly consists of a ResNet18 network model and an attention mechanism network layer. The ResNet18 network model has the following parameters: 18 convolutional layers, 18 activation layers, 1 pooling layer, and 1 fully connected layer. The attention mechanism network layer encodes rich spatial and detailed information, reweights the contextual information, and fuses feature information.

[0106] The training process of the target hand region segmentation network model is as follows:

[0107] Step c1: Use the annotation tool to annotate the left and right hand regions in the image with driver markings, and generate a binarized annotated image as training labels;

[0108] Step c2: Input the training labels into the constructed hand region segmentation network model for training. Through continuous backpropagation, convergence is achieved when the total loss function value tends to stabilize, thus obtaining the target hand region segmentation network model.

[0109] The total loss includes pixel category loss (also known as "primary loss") and auxiliary loss, both of which are ultimately output by the softmax loss function, showing the loss value for each sample pixel in the training set.

[0110]

[0111] In the formula, loss represents the total loss, N represents the number of samples, and i and j both represent samples, i = 1, 2, ..., N, j = 1, 2, ..., N, L i denoted by , represents the loss of the softmax loss function at each sample pixel, and p represents the network's predicted output.

[0112]

[0113] In the formula, L represents the sample loss, α is used to balance the primary and secondary losses, and takes a value between 0 and 1. p Indicates pixel category loss, l i Let X represent the auxiliary loss, W represent the pixel predicted by the network, i represent the i-th pixel, i = 1, 2, ..., K, and K represent the number of pixels.

[0114] Furthermore, the embodiments also disclose an electronic device, which mainly includes a processor and a memory. The processor is mainly used to call the computer program stored in the memory and execute the vision-based hand detachment detection method provided in the above embodiments, which will not be described in detail here.

[0115] Finally, it should be noted that although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and guiding, and not restrictive. Those skilled in the art, guided by this specification, can make many other forms without departing from the scope of protection of the claims of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for detecting hand disengagement from a steering wheel based on vision, characterized in that, The application relates to a method for detecting whether a driver's hand is on a steering wheel, and a device thereof. The method comprises the following steps: Step 1: acquiring image data only including a steering wheel picture, outlining the specific position of the steering wheel in a real-time image by marking points, obtaining a mask area of the steering wheel and the coordinates of the marking points; Step 2: acquiring image data of a vehicle interior scene including the steering wheel and the driver in real time, and obtaining the position data of the driver by a target driver detection neural network model; Step 3: inputting the position data of the driver into a target skeleton key point regression neural network model to regress the position data of the skeleton key points, extracting the position coordinates of the left and right wrist points in the skeleton key points, and respectively performing outer expansion according to a preset requirement with the position coordinates of the left and right wrist points as the center to obtain left and right hand outer expansion area images; Step 4: inputting the left and right hand outer expansion area images into a target hand region segmentation network model to obtain the mask areas of the left and right hands, mapping the mask areas of the left and right hands to the original image with the mask area of the marked steering wheel, respectively calculating the mask iou of the mask area of the steering wheel and the mask area of the left and right hands, and if the mask iou is greater than a first threshold value, it is determined that the corresponding hand is on the steering wheel, otherwise it is determined that the hand is separated from the steering wheel. The target hand region segmentation network model comprises a resnet18 network model and an attention mechanism network layer; the parameters of the Resnet18 network model comprise 18 convolution layers, 18 activation layers, 1 pooling layer and 1 fully connected layer; The attention mechanism network layer is used for encoding the context information, reweighting the context information and fusing the feature information; The training process of the target hand region segmentation network model comprises the following steps: using a marking tool to mark the left and right hand regions in the original image with the driver marking points and generating a binary annotation image as a training label; wherein the total loss includes a pixel category loss and an auxiliary loss, and finally unified by a softmax loss function, outputting the pixel point value of each sample in the training set inputting the training label into the constructed hand region segmentation network model for training until the total loss function value tends to be stable and converges to obtain the target hand region segmentation network model. : In the formula, The step 1 specifically comprises the following steps: total loss is represented by, N the number of samples is represented by, i and j each sample is represented by, i =1,2,……, N , j =1,2,……, N , L i the result of the softmax loss function at each pixel point is represented by, p the prediction output of the network is represented by; wherein, L represents the loss of the sample, for balancing the pixel class loss and the auxiliary loss, taking values between 0 and 1, l p represents the pixel class loss, l i represents the auxiliary loss, X represents the pixel predicted by the network, W represents the pixel of the actual label, i represents the pixel from the first i pixel, i = 1, 2, …, K , K represents the number of pixels.

2. The detection method of claim 1, wherein, Step 1.1: acquiring image data including a steering wheel picture through a camera fixed in a driving test vehicle; Step 1.2: outlining the specific position of the entire steering wheel in the image by marking points to obtain the mask area of the steering wheel and the coordinates of the marking points. In the step 2, the position data of the driver is obtained by a target driver detection neural network model, and specifically comprises the following steps:

3. The method of claim 1, wherein the detecting is performed by a method selected from the group consisting of mass spectrometry, nuclear magnetic resonance, and chromatography. a plurality of rectangular bounding boxes are used as detection boxes to determine the positions of the drivers and passengers in the vehicle interior scene image, and the position data, confidence and classification result corresponding to the detection boxes are recorded; the classification result comprises two categories of having a driver and passenger and not having a driver and passenger; invalid detection boxes with a confidence lower than a second threshold value are filtered; if the number of the detection boxes is greater than 1, the rightmost detection box in the image is selected as the driver detection box, and the position data corresponding to the driver detection box is output. The detection box is a rectangular bounding box, the position of which is determined by the coordinates of the upper left corner and the lower right corner of the rectangular bounding box, or by the coordinates of the upper left corner of the rectangular bounding box and the length and width of the rectangle.

4. The detection method of claim 3, wherein, ​ 5. The detection method as described in claim 3, characterized in that, In step 2, the target driver detection neural network model is a YOLOv5 neural network model, which adopts a CSP+Focus+SPP structure; wherein: the CSP contains a residual structure of a convolution layer, a BN layer and an activation layer; the Focus contains four slice layers for fusing multiple scales of the input image; and the SPP contains three maximum pooling layers for outputting features of different scales. The training process of the target driver detection neural network model comprises: obtaining a plurality of vehicle interior scene image data including a steering wheel and a driver / passenger picture; outlining a detection box for defining the position of the driver / passenger in the vehicle interior scene image, recording the position coordinates and classification of the detection box to form a label corresponding to the detection box, and combining the vehicle interior scene image and the corresponding label data to form a training data set; The YOLOv5 neural network model constructed is trained through a training data set until the total loss of the YOLOv5 neural network model Loss After convergence tends to be stable, a target driver detection neural network model is obtained. The total loss function of the YOLOv5 neural network model is: Loss = λ 1 L cls + λ 2 L obj + λ 3 L loc In the formula: Loss represents the total loss; λ1, λ 2, represents the balance coefficient; L cls represents the classification loss, including two categories of drivers and non-drivers, and the classification loss of positive samples is calculated by using BCE loss, wherein the positive samples are detection boxes with an IOU greater than a threshold; obj represents the target loss, and the IOU between the detection box predicted by the neural network model and the real detection box is calculated by using BCE loss; L loc represents the positioning loss, and the position loss of positive samples is calculated by using IOU loss.

6. The detection method as described in claim 1, characterized in that, Step 3 specifically comprises: Step 3.1: inputting the position data of the driver into the target skeleton key point regression neural network model to obtain the position coordinates of 14 skeleton key points and the confidence of the corresponding coordinates, and extracting the position coordinates of the left and right wrist points that may belong to the skeleton key points; Step 3.2: filtering the position coordinates with a confidence lower than a third threshold value by the maximum suppression method to obtain the position coordinates of the left and right wrist points; Step 3.3: expanding the left and right wrist point position coordinates by 100 pixel values to obtain the left and right hand expansion region images.

7. The detection method of claim 6, wherein, In step 3, the target skeleton key point regression neural network model comprises a ResNet50 network and an SSTN symmetric space transformation network; the parameters and computational complexity of the ResNet50 network comprise 50 convolution layers, 50 activation layers, 1 pooling layer and 1 fully connected layer; and the SSTN symmetric space transformation network is used for spatial transformation of the driver detection box so that the driver is in the center of the detection box. The training process of the target skeleton key point regression neural network model comprises: marking the positions of the 14 skeleton key points of the driver in the vehicle interior scene image containing the steering wheel and the driver / passenger, recording the position coordinates and the corresponding skeleton key point categories to form training labels, and combining the training labels and the image data as a training data set for the skeleton key point regression neural network model; training the ResNet50 network and the SSTN symmetric space transformation network constructed by the training set until the value of the sum of the loss functions tends to be stable and converges to obtain the target skeleton key point regression neural network model; The loss function is a distance loss function L1 loss, and the sum of the loss functions is the sum of the distances between the 14 skeleton key points and the center point of the driver.

8. The method of claim 1, wherein, Step 4 specifically comprises: Step 4.1: obtaining the left and right hand expansion regions and resizing the left and right hand expansion regions to the input size of the target hand region segmentation network model; Step 4.2: inputting the resized left and right hand expansion regions into the target hand region segmentation network model to obtain the mask regions of the left and right hands through the target hand region segmentation network model; Step 4.3: Constructing a binarization array on the original image with the steering wheel punctuation, converting the coordinates of the punctuation of the steering wheel into the binarization array, and mapping the masks of the left and right hands into the binarization array; Step 4.4: Calculate the mask iou of the steering wheel mask area and the mask areas of the left and right hands respectively. If the mask iou is greater than the first threshold, it is determined that the corresponding hand is on the steering wheel; otherwise, it is determined that the hand is off the steering wheel. The formula for calculating mask iou is as follows: mask iou = ( maks_pan) / (mask_ In the formula (hand), mask_hand represents the mask area of ​​the hand, and mask_pan represents the mask area of ​​the steering wheel.

9. An electronic device, comprising: The application relates to a hand-off steering wheel detection method based on vision, comprising a processor and a memory, wherein the processor is used for calling a computer program stored in the memory, and the method is shown in any one of claims 1 to 8.

Images