Driver operating mobile phone behavior recognition detection method combined with bone key points
By detecting skeletal key points and hand regions in driver images, and combining Alphapose and an improved Yolov5 model, the problem of accurately recognizing driver mobile phone operation behavior is solved, improving the accuracy and efficiency of recognition, and is applicable to complex lighting environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING JIAOTONG UNIV
- Filing Date
- 2022-08-25
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies are prone to missed detections and false detections when detecting whether a driver is using a mobile phone, especially when the driver is wearing a mask, making it difficult to accurately identify facial information, resulting in insufficient accuracy and timeliness of identification.
By combining skeletal keypoint detection, the system detects skeletal keypoints in driver images, especially the angle between the left and right arms. Combined with mobile phone target recognition in the hand area, it determines whether the driver is using a mobile phone. The system employs the Alphapose human pose recognition model and an improved Yolov5 target detection model.
It improves the accuracy and efficiency of recognizing drivers' mobile phone operation behavior, reduces missed detections and false detections, and can accurately identify drivers' mobile phone operation behavior in complex lighting environments.
Smart Images
Figure CN115393829B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of driver behavior and posture recognition technology, specifically to a method for recognizing and detecting driver mobile phone operation behavior by combining skeletal key points. Background Technology
[0002] Although traffic accidents in my country have shown a downward trend in recent years, the total number remains enormous, especially during holidays when toll-free highways, combined with travel for self-driving tours, trips home for family visits, and student travel, lead to a significant increase in traffic flow. Among the main causes of road traffic accidents are inattentive driving, such as operating the steering wheel with one hand, making or receiving phone calls, and checking messages on a mobile phone, and this phenomenon is continuously increasing. Therefore, research on real-time monitoring of drivers' mobile phone use under natural driving conditions to ensure driver safety and the implementation of intelligent safety monitoring for vehicles is crucial.
[0003] Currently, methods for determining driver mobile phone operation mainly fall into two categories: one is to first identify the area around the face and then detect the mobile phone within that area; the other is to expand the detection area and integrate other feature information to comprehensively determine mobile phone operation. The former is a relatively simple detection method and is easily affected by objective factors such as uneven lighting and obstructions, resulting in lower accuracy and a high false negative rate. The latter detection method, which integrates multiple information, can not only identify the behavior of making or receiving calls around the face but also the behavior of checking mobile phone messages in the front area of the body, making it a detection method more in line with practical application scenarios. However, the fact that drivers of public transport vehicles wear masks limits most detection methods that extract features from the facial region. Furthermore, the latter detection method requires facial recognition information as a key detection component. Therefore, while the latter method is more in line with practical application scenarios, its reliability is not outstanding.
[0004] In summary, existing technologies for detecting whether drivers are using mobile phones are prone to missed detections and false positives. Therefore, how to accurately identify drivers' mobile phone use during actual driving, thereby reducing traffic safety hazards, has become an urgent problem to be solved. Summary of the Invention
[0005] To address the shortcomings of the existing technology, this invention provides a method for recognizing and detecting driver's mobile phone operation behavior by combining skeletal key points, which can accurately identify the driver's mobile phone operation behavior during actual driving.
[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:
[0007] A method for recognizing and detecting driver mobile phone operation behavior by combining skeletal key points includes the following steps:
[0008] S1. Obtain the driver image to be detected, wherein the driver image includes the driver's entire body area and hand area;
[0009] S2. Select candidate boxes for the driver image, selecting the entire human body area and hand area;
[0010] S3. Using a preset human posture recognition model, perform skeletal key point detection on the entire human body area of the driver image, obtain the coordinates of the preset skeletal key points, and calculate the included angle of the left and right arms based on the coordinates of the skeletal key points; using a preset target detection model, perform mobile phone target recognition on the hand area of the driver image; based on the mobile phone target recognition result of the hand area, the included angle of the left and right arms and the preset included angle conditions, determine whether there is a mobile phone use behavior. If there is, proceed to S4; if not, return to S1.
[0011] S4. Output the driver's use of a mobile phone as the detection result.
[0012] Basic scheme principles and effects:
[0013] In existing technologies, detection methods that integrate multiple information sources are suitable for practical application scenarios. However, because many drivers, especially those in public transport vehicles, are required to wear masks, it is difficult to identify facial information, which is a key detection element. Therefore, current efforts by those skilled in the art are mainly focused on "how to obtain the driver's facial information as accurately and comprehensively as possible while wearing a mask." Given that the overall solution is generally considered effective and reasonable, overcoming the difficulties in the solution would allow for its smooth and comprehensive implementation. However, when the area below the nose is obscured by a mask, analyzing the driver's facial information solely based on the exposed portion and the real-time characteristics of the mask is quite difficult with current facial recognition technology. Therefore, there has been little breakthrough in accuracy and timeliness for those skilled in the art.
[0014] The inventors broke free from the conventional thinking of those skilled in the art, analyzing the issue from a completely new perspective. Specifically, they considered whether it was possible to first analyze the inevitable bodily reactions a driver would have when using a mobile phone. If this reaction occurred while the driver was holding the phone, then it was undeniable that the driver was using the phone. Based on this premise, the inventors divided the act of using a mobile phone into two parts: first, the inevitable bodily reactions that occur when using a mobile phone, which are easily detected; and second, holding the phone. After extensive analysis and verification, the inventors ultimately found that their proposed technical approach was not only highly reliable but also relatively easy to implement. The final conclusion was that due to the driver's posture and body orientation, the angle between their left and right arms (i.e., the degree of arm bending) would inevitably meet a certain angle condition when using a mobile phone. Therefore, if the driver happened to be holding the phone when this angle condition was met, it could be undeniably confirmed that the driver was using the phone.
[0015] Based on the above analysis, the inventors proposed this method. After acquiring the image of the driver to be detected, the entire human body region and the hand region are selected. Then, the angle between the driver's arms is analyzed through the entire human body region to see if it meets the required angle conditions. The presence of a mobile phone target is identified through the hand region. If both conditions are met simultaneously, it indicates that all conditions for using a mobile phone are met. Therefore, the driver's behavior of using a mobile phone can be output as the detection result. Compared with the prior art, this invention provides a completely new technical approach, which has good reliability, is relatively convenient to implement, and has high recognition efficiency.
[0016] In summary, this method can accurately identify a driver's mobile phone operation during actual driving.
[0017] Preferably, the key skeletal points include the left and right wrists, left and right elbows, and left and right shoulders.
[0018] Beneficial effects: These six skeletal key points are the ends or movable joints of the arms. This setup can minimize the number of skeletal key points while ensuring the accuracy of the angle between the two arms, thereby maximizing the detection efficiency of the angle between the two arms and ensuring the timeliness of the overall detection results.
[0019] Preferably, the included angle conditions include: the included angle of one arm is within 55°, and the included angles of both the left and right arms are both outside 55° but the difference in the included angles between the two arms is greater than 30°.
[0020] Beneficial Effects: Drivers typically use mobile phones by holding them to their ear to answer calls, view messages, or send messages. The postures for viewing and sending messages are very similar and can be analyzed as a single mode (i.e., phone operation). When answering a call, one hand needs to hold the phone to the ear. In this case, the angle of the arm holding the phone is undoubtedly within 55°, allowing for stable and relatively comfortable phone operation. When viewing / sending messages, the angle of the driver's arm holding the phone can be calculated to ensure a clear and complete view of the information on the screen. Since the other hand is used to control the steering wheel, the difference in angle between the two arms when viewing / sending messages can also be calculated. Based on this approach, the inventors conducted research and analysis, finding that while the angles of both arms are outside 55°, the difference in angle between the two arms is greater than 30°, consistent with the aforementioned single-handed phone operation scenario. With this setup, the method can accurately identify a driver's actions of answering a phone call or viewing / sending a message with their ear, ensuring the comprehensiveness and accuracy of the identification results and minimizing missed or false detections.
[0021] Preferably, in S4, if the angle of one of the driver's arms is within 55°, then the driver answering the phone with his ear is taken as the detection result; if the angles of both arms are outside 55° but the difference in angles between the two arms is greater than 30°, then the driver operating the mobile phone is taken as the detection result.
[0022] Beneficial effects: When the vehicle's intelligent system provides safety reminders to the driver, it can select the appropriate method to provide the reminder based on the driver's current situation.
[0023] Preferably, in S2, before selecting candidate boxes for the driver image, the driver image is preprocessed; the preprocessing includes image homomorphic filtering.
[0024] Beneficial effects: Preprocessing enhances details in dark areas, thereby reducing lighting variations and sharpening edges or details. It enables the extraction of dark area feature details under complex lighting conditions, uncovering richer feature information.
[0025] Preferably, the human posture recognition model is the Alphapose human posture recognition model.
[0026] Beneficial effects: Alphapose's human posture recognition model is relatively mature and can ensure the accuracy of recognizing the angle between the driver's arms.
[0027] Preferably, the target detection model adopts the Yolov5 model structure, including a feature extraction network unit, a feature enhancement network unit, and a feature classification network unit;
[0028] The feature extraction network unit is used to extract three effective feature layers and generate corresponding effective feature maps respectively;
[0029] The feature network unit comprises three feature enhancement network layers, each receiving a valid feature map from the feature extraction network. Each feature enhancement network layer includes an image scaling module, a convolution module, a downsampling module, a feature fusion module, and an upsampling module. The image scaling module scales the valid feature map to obtain a magnified feature image, an original-sized feature image, and a reduced-sized feature image for each valid feature map, and sends these to the corresponding convolution module. The convolution module performs convolution processing on feature images of different sizes. The downsampling module downsamples the convolutional feature images and sends the downsampling results to the feature fusion module. The feature fusion module uses a structured feature enhancement module (SFEM) to fuse the downsampled results of the convolutions of the three sizes of feature images corresponding to each valid feature map. The upsampling module upsamples the feature fusion results corresponding to each valid feature map and sends the upsampling results to the feature classification network unit.
[0030] The feature classification network unit is used to classify and identify targets based on the upsampling results corresponding to each valid feature map.
[0031] Preferably, the working process of the target detection model includes:
[0032] This is used to extract three effective feature layers and generate corresponding effective feature maps for each.
[0033] Each effective feature map is scaled to obtain a magnified feature image, an original feature image, and a reduced feature image for each effective feature map. Convolution and downsampling are then performed on the feature images of different sizes. Finally, the convolution and downsampling results of the three feature images of each effective feature map are fused and then upsampled.
[0034] Finally, target classification and recognition are performed based on the upsampling results corresponding to each valid feature map.
[0035] Beneficial Effects: In real-world applications, mobile phones are relatively small targets, and they are easily obscured by hands or heads during phone calls, making detection using traditional Yolov5 models quite challenging. This invention introduces the SFEM structured feature enhancement module into the Yolov5 model during the feature extraction stage. This module continuously performs convolution and downsampling, fusing features from different backbone networks, and then continuously upsampling and fusing these features with previous ones. Through multiple iterations, feature information fusion is achieved more effectively, thus ensuring a higher success rate in identifying mobile phone targets.
[0036] Preferably, S3 includes:
[0037] S31. Using a preset human posture recognition model, the human body region of the driver image is subjected to skeletal key point detection, the coordinates of the preset skeletal key points are obtained, and the included angle of the left and right arms is calculated based on the coordinates of the skeletal key points; if the included angle of the left and right arms meets the preset included angle conditions, then proceed to S32, otherwise return to S1.
[0038] S32. Using a preset target detection model, perform mobile phone target recognition on the hand area of the driver's image. If a mobile phone target is detected, determine that there is a mobile phone use behavior and proceed to S4. If no mobile phone target is detected, return to S1.
[0039] Preferably, S3 includes:
[0040] S31. Using a preset target detection model, perform mobile phone target recognition on the hand area of the driver's image. If a mobile phone target is detected, proceed to S32; otherwise, return to S1.
[0041] S32. Using a preset human posture recognition model, the human body region of the driver image is subjected to skeletal key point detection to obtain the coordinates of the preset skeletal key points, and the included angle of the left and right arms is calculated based on the coordinates of the skeletal key points. If the included angle of the left and right arms meets the preset included angle conditions, it is determined that there is a mobile phone use behavior and the process proceeds to S4; otherwise, it returns to S1.
[0042] Preferably, S3 includes:
[0043] S31. Using a preset human posture recognition model, the human body region of the driver image is subjected to skeletal key point detection, the coordinates of the preset skeletal key points are obtained, and the included angle between the left and right arms is calculated based on the coordinates of the skeletal key points.
[0044] S32. Using a preset target detection model, perform mobile phone target recognition on the hand area of the driver's image;
[0045] S33. If the included angle of the left and right arms meets the preset included angle conditions and the mobile phone target is detected, then proceed to S4; otherwise, return to S1. Attached Figure Description
[0046] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:
[0047] Figure 1 This is a schematic diagram of the overall network structure for driver mobile phone usage behavior recognition and detection in Example 1.
[0048] Figure 2 This is a flowchart illustrating the human pose recognition model in Embodiment 1 of the present invention.
[0049] Figure 3 This is a schematic diagram of the improved Yolov5 structure in Embodiment 1 of the present invention. Detailed Implementation
[0050] The following detailed explanation illustrates the specific implementation methods:
[0051] In existing technologies, detection methods that integrate multiple information sources are suitable for practical application scenarios. However, because many drivers, especially those in public transport vehicles, are required to wear masks, it is difficult to identify facial information, which is a key detection element. Therefore, current efforts by those skilled in the art are mainly focused on "how to obtain the driver's facial information as accurately and comprehensively as possible while wearing a mask." Given that the overall solution is generally considered effective and reasonable, overcoming the difficulties in the solution would allow for its smooth and comprehensive implementation. However, when the area below the nose is obscured by a mask, analyzing the driver's facial information solely based on the exposed portion and the real-time characteristics of the mask is quite difficult with current facial recognition technology. Therefore, there has been little breakthrough in accuracy and timeliness for those skilled in the art.
[0052] The inventors broke free from the conventional thinking of those skilled in the art, analyzing the issue from a completely new perspective. Specifically, they considered whether it was possible to first analyze the inevitable physical reactions a driver would have when using a mobile phone. If this physical reaction occurred while the driver was holding the phone, then it was undeniable that the driver was using the phone. Based on this premise, the inventors divided the act of using a mobile phone into two parts: first, the inevitable physical reactions that occur when using a mobile phone, which are easily detected; and second, holding the phone. After extensive analysis and verification, the inventors ultimately found that their proposed technical approach was not only highly reliable but also relatively easy to implement. The final conclusion was that, due to the driver's posture and body orientation, the angle between their left and right arms (i.e., the degree of arm bending) would inevitably meet a certain angle condition when using a mobile phone. Therefore, if the driver was holding the phone when this angle condition was met, it could be undeniably confirmed that the driver was using the phone. The specific method is as follows:
[0053] Example 1
[0054] like Figure 1 As shown, this embodiment discloses a method for recognizing and detecting driver's mobile phone operation behavior by combining skeletal key points, including the following steps:
[0055] S1. Obtain the image of the driver to be detected. The image includes the driver's entire body area and hand area. In practice, the driver image can be obtained through an in-vehicle camera. The specific installation location of the in-vehicle camera can be set according to the vehicle model and the specific spatial layout of the driver's cabin, which will not be elaborated here.
[0056] S2. After preprocessing the driver image, candidate bounding boxes are selected to select the entire human body region and hand region; the preprocessing includes image homomorphic filtering and spatial transformation processing.
[0057] Homomorphic filtering enhancement is an image processing method that combines frequency filtering and spatial grayscale transformation. It improves image quality by compressing the brightness range and enhancing contrast, based on the image's illuminance / reflectance model as a frequency domain processing method.
[0058] An image can be viewed as consisting of two parts:
[0059] f(x,y)=f i (x,y)f r (x,y);
[0060] Among them, f i The illumination components representing different spatial locations are concentrated in the low-frequency part of the image; fr It represents the reflection component of objects reflected to the human eye, and is rich in high-frequency components.
[0061] First, perform a logarithmic transformation on the original graph to obtain two additive components, namely:
[0062] lnf(x,y)=ln f i (x,y)+lnf r (x,y);
[0063] Then, performing a Fourier transform on the logarithmic graph yields its corresponding frequency domain representation:
[0064] DFT[lnf(x,y)]=DFT[ln f i (x,y)]+DFT[ln f r [(x,y)];
[0065] Design another frequency domain filter H(u,v) to perform frequency domain filtering on the logarithmic image; then perform an inverse Fourier transform to return the spatial logarithmic image; finally, take the exponent to obtain the spatial domain filtering result.
[0066] Preprocessing enhances details in dark areas, thereby reducing variations in lighting and sharpening edges or details. It enables the extraction of dark area feature details under complex lighting conditions, uncovering richer feature information.
[0067] S3. Using a preset human posture recognition model, perform skeletal key point detection on the entire human body area of the driver image, obtain the coordinates of the preset skeletal key points, and calculate the included angle of the left and right arms based on the coordinates of the skeletal key points; using a preset target detection model, perform mobile phone target recognition on the hand area of the driver image; based on the mobile phone target recognition result of the hand area, the included angle of the left and right arms and the preset included angle conditions, determine whether there is a mobile phone use behavior. If there is, proceed to S4; if not, return to S1.
[0068] In specific implementation, S3 includes:
[0069] S31. Using a preset human posture recognition model, the human body region of the driver image is subjected to skeletal key point detection, the coordinates of the preset skeletal key points are obtained, and the included angle between the left and right arms is calculated based on the coordinates of the skeletal key points.
[0070] S32. Using a preset target detection model, perform mobile phone target recognition on the hand area of the driver's image;
[0071] S33. If the included angle of the left and right arms meets the preset included angle conditions and the mobile phone target is detected, then proceed to S4; otherwise, return to S1.
[0072] The human pose recognition model mentioned above is the Alphapose human pose recognition model. The working process of the human pose recognition model is as follows: Figure 2 As shown in the figure. Where AA1 is the included angle of the left arm and AA2 is the included angle of the right arm.
[0073] Using the Alphapose human pose recognition model, skeletal key points at the driver's left and right wrists, elbows, and shoulders are detected within the human body region frame. The coordinates of these six key points are obtained, and the angle between the left and right arms is calculated accordingly. These six skeletal key points are the ends or movable joints of the arms. This setup minimizes the number of skeletal key points while ensuring accuracy in recognizing the angle between the arms, maximizing detection efficiency and ensuring timely overall results. Taking the left arm as an example, the wrist, elbow, and shoulder can be considered as the three vertices of a triangle, and the forearm and upper arm as the two sides. The angle between the left arm and upper arm can then be considered as the angle between these two sides. Calculating this angle using standard geometry, given the coordinates of the three vertices, will not be elaborated further here.
[0074] The angle conditions include: the angle of one arm is within 55°, and the angles of both arms are outside 55° but the difference in angle between the two arms is greater than 30°. Drivers typically use mobile phones to answer calls, view messages, or send messages. The postures for viewing and sending messages are very similar and can be analyzed as the same mode (i.e., operating the phone). When answering a call with one's ear, one hand needs to hold the phone to the ear. In this case, the angle of the arm holding the phone will undoubtedly be within 55°, allowing for a stable and relatively comfortable answering. When viewing / sending messages, to clearly see the information on the phone screen, the angle of the driver's arm holding the phone can be calculated. Since the other hand needs to control the steering wheel, the difference in angle between the two arms when viewing / sending messages can also be calculated. Based on this idea, the inventors conducted research and analysis, discovering that the angle between the left and right arms was both beyond 55°, but the difference in the angle between the two arms was greater than 30°, which matches the aforementioned situation of single-handed mobile phone operation. With this setup, this method can accurately identify the driver's actions of answering calls and viewing / sending messages with their ear, ensuring the comprehensiveness and accuracy of the identification results and minimizing missed or false detections.
[0075] In practice, the object detection model adopts the Yolov5 model structure, which includes feature extraction network units, feature enhancement network units, and feature classification network units. For example... Figure 3 As shown.
[0076] Feature extraction network unit (i.e. Figure 3 The Resblock_body layer in the image is used to extract three effective feature layers and generate corresponding effective feature maps. These three effective feature layers have different convolutional depths from top to bottom. Assuming the input image is 640, 640, 3; after the first effective feature layer, the output is 40, 40, 512; after the second effective feature layer, the output is 20, 20, 1024; and after the third effective feature layer, the output is 80, 80, 256. The three effective feature layers progressively increase the fineness of image feature extraction, obtaining increasingly richer semantic information, all representing deep image features.
[0077] Feature network unit (i.e.) Figure 3 The Neck layer (in the network) includes three feature enhancement network layers, each receiving a valid feature map from the feature extraction network. Each feature enhancement network layer includes an image scaling module, a convolution module, a downsampling module, a feature fusion module, and an upsampling module. The image scaling module scales the valid feature map to obtain a magnified feature image, an original feature image, and a reduced feature image for each valid feature map, and sends them to the corresponding convolution module. The convolution module performs convolution processing on feature images of different sizes. The downsampling module downsamples the convolutional feature images and sends the downsampling results to the feature fusion module. The feature fusion module uses a structured feature enhancement module (SFEM) to fuse the downsampled results of the convolution of the three sizes of feature images corresponding to each valid feature map. The upsampling module upsamples the feature fusion results corresponding to each valid feature map and sends the upsampling results to the feature classification network unit.
[0078] To facilitate understanding, the following explanation is provided: The entire neck layer is actually a process of SFEM feature enhancement, including the preceding convolution and sampling processes. The term SFEM is only used in the frame after convolution because the feature information from the three dimensions obtained earlier is passed and fused in this small module. The fused feature information is then passed to subsequent upsampling processes. The advantage of SFEM is that it scales a feature map to three dimensions, allowing for the extraction of more feature information. Fusing this feature information enables the detection of targets of different sizes.
[0079] Feature classification network unit ( Figure 3 The YoloHead layer in the model is used to classify and identify targets based on the upsampling results corresponding to each valid feature map.
[0080] The working process of the target detection model includes:
[0081] This is used to extract three effective feature layers and generate corresponding effective feature maps for each.
[0082] Each effective feature map is scaled to obtain a magnified feature image, an original feature image, and a reduced feature image for each effective feature map. Convolution and downsampling are then performed on the feature images of different sizes. Finally, the convolution and downsampling results of the three feature images of each effective feature map are fused and then upsampled.
[0083] Finally, target classification and recognition are performed based on the upsampling results corresponding to each valid feature map.
[0084] The Yolov5 model in this invention introduces the SFEM structured feature enhancement module during the feature extraction stage. This module continuously performs convolution and downsampling, fusing features from different backbone networks, and then continuously upsamples and fuses these features with previous ones. Through multiple iterations, the fusion of feature information can be better achieved, thereby ensuring a higher success rate in recognizing mobile phone targets.
[0085] A typical YOLOv5 model can be divided into three parts: the Backbone, Feature Pyramid Networks (FPN), and the YOLO Head. The Backbone, also known as the YOLOv5 backbone feature extraction network, primarily extracts features from the input image. These extracted features, called feature layers, are the feature sets of the input image. Three effective feature layers are obtained from the backbone and used in the next step of network construction. The FPN, or enhanced feature extraction network of YOLOv5, fuses the three effective feature layers obtained from the backbone, combining feature information at different scales. The YOLO Head is the classifier and regressor of YOLOv5. It takes the three enhanced effective feature layers, each with width, height, and number of channels, and treats the feature map as a set of feature points, determining whether each feature point corresponds to an object. However, in real-world applications, mobile phone targets are relatively small and easily obstructed by hands or heads during phone calls, increasing the detection difficulty. Therefore, this invention improves the feature extraction stage by introducing a Structured Feature Enhancement Module (SFEM) to better integrate feature information. The SFEM module utilizes Conditional Random Fields (CRF) for feature optimization, transforming the image into three sizes: one magnified to twice its original size, one halved, and one remaining at its original size. These three images, each with a different number of downsampling submodules, ensure that the features have the same size when input into the SFEM structure. The three images at different resolutions are passed through three network branches, undergoing continuous convolution and downsampling to fuse features from different branches. The features are then continuously upsampled and fused with previous features. Through multiple iterations, the fusion of feature information is improved. This setup ensures a high success rate in recognizing mobile phone targets.
[0086] S4. Output the driver's mobile phone use as a detection result. Specifically, if the angle of one of the driver's arms is within 55°, the result is that the driver is directly answering a call with their ear facing the phone; if the angles of both arms are beyond 55° but the difference in angles is greater than 30°, the result is that the driver is operating the mobile phone. With this setting, the vehicle's intelligent system can select the most appropriate method to remind the driver based on their current behavior when issuing safety warnings.
[0087] It should be noted that, to ensure the recognition of driver images applicable to various shooting angles in this invention, before training the models used in this invention, in addition to homomorphic filtering of the training images, spatial transformation processing can also be performed using a spatial transformation network when preparing the training images. The Spatial Transform Network (STN) mainly consists of two parts: a Localization Net and an affine transformation network. The former is designed to generate the latter. The characteristic of this network is its ability to adaptively perform spatial transformations and alignments on data (including translation, scaling, rotation, and other geometric transformations) according to classification or other tasks. By translating and performing left-right symmetry transformations on the images, and then cropping, randomly distributing, and stitching them together, training samples at various angles can be constructed. This ensures the applicability of the trained models.
[0088] In this method, after acquiring the image of the driver to be detected, the entire human body area and the hand area are selected. Then, the angle between the driver's arms is analyzed using the entire human body area to determine if it meets the required angle conditions. The presence of a mobile phone target is identified using the hand area. If both conditions are met simultaneously, it indicates that all conditions for using a mobile phone are met. Therefore, the driver's mobile phone use can be output as the detection result. Compared with existing technologies, this invention provides a novel technical approach with good reliability, convenient implementation, and high recognition efficiency. This method can accurately identify a driver's mobile phone operation during actual driving.
[0089] Example 2
[0090] Compared with Embodiment 1, the only difference in this embodiment is that S3 includes:
[0091] S31. Using a preset human posture recognition model, the human body region of the driver image is subjected to skeletal key point detection, the coordinates of the preset skeletal key points are obtained, and the included angle of the left and right arms is calculated based on the coordinates of the skeletal key points; if the included angle of the left and right arms meets the preset included angle conditions, then proceed to S32, otherwise return to S1.
[0092] S32. Using a preset target detection model, perform mobile phone target recognition on the hand area of the driver's image. If a mobile phone target is detected, determine that there is a mobile phone use behavior and proceed to S4. If no mobile phone target is detected, return to S1.
[0093] Example 3
[0094] Compared with Embodiment 1, the only difference in this embodiment is that S3 includes:
[0095] S31. Using a preset target detection model, perform mobile phone target recognition on the hand area of the driver's image. If a mobile phone target is detected, proceed to S32; otherwise, return to S1.
[0096] S32. Using a preset human posture recognition model, the human body region of the driver image is subjected to skeletal key point detection to obtain the coordinates of the preset skeletal key points, and the included angle of the left and right arms is calculated based on the coordinates of the skeletal key points. If the included angle of the left and right arms meets the preset included angle conditions, it is determined that there is a mobile phone use behavior and the process proceeds to S4; otherwise, it returns to S1.
[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.
Claims
1. A method for recognizing and detecting driver mobile phone operation behavior by combining skeletal key points, characterized in that, Includes the following steps: S1. Obtain the driver image to be detected, wherein the driver image includes the driver's entire body area and hand area; S2. Select candidate boxes for the driver image, selecting the entire human body area and hand area; S3. Using a preset human posture recognition model, perform skeletal key point detection on the entire human body area of the driver image, obtain the coordinates of the preset skeletal key points, and calculate the included angle of the left and right arms based on the coordinates of the skeletal key points; using a preset target detection model, perform mobile phone target recognition on the hand area of the driver image; based on the mobile phone target recognition result of the hand area, the included angle of the left and right arms and the preset included angle conditions, determine whether there is a mobile phone use behavior. If there is, proceed to S4; if not, return to S1. The included angle conditions include: the included angle of one arm is within 55°, and the included angles of both arms are outside 55° but the difference in the included angles of the two arms is greater than 30°. S4. Output the driver's use of a mobile phone as the detection result; S3 includes: S31. Using a preset human posture recognition model, the human body region of the driver image is subjected to skeletal key point detection, the coordinates of the preset skeletal key points are obtained, and the included angle between the left and right arms is calculated based on the coordinates of the skeletal key points. S32. Using a preset target detection model, perform mobile phone target recognition on the hand area of the driver's image; S33. If the included angle of the left and right arms meets the preset included angle conditions and the mobile phone target is detected, then proceed to S4; otherwise, return to S1.
2. The driver's mobile phone operation behavior recognition and detection method combining skeletal key points as described in claim 1, characterized in that: In S4, if the angle of one of the driver's arms is within 55°, the driver is taking the phone call with his ear as the detection result; if the angles of both arms are outside 55° but the difference in angles between the two arms is greater than 30°, the driver is taking the phone operation as the detection result.
3. The driver's mobile phone operation behavior recognition and detection method combining skeletal key points as described in claim 1, characterized in that: In S2, before selecting candidate boxes for the driver image, the driver image is preprocessed; the preprocessing includes image homomorphic filtering.
4. The driver's mobile phone operation behavior recognition and detection method combining skeletal key points as described in claim 1, characterized in that: The human posture recognition model is the Alphapose human posture recognition model.
5. The driver's mobile phone operation behavior recognition and detection method combining skeletal key points as described in claim 1, characterized in that: The target detection model adopts the Yolov5 model structure, including feature extraction network units, feature enhancement network units, and feature classification network units; The feature extraction network unit is used to extract three effective feature layers and generate corresponding effective feature maps respectively; The feature network unit consists of three feature enhancement network layers, each receiving a valid feature map from the feature extraction network; Each feature enhancement network layer includes an image scaling module, a convolution module, a downsampling module, a feature fusion module, and an upsampling module. The image scaling module scales the size of the effective feature maps, obtaining a magnified feature image, an original-size feature image, and a reduced-size feature image for each effective feature map, and sends them to the corresponding convolution module. The convolution module performs convolution processing on feature images of different sizes. The downsampling module downsamples the convolutional feature images and sends the downsampling results to the feature fusion module. The feature fusion module uses the Structured Feature Enhancement Module (SFEM) to perform feature fusion on the convolution and downsampling results of the three sizes of feature images corresponding to each effective feature map. The upsampling module is used to upsample the feature fusion result corresponding to each valid feature map and send the upsampling result to the feature classification network unit. The feature classification network unit is used to classify and identify targets based on the upsampling results corresponding to each valid feature map.
6. The driver's mobile phone operation behavior recognition and detection method combining skeletal key points as described in claim 5, characterized in that: The working process of the target detection model includes: This is used to extract three effective feature layers and generate corresponding effective feature maps for each. Each effective feature map is scaled to obtain a magnified feature image, an original feature image, and a reduced feature image for each effective feature map. Convolution and downsampling are then performed on the feature images of different sizes. Finally, the convolution and downsampling results of the three feature images of each effective feature map are fused and then upsampled. Finally, target classification and recognition are performed based on the upsampling results corresponding to each valid feature map.