Method and system for detecting hand slipping of steering wheel, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a method and a system for detecting hand release of a steering wheel. The method comprises the following steps: acquiring video frame data of a steering wheel area; positioning a steering wheel image and a hand image from a single-frame image of the video frame data; acquiring a mask image of the steering wheel image and key point position information of the hand image; detecting the single-frame image according to the mask image, the key point position information and a preset judgment threshold to obtain a single-frame image detection result; determining whether a steering wheel slip condition exists based on a single-frame image detection result; extracting a feature vector of a multi-frame image sequence of the video frame data under the condition that the steering wheel is determined not to slip out of the hand according to the single-frame image detection result; and comprehensively detecting whether the steering wheel slips out of the hand according to the feature vector of the multi-frame image sequence and the single-frame image detection result. According to the embodiment of the invention, the defects of a traditional sensor scheme are overcome, and the accuracy, stability and safety of overall detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of automobile technology, and in particular to a method for detecting a hands-off steering wheel, a system for detecting a hands-off steering wheel, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the gradual popularization of L2 intelligent assisted driving, the safety of vehicle driving has received increasing attention. As an intermediate stage in the development of autonomous driving technology, intelligent assisted driving still requires the participation of the driver. Therefore, it is particularly important to monitor the driver's status to ensure the safe driving of the vehicle. The Driver Monitoring System (DMS) was born for this purpose. It improves driving safety and comfort by analyzing the driver's behavior and physiological state in real time.

[0003] In existing driver monitoring systems, fatigue and distraction detection technology has been developed. By installing a fixed camera in the cockpit, the system can capture the driver's facial image in real time and analyze facial features through artificial intelligence (AI) algorithms to determine the driver's fatigue and distraction status. These cameras are usually installed in the middle of the car's A-pillar or above the steering wheel column to ensure that the driver's face can be monitored by the camera throughout the normal driving state.

[0004] Hands-off detection is a key technology to ensure that the driver maintains control of the vehicle. Traditional solutions rely on steering wheel sensors that monitor the position of the driver's hands. By monitoring the contact between the hands and the steering wheel, the system can determine in real time whether the driver has taken their hands off the steering wheel and issue a warning when a hands-off situation is detected to remind the driver to hold the steering wheel again.

[0005] Currently, the mainstream steering wheel sensors include the following three types:

[0006] 1. Torque sensing sensor: This type of sensor determines whether the driver is in control of the vehicle by measuring the change in torque of the steering wheel. When the driver holds the steering wheel, the steering wheel will be subjected to a certain torque force, and the sensor can accurately sense the change in this torque force. If the torque force of the steering wheel decreases or disappears, the system will determine that the driver may have taken his hands off the steering wheel.

[0007] 2. Grip force sensor: This sensor is usually installed on the inside of the steering wheel. It determines whether the driver continues to hold the steering wheel by sensing the pressure or tightness of the driver's hand. If the grip force decreases or disappears, the system will sound an alarm to prompt the driver to re-grip the steering wheel.

[0008] 3. Capacitive inductive sensor: This type of sensor determines whether there is hand contact by measuring the change in capacitance between the driver's hand and the steering wheel. When the driver touches the steering wheel, a slight change in capacitance occurs, and the sensor can detect this change to determine whether the driver is holding the steering wheel. Capacitive inductive sensors have the advantages of high sensitivity and strong adaptability to the environment, and can effectively identify slight touches or detachments.

[0009] Although these detection schemes have some effect in practical applications, they still have certain limitations. For example, hand posture or sensor sensitivity may affect the accuracy of detection. With the popularization of intelligent assisted driving technology, some drivers evade sensor detection by fixing water bottles or other objects on the steering wheel, or even using metal stickers to cover the capacitive sensor of the steering wheel. These behaviors cause drivers to take their hands off the steering wheel for a long time in assisted driving mode, increasing safety risks during driving.

[0010] In view of the limitations of traditional hands-off steering wheel detection solutions, the new generation of hands-off steering wheel detection technology has begun to combine more sensor types and intelligent algorithms to improve the accuracy and reliability of detection. Its goal is to issue warning signals more quickly and accurately when it detects that the driver has potential dangerous driving behavior.

[0011] The current process of hands-off steering wheel detection generally includes the following steps:

[0012] S1: When the vehicle starts moving or reaches a certain speed, the system starts and the steering wheel sensor begins to operate.

[0013] S2: The steering wheel sensor feeds back the detection signal to the vehicle system.

[0014] S3: The vehicle system determines whether the driver has taken the hands off the car based on internal logic and preset calibration data.

[0015] S4: If hands-off behavior is detected, the system emits a prompt sound or other warning signal; otherwise, it continues to detect.

[0016] Since the current hands-off detection system completely relies on sensor feedback signals, it is easy to be circumvented, such as interfering with the operation of the sensor through external means. Therefore, the hands-off detection system that relies solely on sensors has certain reliability issues in practical applications, and needs to be combined with more accurate sensors and algorithm optimization to further improve driving safety. Summary of the invention

[0017] In view of the above problems, embodiments of the present invention are proposed to provide a method for detecting a hands-off steering wheel, a system for detecting a hands-off steering wheel, an electronic device, and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0018] In order to solve the above problems, an embodiment of the present invention discloses a method for detecting a hands-off steering wheel, the method comprising: acquiring video frame data of a steering wheel area; locating a steering wheel image and a hand image from a single-frame image of the video frame data; acquiring a mask image of the steering wheel image and key point position information of the hand image; detecting the single-frame image according to the mask image, the key point position information and a preset judgment threshold to obtain a single-frame image detection result; determining whether a hands-off steering wheel situation exists based on the single-frame image detection result; when it is determined that a hands-off steering wheel situation does not exist based on the single-frame image detection result, extracting a feature vector of a multi-frame image sequence of the video frame data; and comprehensively detecting whether a hands-off steering wheel situation exists based on the feature vector of the multi-frame image sequence and the single-frame image detection result.

[0019] Optionally, the single-frame image is detected according to the mask image, the key point position information and a preset judgment threshold to obtain a single-frame image detection result, including: calculating the proportion information of the finger key points in the single-frame image located in the mask image according to the mask image and the key point position information; and comparing the proportion information with the judgment threshold to obtain the single-frame image detection result.

[0020] Optionally, the determining whether the steering wheel is out of hands based on the single-frame image detection result includes: when the judgment threshold is a maximum ratio of finger key points outside the mask image, determining that the steering wheel is out of hands based on the single-frame image detection result indicating that the proportion information is greater than the judgment threshold; determining that the steering wheel is not out of hands based on the single-frame image detection result indicating that the proportion information is less than or equal to the judgment threshold.

[0021] Optionally, the comprehensive detection of whether the steering wheel is out of hands based on the feature vector of the multi-frame image sequence and the single-frame image detection result includes: merging the feature vector and the normalized data of the key point position information into a feature vector matrix; classifying the feature vector matrix to obtain the classification result of the multi-frame image sequence; when the classification result of the multi-frame image sequence is greater than a preset classification threshold, determining whether there is a single-frame image detection result that satisfies a preset steering wheel hands-off condition in the multi-frame image sequence; when the single-frame image detection result that satisfies the preset steering wheel hands-off condition exists in the multi-frame image sequence, determining that the steering wheel is out of hands.

[0022] Optionally, after classifying the feature vector matrix to obtain the classification result of the multi-frame image sequence, the method further includes: determining that there is no hands-off steering wheel situation when the classification result of the multi-frame image sequence is less than or equal to the preset classification threshold.

[0023] Optionally, after classifying the feature vector matrix to obtain the classification result of the multi-frame image sequence, the method further includes: determining that the steering wheel is not out of hand when the classification result of the multi-frame image sequence is greater than the preset classification threshold and there is no single-frame image detection result in the multi-frame image sequence that satisfies the preset steering wheel out of hand condition.

[0024] Optionally, obtaining the mask image of the steering wheel image and the key point position information of the hand image includes: performing edge detection and ellipse matching processing on the steering wheel image to obtain the mask image; rotating the hand image through corresponding corner points, inputting it into the BlazeHand model with a forward hand posture, and outputting the key point position information.

[0025] An embodiment of the present invention also discloses a steering wheel hands-off detection system, the system comprising: a video frame data acquisition module, used to acquire video frame data of a steering wheel area; an image positioning module, used to locate a steering wheel image and a hand image from a single frame image of the video frame data; a mask image and key point position acquisition module, used to acquire the mask image of the steering wheel image and the key point position information of the hand image; a single frame image detection result determination module, used to detect the single frame image according to the mask image, the key point position information and a preset judgment threshold to obtain a single frame image detection result; a hands-off situation determination module, used to determine whether there is a steering wheel hands-off situation based on the single frame image detection result; a feature vector extraction module, used to extract feature vectors of a multi-frame image sequence of the video frame data when it is determined that there is no steering wheel hands-off situation according to the single frame image detection result; and a hands-off situation detection module, used to comprehensively detect whether there is a steering wheel hands-off situation based on the feature vectors of the multi-frame image sequence and the single frame image detection result.

[0026] Optionally, the single-frame image detection result determination module includes: a proportion information calculation module, used to calculate the proportion information of the finger key points in the single-frame image located in the mask image based on the mask image and the key point position information; a proportion threshold comparison module, used to compare the proportion information with the judgment threshold to obtain the single-frame image detection result.

[0027] Optionally, the hands-off situation determination module is used to determine that the steering wheel is off-hand if the judgment threshold is a maximum ratio of finger key points outside the mask image, based on the single-frame image detection result indicating that the proportion information is greater than the judgment threshold; and to determine that the steering wheel is not off-hand if the single-frame image detection result indicates that the proportion information is less than or equal to the judgment threshold.

[0028] Optionally, the hands-off situation detection module includes: a feature vector merging module, used to merge the feature vector and the normalized data of the key point position information into a feature vector matrix; a vector matrix classification module, used to classify the feature vector matrix to obtain a multi-frame image sequence classification result; a single-frame result judgment module, used to determine whether there is a single-frame image detection result that meets a preset steering wheel hands-off condition in the multi-frame image sequence when the classification result of the multi-frame image sequence is greater than a preset classification threshold; a steering wheel hands-off determination module, used to determine whether there is a steering wheel hands-off situation when there is a single-frame image detection result that meets the preset steering wheel hands-off condition in the multi-frame image sequence.

[0029] Optionally, the system also includes: a steering wheel hands-on determination module, which is used to determine that the steering wheel is not out of hand after the vector matrix classification module classifies the feature vector matrix to obtain a multi-frame image sequence classification result, and if the classification result of the multi-frame image sequence is less than or equal to a preset classification threshold.

[0030] Optionally, the steering wheel not being released from hands determination module is further used to determine that the steering wheel is not being released from hands after the vector matrix classification module classifies the feature vector matrix to obtain a multi-frame image sequence classification result, if the classification result of the multi-frame image sequence is greater than a preset classification threshold, and there is no single-frame image detection result in the multi-frame image sequence that satisfies a preset steering wheel release condition.

[0031] Optionally, the mask image and key point position acquisition module includes: a mask image acquisition module, used to perform edge detection and ellipse matching processing on the steering wheel image to obtain the mask image; a key point acquisition module, used to rotate the hand image through corresponding corner points, input it into the BlazeHand model with a positive hand posture, and output the key point position information.

[0032] An embodiment of the present invention also discloses an electronic device, comprising: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enables the electronic device to execute the method for detecting hands-off steering wheel as described above.

[0033] The embodiment of the present invention further discloses a computer-readable storage medium, wherein the computer program stored in the storage medium enables a processor to execute the above-mentioned method for detecting a hands-off steering wheel.

[0034] The embodiments of the present invention include the following advantages:

[0035] The steering wheel hands-off detection scheme provided in an embodiment of the present invention obtains video frame data of the steering wheel area. Then, the steering wheel image and the hand image are located from the single-frame image of the video frame data. Furthermore, the mask image of the steering wheel image and the key point position information of the hand image are obtained. Next, the single-frame image is detected according to the mask image, the key point position information and the preset judgment threshold to obtain the single-frame image detection result, and based on the single-frame image detection result, it is determined whether the steering wheel hands-off situation exists. When it is determined that the steering wheel hands-off situation does not exist according to the single-frame image detection result, the feature vector of the multi-frame image sequence of the video frame data is extracted, and then it is comprehensively detected whether the steering wheel hands-off situation exists based on the feature vector of the multi-frame image sequence and the single-frame image detection result.

[0036] The embodiment of the present invention not only relies on the detection result of a single-frame image, but also introduces the feature vector of a multi-frame image sequence for comprehensive detection. This detection method combining single-frame images and multi-frame images makes up for the accidental errors that may be caused by only single-frame detection. Compared with the method that relies on a single sensor or simple image recognition, it is more robust and can effectively improve the accuracy of detection. By obtaining the mask image of the steering wheel image and the key point position information of the hand image, the driver's hand state can be more accurately identified. The mask image is an image processing technology that highlights the steering wheel area. Combined with the key point position information of the hand, it can be clear whether the driver's hand is off the steering wheel. This is more accurate than relying on sensors to detect physical signals such as contact force and torque changes, and also reduces the detection error caused by factors such as the driver's hand posture or gloves. In the detection process, a preset judgment threshold is introduced, that is, the hand-off situation is judged by setting a reasonable indicator. The judgment mechanism that combines the single-frame image detection result and the feature vector of the multi-frame image sequence makes the hand-off detection more intelligent. By extracting the feature vector of the multi-frame image sequence, the driver's operating state over a period of time can be continuously monitored, rather than relying solely on a single-frame image at a certain moment. This continuous image sequence analysis can capture the dynamic characteristics of the hands-off behavior and avoid misjudgment caused by detection errors at a certain moment.

[0037] In summary, the embodiments of the present invention not only overcome the shortcomings of traditional sensor solutions, but also improve the accuracy, stability and security of overall detection by introducing multiple technical means such as multi-frame image sequence analysis, combination of mask images and hand key points, and intelligent threshold judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flowchart of the steps of a method for detecting a hands-off steering wheel according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of images collected in a multi-scenario hands-off steering wheel detection solution of a multi-model joint inter-frame verification according to an embodiment of the present invention;

[0040] Figure 3 It is a schematic diagram of an algorithm flow of a multi-scenario hands-off steering wheel detection solution of a multi-model joint inter-frame verification according to an embodiment of the present invention;

[0041] Figure 4a is a structural schematic diagram of the key point positions of a hand according to an embodiment of the present invention;

[0042] Figure 4b is a structural schematic diagram of a hand key point position normalization operation according to an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of the overall structure of a GRU model according to an embodiment of the present invention;

[0044] Figure 6 The present invention is a structural block diagram of a system for detecting a hands-off steering wheel according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] The steering wheel hands-off detection scheme proposed in the embodiment of the present invention detects whether the driver has taken the hands off the steering wheel by acquiring and processing video frame data, and makes a more accurate judgment by combining the analysis of single-frame images and multi-frame image sequences. Specifically, the steering wheel image and hand image are located from the single-frame image of the video frame data, a mask image of the steering wheel image is generated, and the key point position information of the hand image is identified. Then, the single-frame image is detected based on the mask image, the key point position information, and the preset judgment threshold to determine whether the hands are off the steering wheel. In the case that the single-frame image does not show hands-off, the feature vector of the multi-frame image sequence is extracted, and the single-frame image detection result and the feature vector of the multi-frame image sequence are combined to finally determine whether the hands are off the steering wheel. The accuracy, reliability, and anti-interference ability of hands-off detection are improved.

[0047] Reference Figure 1 , shows a flowchart of the steps of a method for detecting a hands-off steering wheel according to an embodiment of the present invention. The method for detecting a hands-off steering wheel can be applied to a detection system for a hands-off steering wheel, referred to as a system. The method for detecting a hands-off steering wheel can specifically include the following steps:

[0048] Step 101, obtaining video frame data of the steering wheel area.

[0049] In an embodiment of the present invention, the system captures video frame data of the driver and steering wheel area in real time through a camera installed in the car. Video frame data refers to the image at each moment in the video stream, and dynamic scenes are recorded by continuously acquiring frames. The acquisition operation here can be real-time acquisition, which means that the system can continuously and synchronously capture video frame data without delay, and can reflect the driver's behavior changes in a timely manner.

[0050] For example, suppose a car equipped with an intelligent assisted driving system starts driving. After the system is started, the camera will continue to record the driver's hands and the steering wheel area. If the camera's frame rate is 30 frames per second (fps), the system will collect 30 images per second for analysis. These images can accurately capture the dynamic changes of the driver's hands. Whether holding the steering wheel, raising the hand, or moving the hand position, the system can respond quickly through these images.

[0051] Step 102: locate the steering wheel image and the hand image from a single frame image of the video frame data.

[0052] In an embodiment of the present invention, each video frame data contains a single frame image, and the system locates the steering wheel and the driver's hand from these single frame images. Positioning refers to finding the position of the target object in the image through an algorithm, which refers to the area of ​​the steering wheel and the hand. It is usually implemented using an object detection algorithm based on deep learning, such as Convolutional Neural Networks (CNN), which can automatically detect and annotate specific objects in an image.

[0053] For example: After the system receives a frame of image, it first needs to determine the position of the steering wheel and the hand. For example, the system locks the area of ​​the circular structure (steering wheel) and the shape of the human hand (such as fingers and palms). If a frame of image shows that the driver's hands are placed at 10 o'clock and 2 o'clock on the steering wheel, the system will accurately mark these two positions in the image. Even if the driver's hand posture changes slightly (for example, the hand slides a few centimeters), the system can still dynamically track these changes.

[0054] Step 103: Obtain the mask image of the steering wheel image and key point position information of the hand image.

[0055] In an embodiment of the present invention, a mask image is a binary image used to mark a specific area in an image. It marks the steering wheel area with white or other colors, while other parts are covered. By generating a mask image of the steering wheel, the system can highlight the outline of the steering wheel more clearly. The key points of the hand image refer to specific joints or bone nodes of the hand, such as fingertips, joint positions, etc. These points are marked by a posture estimation algorithm. The key point position information of the hand can help the system understand the specific posture of the hand and its precise position relative to the steering wheel.

[0056] For example: Assuming that the system detects the steering wheel area, it will generate a mask image to mark the position and shape of the steering wheel. The mask image may be a circular area, representing the area where the steering wheel is located. At the same time, the system uses the hand posture estimation algorithm to analyze the hand image and mark the key positions of the finger joints and palm. For example, the system may mark the top of the index finger, the joint of the thumb, and the center of the palm. These points can help the system accurately determine whether the hand is holding the steering wheel.

[0057] Step 104 , detecting the single-frame image according to the mask image, the key point position information and the preset determination threshold to obtain the single-frame image detection result.

[0058] In an embodiment of the present invention, in this step, the system combines the mask image of the steering wheel image and the key point position information of the hand image, and uses a preset judgment threshold to determine whether the driver is holding the steering wheel or the steering wheel is out of hand. The threshold refers to a certain condition or standard set by the system. When the data exceeds or falls below this value, a specific response will be triggered. Here, the judgment threshold may include multiple parameters such as the distance between the hand and the steering wheel, the contact area, and the grip strength.

[0059] For example: Assume that the judgment threshold set by the system requires that some key points of the driver's hand must be within the mask area of ​​the steering wheel, otherwise it is judged as hands-off. If the system finds that the key points of the driver's hand are outside the mask in a single-frame image, or the contact area of ​​the hand is smaller than the preset judgment threshold, the system will preliminarily judge that this may be hands-off behavior, and record the single-frame image detection result of the single-frame image as hands-off.

[0060] Step 105, determining whether the hands on the steering wheel are off the steering wheel based on the single-frame image detection result.

[0061] In the embodiment of the present invention, in a single-frame image, if the single-frame image detection result shows that the contact between the hand and the steering wheel does not meet the preset conditions (i.e., the hand does not touch the steering wheel or the contact is insufficient), the system will directly determine that the driver has taken his hands off the steering wheel, i.e., the steering wheel is out of hand. The system will immediately issue a prompt or warning to remind the driver to hold the steering wheel again.

[0062] For example, if the key points of the driver's hands are far away from the steering wheel in a single-frame image, and there are not enough key points of the fingers in the mask image, the system will directly determine that the steering wheel is out of hand based on the detection result of this single-frame image. Then, the system will issue a warning sound or display a warning message on the dashboard, prompting the driver to hold the steering wheel tightly again to ensure safety.

[0063] Step 106 , when it is determined based on the single-frame image detection result that the hands on the steering wheel are not released, extract the feature vector of the multi-frame image sequence of the video frame data.

[0064] In an embodiment of the present invention, if it is determined based on the single-frame image detection result that the hands-off behavior has not occurred, that is, the steering wheel has not been taken off the steering wheel, the system will further analyze the multi-frame image sequence. The multi-frame image sequence refers to image data extracted from multiple consecutive video frames, and the system will analyze the long-term interaction between the hands and the steering wheel through these multi-frame image sequences. The system extracts feature vectors from these multi-frame image sequences. The feature vector is a set of numbers used to represent features in the multi-frame image sequence, such as the movement trajectory of the hand, position changes, etc.

[0065] For example, if the detection result of a single frame image indicates that the driver does not take the hands off the steering wheel, the system will extract the hand position and movement trajectory in the next few seconds (for example, 10 frames of images) as feature vectors. By analyzing whether the hand remains in contact with the steering wheel or moves during this period, the system can more comprehensively determine whether the driver takes the hands off the steering wheel.

[0066] Step 107 , comprehensively detecting whether the hands on the steering wheel are off the steering wheel based on the feature vectors of the multi-frame image sequence and the detection results of the single-frame image.

[0067] In the embodiment of the present invention, the system comprehensively analyzes the single-frame image detection result and the feature vector of the multi-frame image sequence. If the single-frame image detection result and the feature vector of the multi-frame image sequence both indicate that the driver's hands are not touching the steering wheel, the system will ultimately determine that the steering wheel is out of hand. Through comprehensive judgment, the system can reduce false positives or false negatives and ensure the accuracy of the detection results.

[0068] For example, a single-frame image detection result shows that the driver's hands temporarily left the steering wheel, but the feature vector of the multi-frame image sequence shows that the hands resumed contact in a short period of time. Therefore, the system will comprehensively judge that there is no hand-off situation on the steering wheel to avoid misjudging it as hand-off due to a short action. On the contrary, if the feature vector of the multi-frame image sequence shows that the hands left the steering wheel, the system will ultimately judge that there is a hand-off situation on the steering wheel and trigger the corresponding warning.

[0069] The steering wheel hands-off detection scheme provided in an embodiment of the present invention obtains video frame data of the steering wheel area. Then, the steering wheel image and the hand image are located from the single-frame image of the video frame data. Furthermore, the mask image of the steering wheel image and the key point position information of the hand image are obtained. Next, the single-frame image is detected according to the mask image, the key point position information and the preset judgment threshold to obtain the single-frame image detection result, and based on the single-frame image detection result, it is determined whether the steering wheel hands-off situation exists. When it is determined that the steering wheel hands-off situation does not exist according to the single-frame image detection result, the feature vector of the multi-frame image sequence of the video frame data is extracted, and then it is comprehensively detected whether the steering wheel hands-off situation exists based on the feature vector of the multi-frame image sequence and the single-frame image detection result.

[0070] The embodiment of the present invention not only relies on the detection result of a single-frame image, but also introduces the feature vector of a multi-frame image sequence for comprehensive detection. This detection method combining single-frame images and multi-frame images makes up for the accidental errors that may be caused by only single-frame detection. Compared with the method that relies on a single sensor or simple image recognition, it is more robust and can effectively improve the accuracy of detection. By obtaining the mask image of the steering wheel image and the key point position information of the hand image, the driver's hand state can be more accurately identified. The mask image is an image processing technology that highlights the steering wheel area. Combined with the key point position information of the hand, it can be clear whether the driver's hand is off the steering wheel. This is more accurate than relying on sensors to detect physical signals such as contact force and torque changes, and also reduces the detection error caused by factors such as the driver's hand posture or gloves. In the detection process, a preset judgment threshold is introduced, that is, the hand-off situation is judged by setting a reasonable indicator. The judgment mechanism that combines the single-frame image detection result and the feature vector of the multi-frame image sequence makes the hand-off detection more intelligent. By extracting the feature vector of the multi-frame image sequence, the driver's operating state over a period of time can be continuously monitored, rather than relying solely on a single-frame image at a certain moment. This continuous image sequence analysis can capture the dynamic characteristics of the hands-off behavior and avoid misjudgment caused by detection errors at a certain moment.

[0071] In summary, the embodiments of the present invention not only overcome the shortcomings of traditional sensor solutions, but also improve the accuracy, stability and security of overall detection by introducing multiple technical means such as multi-frame image sequence analysis, combination of mask images and hand key points, and intelligent threshold judgment.

[0072] In an exemplary embodiment of the present invention, a single-frame image is detected according to a mask image, key point position information and a preset judgment threshold to obtain a single-frame image detection result. An implementation method is to calculate the proportion information of the finger key points in the single-frame image in the mask image according to the mask image and the key point position information; and compare the proportion information with the judgment threshold to obtain the single-frame image detection result. By calculating the proportion information of the key points of the driver's hand in the steering wheel mask image, it is determined whether the driver has maintained control of the steering wheel. If the proportion is lower than the set judgment threshold, the system will determine that the driver may have taken his hands off the steering wheel, thereby further triggering warnings or other safety measures.

[0073] In practical applications, it is necessary to clearly define the mask of the steering wheel. A mask is a binary image used to mark a specific area in an image, where the mask area corresponds to the position of the steering wheel and the rest of the image is masked. In a hands-off steering wheel detection system, the mask is usually generated by a computer vision algorithm that can identify the steering wheel in the image and mark its contour to form a mask. The generation of the mask relies on image segmentation technology, which can accurately distinguish the steering wheel from the background and other irrelevant image elements.

[0074] At the same time, the system will obtain the key point position information of the driver's hand through the hand posture estimation algorithm. These key points usually include the top of the finger, the joints and the center of the palm. The posture estimation algorithm detects the human skeleton in the image and locates the key point position information of the hand through a deep learning model (such as a convolutional neural network). For each frame of the image, the system will extract these key points of the hand in real time to understand the specific position and posture of the hand in space.

[0075] After obtaining the mask image and hand key point information, the system will calculate the proportion of finger key points in the mask image. This proportion represents the distribution of hand key points in the steering wheel area. The specific steps are as follows:

[0076] The system checks the coordinates of each finger key point to determine whether these coordinate points are within the steering wheel mask. Each pixel in the mask has a label value (usually 0 or 1), and the area marked with 1 is the steering wheel area. Therefore, the system only needs to check the pixel value corresponding to each hand key point. If the value is 1, it means that the hand key point is within the steering wheel area.

[0077] Then, the system calculates the number of key points in the mask image among all the finger key points, and calculates the ratio of the number of key points to the total number of hand key points. This ratio is called proportion information.

[0078] For example, if a driver's hand has 5 key points (one for each finger), 3 of which are inside the mask and the other two are outside the mask, then the image's information share is 60% (i.e. 3 / 5).

[0079] In the detection system, the judgment threshold is a preset standard value, which the system will use to compare with the proportion information to determine whether the driver is holding the steering wheel. The threshold is generally set by the developer based on the needs of the actual driving scenario and safety requirements. For example, a reasonable threshold may be 80%, that is, when the proportion of the key points of the hand in the steering wheel area is less than 80%, the system will assume that the driver is not fully holding the steering wheel and may have let go of the steering wheel.

[0080] The threshold can be adjusted according to the specific application scenario. For example, on the highway, the threshold can be set higher (such as 90%) to ensure that the driver has both hands on the steering wheel. When driving at low speed in the city, the threshold can be relatively low (such as 70%) to allow the driver to make some slight hand movements while maintaining control.

[0081] The system compares the calculated ratio information with the judgment threshold to obtain the detection result of a single frame image. If the ratio information is higher than or equal to the judgment threshold, the system will assume that the driver's hands are normally holding the steering wheel in the frame image; if the ratio information is lower than the judgment threshold, the system will assume that the driver may have taken his hands off the steering wheel.

[0082] For example, in a certain frame, suppose the calculated percentage is 75%, and the judgment threshold is set to 80%. Since 75% is lower than 80%, the system will judge that the driver's hands are not completely on the steering wheel in this frame, and may have taken their hands off the steering wheel. On the contrary, if the percentage is 85%, the system will judge that the driver has not taken his hands off the steering wheel and is still in control.

[0083] Compared with the traditional detection method that only relies on the steering wheel sensor, the embodiment of the present invention uses image data for hands-off detection, which can more intuitively and accurately reflect the driver's hand movements. By analyzing the specific position of the fingers, the system can more accurately determine whether the driver actually lets go, rather than relying solely on the contact pressure of the hand or the torque change of the steering wheel. Traditional steering wheel sensors are susceptible to external interference. For example, some drivers cheat the system by placing heavy objects on the steering wheel. The image analysis-based method directly captures hand behavior through vision, which can effectively prevent such cheating. The judgment threshold can also be dynamically adjusted according to different driving scenarios and the driver's personal habits to provide a more flexible detection solution. For example, in different driving modes (high speed, low speed, curves, etc.), the system can adaptively adjust the hands-off judgment criteria to ensure that the driver can maintain proper control in every situation.

[0084] In an exemplary embodiment of the present invention, one implementation method for determining whether there is a hands-off situation of the steering wheel based on the single-frame image detection result is that, when the judgment threshold is the maximum ratio of the finger key points outside the mask image, the single-frame image detection result indicates that the proportion information is greater than the judgment threshold, and it is determined that there is a hands-off situation of the steering wheel; based on the single-frame image detection result indicating that the proportion information is less than or equal to the judgment threshold, it is determined that there is no hands-off situation of the steering wheel. In the steering wheel hands-off detection system, single-frame image detection is a key step, and whether the driver's hands are located in the steering wheel area is determined to determine whether the driver has a hands-off behavior. This implementation method determines whether there is a hands-off behavior based on the proportion of the finger key points outside the mask image. This method sets a judgment threshold (maximum ratio) and compares it with the proportion information in the image. If the proportion information of the hand key points exceeds the judgment threshold, the system will consider that the driver has a hands-off behavior.

[0085] In actual application, after obtaining the mask image and hand key points, calculate the proportion of finger key points outside the mask image. The specific steps are as follows:

[0086] The system checks the coordinates of each hand key point one by one to determine whether they are within the steering wheel area marked by the mask image. This is done by checking whether the pixel value corresponding to the key point is the marked value in the mask image (usually 1). If the pixel value is 1, it means that the key point is within the steering wheel area; if it is 0, it means that the key point is outside the steering wheel.

[0087] The system calculates the number of finger key points that are outside the mask map and compares this number with the total number of finger key points to obtain a ratio information. This ratio information can be used to indicate the degree to which the driver's hands are off the steering wheel.

[0088] For example, assuming that the system detects 5 key points of the driver's hands, 3 of which are located outside the steering wheel, the system will calculate that the proportion of information in the image is 60% (i.e. 3 / 5).

[0089] The judgment threshold is defined as the maximum ratio of finger key points outside the mask image. That is, the system sets a standard. If the proportion of finger key points in a frame image exceeds this ratio, it is considered that the driver may have taken the hands off the phone.

[0090] In single-frame image detection, the system compares the percentage of finger key points outside the steering wheel mask with the preset judgment threshold. If the percentage is greater than or equal to the judgment threshold, the system will determine that the hands are off the steering wheel; otherwise, the system will determine that there is no hands-off situation.

[0091] Specifically, suppose the finger key points in a certain frame account for 75%, and the system's judgment threshold is set to 70%. Because 75% is greater than 70%, the system will determine that the driver has taken his hands off the steering wheel in this image frame. If the proportion is 65%, the system will assume that the driver has not taken his hands off the steering wheel and will continue to monitor the contact status between the driver's hands and the steering wheel.

[0092] Suppose a driver is driving on a highway, and the system monitors the driver's hand status in real time through a camera. At a certain moment, one of the driver's hands temporarily leaves the steering wheel, perhaps to adjust the in-car equipment. The system detects that of the five key points of the driver's hand, four are outside the mask map (accounting for 80%), and the system's judgment threshold is 70%. Therefore, the system determines that the driver may have taken his hands off the steering wheel and sends a warning signal to remind the driver to hold the steering wheel again.

[0093] On the contrary, if only 2 of the hand key points are outside the mask map when the driver performs a similar action (accounting for 40%), the system will assume that the driver's hands are still under control and will not issue any warnings.

[0094] The embodiment of the present invention can more accurately reflect the actual behavior of the driver's hand through image analysis. It not only relies on physical contact, but also analyzes the state of the hand through spatial position. The system has a strong tolerance for slight hand movements of the driver. When the driver adjusts the gesture, the system can judge whether the hand is really out of the hand by the overall proportion of key points, reducing false alarms. The judgment threshold can be adjusted according to different driving environments or driver habits. For example, a higher threshold can be set when driving at high speed, and the threshold can be slightly lower when driving at low speed, which enhances the adaptability of the system.

[0095] In an exemplary embodiment of the present invention, an implementation method for comprehensively detecting whether there is a hands-off steering wheel situation based on the feature vectors of a multi-frame image sequence and the single-frame image detection results is to merge the feature vectors and the normalized data of the key point position information into a feature vector matrix; classify the feature vector matrix to obtain the classification result of the multi-frame image sequence; when the classification result of the multi-frame image sequence is greater than a preset classification threshold, determine whether there is a single-frame image detection result in the multi-frame image sequence that meets a preset hands-off steering wheel condition; when there is a single-frame image detection result in the multi-frame image sequence that meets the preset hands-off steering wheel condition, determine that there is a hands-off steering wheel situation.

[0096] Normalization is a data processing technique used to adjust data of different dimensions to the same range for more accurate comparison. In actual application, the system normalizes the feature vector and the position information of the key points of the hand and combines them into a feature vector matrix. The matrix includes features of multiple dimensions such as the movement trajectory and position change of the hand.

[0097] For example, if the coordinate values ​​of the key points of the hand are large, while the motion trajectory in the feature vector changes little, they can be adjusted to the same range (for example, between 0 and 1) by normalization, which can eliminate the differences between different features. Then, the system integrates these normalized data into a feature matrix for further analysis. For example, each column in the matrix represents the state of the hand in a frame image, and each row represents a different feature, such as the position of the hand, the direction of movement, etc.

[0098] Next, the feature vector matrix is ​​classified to obtain the classification results of the multi-frame image sequence. The system analyzes the feature vector matrix through a classification algorithm. Common classification algorithms include support vector machines (SVM) and neural networks, which can classify the data in the feature matrix and determine whether the driver has taken his hands off the car. The classification results usually include two categories: hands off or hands not off.

[0099] For example: Assume that the system uses a neural network to classify the feature matrix. The trained model analyzes the state of the hand in multiple frames and determines that the driver's hand has been away from the steering wheel for multiple frames. At this time, the classification algorithm marks the result as hands-off. If the hand is in contact with the steering wheel for most of the time, the classification result is not-off.

[0100] In order to improve the accuracy of hands-off detection, the system not only relies on the detection results of a single frame of image, but also analyzes a multi-frame image sequence over a period of time. The so-called multi-frame image sequence refers to a number of frames of images of the driver's hands and steering wheel collected continuously by the system over a period of time.

[0101] In each single-frame image, the system extracts the key point information of the hand and the mask of the steering wheel, and combines it with the single-frame detection algorithm mentioned above to determine whether the hand is off the steering wheel. However, since the driver's hands may move briefly during normal driving, such as to adjust the seat, turn on the air conditioner, etc., relying on single-frame detection is prone to false alarms. Therefore, the system also makes a comprehensive judgment based on image frames over a period of time, which is the so-called multi-frame image sequence analysis.

[0102] Image classification technology uses a trained deep learning model to classify the feature vectors of each frame of the image. Feature vectors are digital representations extracted from images to describe the image, usually including information such as hand key points, hand contact area, and finger positions. The model generates a classification result based on these feature vectors, indicating whether the frame of the image may have been taken out of the hand.

[0103] The system will classify the multi-frame image sequence and combine the classification results of each frame to generate a classification result of the multi-frame image sequence. The classification result can be understood as the overall judgment of multiple image frame detections, reflecting whether the driver has continued to take his hands off the car during the time period.

[0104] The classification threshold is a key parameter used to judge a multi-frame image sequence. The classification threshold can be a preset value used to limit which frame image classification results in the multi-frame image sequence indicate that the driver may take the driver's hands off the car. For example, the system may set a classification threshold. If more than 50% of the images in the classification results of the multi-frame image sequence show a hands-off state, then the multi-frame image sequence will be marked as dangerous.

[0105] By setting the classification threshold, the system can ensure that the alarm will not be easily triggered due to the misjudgment of a single frame of image. Especially when the driver's hands leave the steering wheel for a short time, such as adjusting the air conditioner or controlling the wipers, the system can judge whether the driver has really lost control of the steering wheel through the continuity of multiple frames of image.

[0106] For example, suppose the classification threshold of the system is set to 60%. If 10 frames of images are collected within a certain period of time, and the classification results of 6 frames show that the driver has taken his hands off the car, then the classification results of the multi-frame image sequence will be considered to have exceeded the threshold, indicating that there is a high probability of hands-off behavior.

[0107] Once the classification results of a multi-frame image sequence are greater than the preset classification threshold, the system will further check the single-frame image detection results in these image sequences to ensure that these detection results are consistent with the actual situation.

[0108] During this process, the system needs to determine whether one or more frames of image detection results meet the steering wheel hands-off condition. For example, some frames may show that the driver's hands are completely off the steering wheel, or even multiple finger key points are outside the mask image. This situation will trigger a single-frame image hands-off alarm.

[0109] If in a multi-frame image sequence, the single-frame image detection result of at least one frame indicates that the driver has indeed taken his hands off the camera, then combined with the overall judgment of the classification results of the multi-frame image sequence, the system will determine that the driver has taken his hands off the camera.

[0110] By combining the classification results of multi-frame image sequences with the detection results of single-frame images, the system can more accurately determine whether the driver has taken his hands off the steering wheel.

[0111] For example, suppose the system captures 30 frames of images within 5 seconds, and the classification results of 20 frames show that the driver's hands are off the steering wheel. At the same time, the single-frame detection results show that the finger key points of several frames have completely exceeded the steering wheel area. Combining these two pieces of evidence, the system will determine that the driver has taken his hands off the steering wheel and issue a warning signal in time.

[0112] The embodiment of the present invention significantly improves the accuracy and robustness of hands-off detection by combining the multi-frame image classification results with the single-frame detection results. It avoids the false alarm problem that may be caused by single-image detection, and also reduces the situation where unnecessary warnings are triggered due to the driver's brief hand movement. The driver's hands cannot remain completely still during daily driving, and this multi-frame detection method can tolerate the driver's slight hand movement and will not be misjudged as hands-off due to some irrelevant movements.

[0113] In an exemplary embodiment of the present invention, after classifying the feature vector matrix to obtain the classification result of the multi-frame image sequence, an implementation method is to determine that there is no hands-off situation when the classification result of the multi-frame image sequence is less than or equal to the preset classification threshold. If the classification result of the multi-frame image sequence is less than or equal to the preset classification threshold, the system determines that the driver did not take his hands off the steering wheel. This means that during this time period, the system did not find that the driver's hands left the steering wheel. This judgment method effectively reduces the possibility of false alarms. For example, when the driver may briefly move his hands to other locations for operation, the system will not trigger an alarm, thereby reducing the driver's tension and discomfort.

[0114] By setting a reasonable classification threshold, the system of the embodiment of the present invention can avoid misjudging the driver's state due to accidental factors. When the driver's hands only leave the steering wheel for a short time, the system can still confirm that they have not really taken their hands off the steering wheel through the analysis of multiple frames of images. This feature is particularly important in actual driving because it can reduce the driver's distrust of the system and allow them to stay relaxed during driving.

[0115] In an exemplary embodiment of the present invention, after classifying the feature vector matrix to obtain the classification result of the multi-frame image sequence, an implementation method is to determine that the steering wheel is not released when the classification result of the multi-frame image sequence is greater than a preset classification threshold and there is no single-frame image detection result in the multi-frame image sequence that meets the preset steering wheel release condition.

[0116] In the specific implementation process, it is assumed that the multi-frame image sequence set by the system is 20 consecutive frames, and the classification threshold is set to 0.7 (indicating a 70% probability of believing that the hands have been dropped). In a certain detection, the classification result obtained by the system after analyzing these 20 frames of images is 0.8, which exceeds the preset 0.7 threshold. According to traditional methods, this may be directly judged as a hands-off behavior. However, this implementation method also requires further inspection of each of the 20 frames.

[0117] Further analysis revealed that in these 20 frames, none of the single-frame image detection results met the steering wheel off-hand condition. This may be because the driver had slight hand movements (such as adjusting the grip or releasing the grip briefly and then immediately holding it back), which led to a high multi-frame classification result, but in fact the driver always maintained control of the vehicle.

[0118] In this case, although the classification results of the multi-frame image sequence exceeded the threshold, the system ultimately determined that there was no hands-off state because no single frame image showed a clear hands-off state. This method effectively avoids false positives for brief and safe hand movements.

[0119] The embodiment of the present invention greatly reduces the possibility of false alarms by considering both multi-frame sequences and single-frame results. This dual verification mechanism can better distinguish between true hands-off behavior and normal driving actions. It has a stronger tolerance for short-term hand movements (such as quickly adjusting the steering wheel position) and will not trigger alarms due to instantaneous hands-off, thereby improving the stability and reliability of the system. By reducing false alarms, frequent and unnecessary warnings can be avoided, and the driver's trust in the system and comfort in use can be improved.

[0120] In an exemplary embodiment of the present invention, an implementation method of locating a steering wheel image and a hand image from a single frame image of video frame data is to input the single frame image into YOLOX (YOLOX is the abbreviation of "You Only Look Once X". It is an improved version of the YOLO (You Only Look Once) series target detection algorithm) and output the steering wheel image and the hand image.

[0121] In actual application, the system first extracts a single frame image from the video stream. This single frame image contains the entire cockpit scene, including the steering wheel area and the driver's hands. Next, this single frame image is input into the pre-trained YOLOX model. The YOLOX model will quickly locate and identify the position of the steering wheel and hands in the image, and output images of these two key areas.

[0122] For example, there is a cockpit video stream with a 1080p resolution (1920x1080 pixels). The system extracts 30 frames of images from this video stream per second for analysis. For one frame of the image, the YOLOX model completes the processing in about 20 milliseconds and outputs a result similar to the following:

[0123] 1. Steering wheel image: position (x=800, y=500), size 200x200 pixels

[0124] 2. Left hand image: position (x=750, y=600), size 100x150 pixels

[0125] 3. Right hand image: position (x=950, y=600), size 100x150 pixels

[0126] These outputs include not only the location of the object, but also a confidence score, which indicates how sure the model is about the detection.

[0127] In an embodiment of the present invention, YOLOX is a single-stage target detection algorithm that can simultaneously complete target location and classification in one forward propagation. This means that it can complete the analysis of the entire image in a very short time (usually milliseconds), which is very suitable for the needs of real-time monitoring systems. Compared with earlier versions of YOLO, YOLOX has significant improvements in small target detection and bounding box positioning. This is particularly important for accurately capturing the position of the hand, because at certain angles, the hand may only occupy a small part of the image. YOLOX has powerful feature extraction capabilities and can maintain good detection performance in complex situations such as different lighting conditions and changes in hand postures. This ensures the stability of the system in various driving environments.

[0128] In an exemplary embodiment of the present invention, an implementation method for obtaining the mask image of the steering wheel image and the key point position information of the hand image is to perform edge detection processing and ellipse matching processing on the steering wheel image to obtain the mask image; the hand image is rotated at the corresponding corner points and input into the BlazeHand model (a lightweight hand posture estimation model) with a forward hand posture, and the key point position information is output.

[0129] In actual application, the steering wheel mask generation process is:

[0130] 1. Edge detection processing: The system first performs edge detection on the steering wheel image. Common edge detection algorithms include Canny edge detector or Sobel operator. This step can highlight the outline of the steering wheel.

[0131] 2. Ellipse matching processing: Since most steering wheels are elliptical, the system uses an ellipse matching algorithm (such as Hough transform) to fit the detected edges to obtain the best matching ellipse shape.

[0132] 3. Generate mask image: Based on the matched ellipse, the system creates a binary image where the steering wheel area is 1 (white) and other areas are 0 (black).

[0133] The hand key point detection process is:

[0134] 1. Hand image rotation: The system first rotates the hand image to make it appear in a positive posture. This step helps improve the accuracy of subsequent detection.

[0135] 2. Input into BlazeHand model: The rotated hand image is input into the pre-trained BlazeHand model.

[0136] 3. Output key point position information: The BlazeHand model outputs the key point positions of the hand, usually including 21 key points, representing the main joint positions of the palm and fingers.

[0137] For example, suppose there is a 1280x720 pixel image of the steering wheel area. First, perform Canny edge detection on this image to obtain a binary image with prominent contours. Then, use Hough transform to perform ellipse matching to find the best matching ellipse parameters (center point coordinates, major axis, minor axis and rotation angle). Based on these parameters, a 1280x720 binary mask image is generated, in which the steering wheel area is white (1) and other areas are black (0).

[0138] For the hand image, assume that the hand is at a 45-degree angle in the original image. The system first rotates the image by -45 degrees to make the hand present a positive posture. Then, this rotated image of 224x224 pixels (the standard input size of BlazeHand) is input to the BlazeHand model. The model outputs the coordinates of 21 key points.

[0139] Through edge detection and ellipse matching, the embodiment of the present invention enables the system to accurately locate the steering wheel and maintain good results even under complex backgrounds or lighting conditions. The ellipse matching algorithm can adapt to steering wheels of different shapes and sizes, making the system suitable for various types of vehicles. The BlazeHand model is designed for mobile and embedded devices and can complete hand posture estimation within milliseconds to meet the needs of real-time monitoring. Compared with simple hand detection, key point information provides a detailed description of the hand posture, providing rich data for subsequent grip analysis. By performing rotation preprocessing on the hand image, the system can handle various hand postures and improve the robustness of the detection.

[0140] In an exemplary embodiment of the present invention, an implementation method of classifying a feature vector matrix to obtain a classification result of a multi-frame image sequence is to classify the feature vector matrix using a temporal classification model based on a gated recurrent unit (Gated Recurrent Unit, GRU for short) to obtain a classification result of a multi-frame image sequence. Among them, GRU is a variant of a recurrent neural network (RNN for short), which is specially designed to process sequence data. GRU can effectively capture long-term dependencies through its special gating mechanism, while avoiding the gradient vanishing problem in traditional RNN.

[0141] In this implementation, the system first constructs a feature vector matrix that contains the feature information of each frame in the multi-frame image sequence. Then, the feature vector matrix is ​​input into the GRU-based temporal classification model for processing. The model finally outputs the classification result of the multi-frame image sequence, that is, whether there is a hands-off behavior in the time sequence.

[0142] For example, suppose there is a 30-frame (about 1 second) image sequence. For each frame, the following features have been extracted:

[0143] 1. The position of the key points of the hand (21 points, each with x and y coordinates)

[0144] 2. Relevant features of the steering wheel mask (such as steering wheel center coordinates, semi-major axis, semi-minor axis, etc.)

[0145] 3. Other potentially relevant features (such as the overlap ratio between the hand and the steering wheel)

[0146] Thus, each frame may have a feature vector of about 50 dimensions. For a sequence of 30 frames, a 30x50 feature vector matrix is ​​obtained.

[0147] Next, this 30x50 matrix is ​​input into the GRU model. Assume that the GRU model has the following structure:

[0148] 1. Input layer: accepts a 30x50 feature matrix

[0149] 2. GRU layer 1: 64 hidden units

[0150] 3. GRU layer 2: 32 hidden units

[0151] 4. Fully connected layer: 16 neurons

[0152] 5. Output layer: 2 neurons (indicating the probability of releasing and not releasing the hands)

[0153] The model processes the 30-frame sequence and finally outputs a binary classification result, such as [0.05, 0.95], which means that there is a 95% probability that the hand has been dropped in this sequence.

[0154] In an embodiment of the present invention, GRU can effectively capture the temporal dependencies in sequence data. This is crucial for analyzing the driver's hand movements, because certain actions (such as temporarily letting go to adjust the grip) may appear to be out of hand in a single frame, but are actually safe in the entire sequence. Compared with traditional RNN, GRU is better at capturing long-term dependencies. This enables the model to take into account driving behavior patterns over a longer time frame and improve classification accuracy. The gating mechanism of GRU enables it to selectively memorize important information and forget irrelevant information. This improves the model's resistance to noise and short-term interference, making the classification results more stable.

[0155] Based on the above description of a method for detecting a hands-off steering wheel, a multi-scenario hands-off steering wheel detection scheme (referred to as a detection scheme) with multi-model joint inter-frame verification is introduced below. The detection scheme can be applied to a driver detection system. The detection scheme can include the following two parts: 1. Multi-model joint detection and judgment; 2. Inter-frame verification.

[0156] In this detection solution, the camera is deployed on the A-pillar. Due to different car models, the angle and height of the A-pillar are different. The camera is selected based on the field of view (FOV) that can observe the complete steering wheel. Specifically, it can be an infrared camera (IR). The collected image is as follows Figure 2 shown.

[0157] The learning of visual tasks based on deep learning models is inseparable from image data and corresponding labels. The acquisition process adopted by this detection solution is as follows:

[0158] 1) Collection personnel: The number of drivers is required to be 20 or more, and the height distribution is 160-190cm, which is roughly normally distributed and meets the diversity requirements.

[0159] 2) Collection equipment: The wide-angle IR camera is located in the A-pillar, and its field of view can fully present the driver's facial features and the complete steering wheel.

[0160] 3) Data acquisition format: continuous frame short video, which is convenient for frame extraction and subsequent multi-frame verification.

[0161] 4) Collection process:

[0162] a. After getting on the vehicle, the personnel adjust the seat to a suitable driving position and hold the steering wheel to start collecting data.

[0163] b. Turn the wheel to the left for one circle while driving normally, and repeat 5 times;

[0164] c. Turn the wheel right once during normal driving, repeat 5 times;

[0165] d. Use one hand to roll the wheel on both sides of the left hand for one circle each, repeat 5 times;

[0166] e. Use one hand to turn the wheel on both sides of the right hand for one circle each, repeat 5 times;

[0167] f. Hold the steering wheel with one hand and make a phone call with the other hand. Repeat this 10 times, switching hands.

[0168] g. Move both hands away from the steering wheel and make various gestures with your hands at different heights. Repeat 10 times.

[0169] h. Move both hands away from the steering wheel, use one hand to make a phone call while using the other hand to perform the specified action. Repeat 20 times, switching between left and right.

[0170] i. Move the steering wheel, readjust the seat to a suitable position, and repeat steps b to h once more.

[0171] Since the current data collection is performed according to the specified actions, video annotation can be completed directly through different actions. The hands and steering wheel need to rely on the annotation tool to complete the adjustment and calibration of the corresponding annotation boxes based on the pre-annotation.

[0172] The data is divided into image data and corresponding target detection labels, and continuous frame short video data and corresponding video labels. The former participates in the training of the target detection model, and the model after training has the ability to recognize the location of the opponent and the steering wheel image in the current picture; the latter data participates in the training of the behavior recognition model, and the model after training has the ability to judge the behavior of the current continuous frame, that is, whether the driver is driving with the steering wheel off.

[0173] Reference Figure 3 , shows a schematic diagram of the algorithm flow of a multi-scenario steering wheel hands-off detection solution with multi-model joint inter-frame verification in an embodiment of the present invention.

[0174] The overall process of single frame detection:

[0175] 1) Obtain a real-time single-frame image on the camera side, and pre-process the real-time image of the IR camera to a format that meets the model input requirements.

[0176] 2) Perform 2D target detection through the trained 2D target detection model (the model can effectively identify the features of the steering wheel and the driver's hands under the current camera angle), accurately locate the positions of the steering wheel and the driver's hands in the picture, and intercept these positions to obtain the steering wheel position image and the driver's hands position image.

[0177] 3) The steering wheel position image is subjected to edge detection in the classic image algorithm and an accurate mask image, i.e., a contour image, of the steering wheel captured in the current image is obtained through ellipse matching.

[0178] 4) The hand position image is rotated at the corresponding corner points and input into the key point detection model with the forward hand posture to detect the hand key points, and obtain the position information of 21 key points under the current hand posture, such as Figure 4a As shown, normalization operation is performed at the same time to prepare for the next step of information fusion, such as Figure 4b shown.

[0179] 5) Perform hands-off judgment on a single-frame image based on the current steering wheel position of the hand key points and the current steering wheel position in the picture: take the percentage of finger key points in the steering wheel mask as the judgment threshold. If more than 7 of the 12 key points of the finger leave the mask area of ​​the steering wheel, it is directly judged as hands-off and a hands-off reminder is given; otherwise, enter the joint judgment process to reduce errors through joint judgment calculation.

[0180] The overall process of multi-frame detection:

[0181] 1) According to the actual vehicle algorithm operation efficiency, 10 frames are temporarily used as a queue, with an interval of two frames each time, that is, 20 frames are accumulated each time for sliding window reading. Each model input is a 10-frame image sequence, and the Backbone model is used to extract the continuous multi-frame feature vectors to be detected.

[0182] 2) The feature vectors of 10 frames are combined, and the normalized data of the hand key points in each frame are fused into information vectors to form a 10-dimensional feature vector matrix.

[0183] 3) Input the feature vector matrix into a continuous frame detection model (such as the Long Short-Term Memory (LSTM) model) for multi-frame judgment by the LSTM model, specifically classify the 10-dimensional feature vector matrix to determine whether the hand is released. Note that the current hand-release result is the result of feedback after integrating the previous 10 frames of images, and the result is locked to the last frame.

[0184] Joint judgment process:

[0185] 1) If the single-frame image detection result of the current frame directly determines that the hands are off the steering wheel, the current frame is deemed to be hands-off, and an alarm information is reported.

[0186] 2) If the current single-frame image detection result is that the hand is not released, the classification results of the multi-frame image sequence are continuously counted and saved in the queue.

[0187] 3) The classification results of the multi-frame image sequence are jointly judged with the detection results of each single-frame image. If more than 50% of the current multi-frame image sequence classification results are hand-offs, and there are hand-off results in the single-frame image detection results, it is determined to be hand-off, otherwise it is determined to be not hand-off.

[0188] Three deep learning models are used in the overall process:

[0189] 1) Object detection model for hands and steering wheels:

[0190] YOLOX was selected as the main target detection model. YOLOX performs well in target detection, especially in real-time application scenarios. In order to improve the detection efficiency and frame rate of the model, YOLOX was optimized. First, the input data was converted from multi-channel to single-channel to reduce the complexity of data processing. In addition, in order to increase the detection speed, the Backbone of YOLOX was replaced with MobileNetV2. The lightweight structure and efficient convolution operation of MobileNetV2 enable it to maintain a high detection speed and accuracy even with limited computing resources. The model will be trained using the currently collected specific scene data to better adapt to the target characteristics in actual applications.

[0191] 2) Hand key point detection model:

[0192] The BlazeHand model is used to detect the key points of the hand. BlazeHand is a model optimized for hand detection that can accurately identify and locate multiple key points of the hand. The model input is a single-channel hand image, which is preprocessed to adapt to the input requirements of the BlazeHand model.

[0193] The BlazeHand model provides detailed information on hand posture and movement, which provides more accurate finger position and status for subsequent hands-off analysis.

[0194] The BlazeHand model uses a keypoint regression method to predict the positions of the hand keypoints. For each hand keypoint k (such as a finger joint), the model outputs a predicted position It can be expressed as:

[0195]

[0196] Where x is the input image, f(x) is the feature map extracted by the convolutional neural network (CNN), and MLP stands for multi-layer perceptron, which is used to regress the specific coordinates of each key point.

[0197] The loss function is the mean squared error (MSE):

[0198]

[0199] Where N is the number of key points, is the predicted position of the ith keypoint, p i is the corresponding real position.

[0200] After that, the key point positions are normalized. The normalization formula is:

[0201]

[0202] in, is the normalized key point position, μ and σ are the mean and standard deviation of the position data respectively. The prediction result is then restored to the original scale through the denormalization formula:

[0203]

[0204] 3) Continuous frame hands-free scene classification model:

[0205] Use the GRU-based temporal classification model to process continuous frame data. The overall structure of the GRU model is as follows: Figure 5 shown.

[0206] The GRU model is good at processing time series data and can analyze continuous frame data by capturing dynamic changes in time.

[0207] The core lies in the control of its door, update door

[0208] z t =σ(W z ·[h t"1 ,x t ]+b z )

[0209] Where σ is the sigmoid activation function, W z is the weight matrix of the update gate, b z is the bias term, h t-1 is the hidden state at the previous moment, x t is the current input. Update gate z t Controls the hidden state h of the previous moment t-1 The degree of preservation at the current time t. It determines the hidden state h at the current time t and the hidden state h at the previous moment t-1 The weighted ratio controls the transmission and forgetting of information. The weight matrix W z and the bias term b z The σ function generates a value between 0 and 1, representing the degree of retention.

[0210] Reset Gate

[0211] r t =σ(W r ·[h t"1 ,x t ]+b r )

[0212] Among them, W r is the weight matrix of the reset gate, b r is the bias term. Reset gate r t Determine the previous moment h t-1 The influence of the hidden state in calculating the candidate hidden state. It controls the current input x t and the hidden state h at the previous moment t-1 The combination method affects how to "reset" previous memory information.

[0213] The final hidden state formula is as follows:

[0214]

[0215] The final hidden state h t is the hidden state at the current moment, which is the previous hidden state h t-1and a weighted combination of candidate hidden states. Update gate z t The contribution ratio of the previous hidden state and the candidate hidden state to the current hidden state is determined to balance long-term dependencies and short-term changes.

[0216] In order to enrich the information of a single frame, the input side normalizes and merges the detection feature vector of the image data with the key point information of the hand. This merger not only enhances the expressiveness of the single frame data, but also helps the GRU model better understand the changes of the hand and steering wheel in the time dimension. Ultimately, the GRU model will classify the hands-off scenes based on the time series data, effectively improving the accuracy of scene classification and real-time response capabilities.

[0217] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0218] Reference Figure 6 , shows a structural block diagram of a steering wheel hands-off detection system according to an embodiment of the present invention. The steering wheel hands-off detection system may specifically include the following modules.

[0219] A video frame data acquisition module 61 is used to acquire video frame data of a steering wheel area;

[0220] An image positioning module 62, configured to locate a steering wheel image and a hand image from a single frame image of the video frame data;

[0221] A mask image and key point position acquisition module 63, used to acquire the mask image of the steering wheel image and the key point position information of the hand image;

[0222] A single-frame image detection result determination module 64 is used to detect the single-frame image according to the mask image, the key point position information and a preset determination threshold to obtain a single-frame image detection result;

[0223] A hands-off situation determination module 65, configured to determine whether there is a hands-off situation of the steering wheel based on the single-frame image detection result;

[0224] A feature vector extraction module 66 is used to extract feature vectors of a multi-frame image sequence of the video frame data when it is determined according to the single-frame image detection result that there is no hands-off steering wheel;

[0225] The hands-off situation detection module 67 is used to comprehensively detect whether the steering wheel is off-hand according to the feature vector of the multi-frame image sequence and the single-frame image detection result.

[0226] In an exemplary embodiment of the present invention, the single-frame image detection result determination module 64 includes:

[0227] A proportion information calculation module, used to calculate the proportion information of the finger key points in the single frame image located in the mask image according to the mask image and the key point position information;

[0228] The ratio threshold comparison module is used to compare the ratio information with the determination threshold to obtain the single-frame image detection result.

[0229] In an exemplary embodiment of the present invention, the hands-off situation determination module 65 is used to determine that the steering wheel is off-hand when the judgment threshold is a maximum ratio of finger key points outside the mask image, based on the single-frame image detection result indicating that the proportion information is greater than the judgment threshold; and to determine that the steering wheel is not off-hand when the single-frame image detection result indicates that the proportion information is less than or equal to the judgment threshold.

[0230] In an exemplary embodiment of the present invention, the hands-off situation detection module 67 includes:

[0231] A feature vector merging module, used for merging the feature vector and the normalized data of the key point position information into a feature vector matrix;

[0232] A vector matrix classification module is used to classify the feature vector matrix to obtain a classification result of a multi-frame image sequence;

[0233] a single-frame result judgment module, configured to determine whether there is a single-frame image detection result satisfying a preset steering wheel hands-off condition in the multi-frame image sequence when the classification result of the multi-frame image sequence is greater than a preset classification threshold;

[0234] The steering wheel hands-off determination module is used to determine whether the steering wheel hands-off situation exists when there is a single-frame image detection result that meets the preset steering wheel hands-off condition in the multi-frame image sequence.

[0235] In an exemplary embodiment of the present invention, the system further comprises:

[0236] The steering wheel is not out of hands determination module is used to determine that the steering wheel is not out of hands after the vector matrix classification module classifies the feature vector matrix to obtain the multi-frame image sequence classification result, and if the classification result of the multi-frame image sequence is less than or equal to the preset classification threshold.

[0237] In an exemplary embodiment of the present invention, the steering wheel is not out of hands determination module is also used to determine that the steering wheel is not out of hands after the vector matrix classification module classifies the feature vector matrix to obtain a multi-frame image sequence classification result, if the classification result of the multi-frame image sequence is greater than a preset classification threshold, and there is no single-frame image detection result that satisfies the preset steering wheel out of hands condition in the multi-frame image sequence.

[0238] In an exemplary embodiment of the present invention, the image positioning module 62 is used to input the single frame image into YOLOX, and output the steering wheel image and the hand image.

[0239] In an exemplary embodiment of the present invention, the mask image and key point position acquisition module 63 includes:

[0240] A mask image acquisition module, used for performing edge detection processing and ellipse matching processing on the steering wheel image to obtain the mask image;

[0241] The key point acquisition module is used to rotate the hand image through corresponding corner points, input the hand image into the BlazeHand model in a positive hand posture, and output the key point position information.

[0242] In an exemplary embodiment of the present invention, the vector matrix classification module is used to classify the feature vector matrix using a GRU-based temporal classification model to obtain a classification result of the multi-frame image sequence.

[0243] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0244] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0245] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0246] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0247] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0248] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0249] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0250] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0251] The above is a detailed introduction to a method for detecting a hands-off steering wheel and a system for detecting a hands-off steering wheel provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting a hands-off steering wheel, characterized in that: The method comprises: Get the video frame data of the steering wheel area; Locating a steering wheel image and a hand image from a single frame image of the video frame data; Obtaining a mask image of the steering wheel image and key point position information of the hand image; Detecting the single-frame image according to the mask image, the key point position information and a preset determination threshold to obtain a single-frame image detection result; Determining whether there is a hands-off situation of the steering wheel based on the single-frame image detection result; When it is determined according to the single-frame image detection result that there is no hands-off steering wheel, extracting a feature vector of a multi-frame image sequence of the video frame data; Whether the hands on the steering wheel are off is comprehensively detected based on the feature vectors of the multi-frame image sequence and the single-frame image detection result.

2. The method according to claim 1, characterized in that The detecting the single-frame image according to the mask image, the key point position information and a preset determination threshold to obtain a single-frame image detection result includes: Calculating the proportion of finger key points in the single-frame image located in the mask image according to the mask image and the key point position information; The proportion information is compared with the determination threshold to obtain the single-frame image detection result.

3. The method according to claim 2, characterized in that The determining whether there is a hands-off situation of the steering wheel based on the single-frame image detection result includes: When the judgment threshold is the maximum ratio of finger key points outside the mask image, it is determined that the steering wheel is out of hands based on the single-frame image detection result indicating that the proportion information is greater than the judgment threshold; and it is determined that the steering wheel is not out of hands based on the single-frame image detection result indicating that the proportion information is less than or equal to the judgment threshold.

4. The method according to claim 1, characterized in that: The step of comprehensively detecting whether there is a hands-off situation of the steering wheel according to the feature vector of the multi-frame image sequence and the single-frame image detection result comprises: Combining the feature vector and the normalized data of the key point position information into a feature vector matrix; Classifying the feature vector matrix to obtain a classification result of the multi-frame image sequence; When the classification result of the multi-frame image sequence is greater than a preset classification threshold, determining whether there is a single-frame image detection result that satisfies a preset steering wheel hands-off condition in the multi-frame image sequence; When there is a single-frame image detection result satisfying the preset hands-off condition for the steering wheel in the multi-frame image sequence, it is determined that the steering wheel is in a hands-off situation.

5. The method according to claim 4, characterized in that After classifying the feature vector matrix to obtain the classification result of the multi-frame image sequence, the method further includes: When the classification result of the multi-frame image sequence is less than or equal to the preset classification threshold, it is determined that there is no hands-off steering situation.

6. The method according to claim 4, characterized in that After classifying the feature vector matrix to obtain the classification result of the multi-frame image sequence, the method further includes: When the classification result of the multi-frame image sequence is greater than the preset classification threshold and there is no single-frame image detection result in the multi-frame image sequence that meets the preset steering wheel hands-off condition, it is determined that there is no steering wheel hands-off situation.

7. The method according to claim 1, characterized in that The obtaining of the mask image of the steering wheel image and the key point position information of the hand image includes: Performing edge detection and ellipse matching processing on the steering wheel image to obtain the mask image; The hand image is rotated through corresponding corner points and input into the BlazeHand model in a positive hand posture, and the key point position information is output.

8. A steering wheel hands-off detection system, characterized in that: The system comprises: A video frame data acquisition module, used to acquire video frame data of a steering wheel area; An image positioning module, used to locate a steering wheel image and a hand image from a single frame image of the video frame data; A mask image and key point position acquisition module, used to acquire the mask image of the steering wheel image and the key point position information of the hand image; A single-frame image detection result determination module, used to detect the single-frame image according to the mask image, the key point position information and a preset determination threshold to obtain a single-frame image detection result; A hands-off situation determination module, used to determine whether there is a hands-off situation of the steering wheel based on the single-frame image detection result; A feature vector extraction module, for extracting feature vectors of a multi-frame image sequence of the video frame data when it is determined according to the single-frame image detection result that the hands on the steering wheel do not exist; The hands-off situation detection module is used to comprehensively detect whether the steering wheel is off-hand according to the feature vector of the multi-frame image sequence and the single-frame image detection result.

9. An electronic device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the electronic device to perform the method for detecting a hands-off steering wheel as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer program stored therein enables the processor to execute the method for detecting hands-off steering wheel as described in any one of claims 1 to 7.

Images