A driver hand action perception method, system, device and storage medium

Abstract

The application provides a driver hand action perception method, comprising the following steps: extracting hand image feature information; extracting vehicle feature information; fusing the hand image feature information and the vehicle feature information; extracting gesture action label text features; training a clip model to classify the fused features.

Description

Technical Field

[0001] This invention belongs to the field of visual perception technology, specifically relating to a method, system, device, and storage medium for sensing driver hand movements. Background Technology

[0002] In driver training, the ability to perceive hand gestures is crucial for recognizing driver intentions and ensuring proper operation. Currently, some gesture recognition devices exist on the market, but these devices often suffer from technical limitations, such as low sensitivity and high false alarm rates, which restrict their application scope and practical effectiveness. Summary of the Invention

[0003] This invention can effectively improve the sensitivity and accuracy of gesture recognition. To solve the above problems, this invention adopts the following technical solution:

[0004] This invention provides a method for sensing driver hand movements, including:

[0005] Extract feature information from hand images;

[0006] Extract vehicle feature information;

[0007] The hand image feature information and the vehicle feature information are fused together;

[0008] Extract gesture action label text features;

[0009] Train the clip model to classify the fused features.

[0010] Furthermore, the method for extracting hand image feature information includes:

[0011] Get the front image of the hand; get the back image of the hand.

[0012] Extract the hand region from the front image of the hand and the back image of the hand;

[0013] The extracted hand image is filtered to remove noise interference;

[0014] The visual transform network is used to extract features from the filtered image.

[0015] Furthermore, the vehicle characteristic information includes the vehicle engine speed and vehicle gear information.

[0016] Furthermore, the method for fusing feature information involves aligning the features according to their temporal order and then fusing them, as shown in the formula:

[0017]

[0018] Furthermore, the gesture action label text features include at least one of the following: turning the steering wheel to the left, turning the steering wheel to the right, shifting gears, adjusting the seat, adjusting the left rearview mirror, adjusting the right rearview mirror, turning the steering wheel with one hand, and turning the steering wheel with both hands crossed.

[0019] The present invention also provides a driver hand motion sensing system, comprising:

[0020] The image acquisition module is used to acquire images of the hand;

[0021] The data acquisition module is used to acquire vehicle data;

[0022] Feature fusion module: used to fuse the hand image and the vehicle data;

[0023] The gesture extraction module is used to obtain the text features of gesture tags;

[0024] The classification module is used to obtain classifications based on fused features by training the model.

[0025] Furthermore, the image acquisition module includes a first camera and a second camera, used to acquire images of the front and back of the hand.

[0026] Furthermore, the data acquisition module includes a vehicle OBD unit and an environmental perception radar.

[0027] The present invention also provides a driver's hand movement sensing device, comprising:

[0028] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions implementing the driver hand movement sensing method as claimed.

[0029] The present invention also provides a computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to implement the driver hand movement perception method described in the claims.

[0030] The positive effects of this invention are:

[0031] 1. High sensitivity. This algorithm can accurately recognize hand movements, even minute ones.

[0032] 2. Low false positive rate. This algorithm can effectively remove noise interference and avoid false positives.

[0033] 3. Low power consumption. This unit has low power consumption, making it suitable for embedded systems and low-power devices.

[0034] 4. Low cost. The unit has a low manufacturing cost, making it suitable for mass production and application. Attached Figure Description

[0035] Figure 1 A schematic diagram of model training according to an embodiment of the present invention;

[0036] Figure 2 A schematic diagram illustrating the application of a model according to an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0038] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1 As shown: In one embodiment of the present invention, two cameras are set at the front and back. The two cameras respectively acquire images of the front and back of the hand, and identify the hand region based on preset pixel value range, color, shape and other features, and extract the hand region image.

[0039] The extracted hand image is filtered to remove noise interference.

[0040] The visual transform network is used to extract features from the denoised image.

[0041] The system collects vehicle OBD data and vehicle sensor data, and extracts features from the collected data. For example, it collects vehicle speed data and vehicle gear information.

[0042] The hand image features and vehicle features are aligned in chronological order and then fused.

[0043] The formula is

[0044] Using NLP technology to extract text features of gesture action labels, the gesture labels to be identified are: turning the steering wheel to the left, turning the steering wheel to the right, shifting gears, adjusting the seat, adjusting the left rearview mirror, adjusting the right rearview mirror, turning the steering wheel with one hand, and turning the steering wheel with both hands crossed.

[0045] The fused features are used as input to train the clip model, which classifies the multi-sensor fused features.

[0046] Feature extraction and feature fusion are performed based on hand images obtained from front and rear cameras and vehicle sensor data. A trained model is used as input for fused features and output for corresponding gesture labels.

[0047] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for sensing driver hand movements, characterized in that, include: Extract feature information from hand images; Extract vehicle feature information; The hand image feature information and the vehicle feature information are fused together; Extract gesture action label text features; Train the clip model to classify the fused features; The extraction of hand image feature information includes: Get the front image of the hand; get the back image of the hand. Extract the hand region from the front image of the hand and the back image of the hand; The extracted hand image is filtered to remove noise interference; The visual transform network is used to extract features from the filtered image; The vehicle characteristic information includes the vehicle engine speed and vehicle gear information; The method for fusing feature information involves aligning the features according to their temporal order and then fusing them. The formula is as follows: ; The gesture action label text features include at least one of the following: turning the steering wheel to the left, turning the steering wheel to the right, shifting gears, adjusting the seat, adjusting the left rearview mirror, adjusting the right rearview mirror, turning the steering wheel with one hand, and turning the steering wheel with both hands crossed.

2. A driver's hand movement sensing system, characterized in that, This system can implement the driver's hand movement sensing method as described in claim 1, and the system includes: The image acquisition module is used to acquire images of the hand; The data acquisition module is used to acquire vehicle data; Feature fusion module: used to fuse the hand image and the vehicle data; The gesture extraction module is used to obtain the text features of gesture tags; The classification module is used to obtain classifications based on fused features by training the model.

3. A driver hand movement sensing system according to claim 2, characterized in that... : The image acquisition module includes a first camera and a second camera, used to acquire images of the front and back of the hand.

4. A driver hand movement sensing system according to claim 3, characterized in that: The data acquisition module includes a vehicle OBD unit and an environmental perception radar.

5. A driver's hand movement sensing device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions implementing the driver hand movement perception method of claim 1.

6. A computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to implement the driver hand movement perception method as described in claim 1.

Images