A method for reducing gesture misrecognition based on region optimization strategy

Abstract

The present invention discloses a method for reducing gesture misrecognition based on a regional optimization strategy, wherein a gesture response area is set in a real vehicle, corresponding to the effective camera area range corresponding to the vehicle-mounted camera; the input event is output to the corresponding application interface for functional response through gesture recognition in the response area, thereby reducing the false trigger rate and improving the user's cockpit gesture control function experience; the effective gesture control area in the vehicle is specified to reduce false recognition, greatly reducing gesture misrecognition and improving the user experience. Further, based on the user's hand motion information, dynamic gesture actions are classified. At the same time, combined with the user usage scenarios of dynamic gestures, the dynamic gesture response area is increased to reduce the false recognition of similar gestures such as steering, touching the belly, patting the arm, and unconscious hand waving, thereby reducing the false trigger rate and improving the user's cockpit gesture control function experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of cockpit interaction, and in particular relates to a method for reducing gesture misrecognition based on a region optimization strategy. Background Art

[0002] The wave of intelligence has swept the world, and the automotive field is also undergoing an iteration from the era of large screens to the era of smart cockpits. Smart cockpits do not change cars, but the way users communicate with cars. Smart cockpits bring users better convenience and more humanized interaction. Traditional cockpit interaction methods are based on physical buttons, which gradually developed into touch screen control on the car side, and then developed into voice and gesture interaction control. In-car voice interaction has become relatively popular, and many new car models are equipped with it. But it still has its own limitations, such as recognition rate, recognition speed, and support for natural speech, which are still imperfect. Among them, gesture recognition is natural and in line with user control logic, which can bring users the advantage of non-sensory interaction experience, and can well make up for the inconvenience and embarrassment of voice control, making the interaction method more convenient and intelligent. Therefore, improving the gesture recognition rate and accurately judging the user's gesture intention are particularly important for improving the user's intelligent control experience.

[0003] Thanks to the rapid development of computer vision and machine learning, especially convolutional neural networks in deep learning, the research results in the field of gesture recognition are relatively mature. It is now common to use it in scenarios such as smart home and smart entertainment devices. In the field of smart cockpits, interactive technologies based on gesture recognition are relatively mature. For example, the "gesture recognition method and system" disclosed in CN104933408B is applicable to the field of human-computer interaction technology. The method includes: when the starting coordinates of a gesture are detected, the motion trajectory information starting from the starting coordinates of the gesture is recorded; fixed feature information is extracted from the motion trajectory information; the fixed feature information is recognized by a preset gesture recognition model, and the recognition result is output; it is determined whether there are samples with incorrect recognition in the recognition result; if so, specific feature information is extracted from the samples with incorrect recognition; the specific feature information is recognized by the preset gesture recognition model, and the recognition result is output. This invention can not only ensure the real-time performance of gesture recognition, but also greatly improve the accuracy of gesture recognition. For example, CN201810811365.0 discloses "A dynamic gesture recognition method and system based on gesture key points". This method solves the technical problems of poor scalability and low accuracy caused by the need for manually designed templates, as well as the technical problem of long training cycle caused by the need for a large number of training samples in existing dynamic gesture recognition methods. However, this method is difficult to cope with the misrecognition of gestures caused by many problems such as strong interference in the in-vehicle environment (such as light changes, human interference, occlusion, etc.), uncertainty in key areas, and irregularities in user operation gestures. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method for reducing gesture misrecognition based on a regional optimization strategy, so as to solve many problems in the prior art such as strong interference from the in-vehicle environment and uncertainty in key areas, resulting in a high gesture misrecognition rate, a poor user experience, and even hidden dangers in arrangements.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A method for reducing gesture misrecognition based on a regional optimization strategy, wherein a gesture response area is set in a real vehicle, corresponding to the effective camera area range of a vehicle-mounted camera; through gesture recognition in the response area, input events are output to a corresponding application interface for functional response, thereby reducing the false trigger rate and improving the user's cockpit gesture control function experience; the gesture response area is the vehicle's central control position and the middle position of the rear row; and dynamic gestures in the gesture response area are collected by setting the position of the vehicle-mounted camera.

[0007] The dynamic includes positive sample data made according to the gesture definition, and the proximity gesture definition considered according to the user scenario, and establishes a dynamic gesture recognition model.

[0008] Furthermore, the dynamic gestures are classified into three categories: left swipe, right swipe and ignore.

[0009] If all the key points of the hand in at least one frame of the first N frames of the gesture output by the dynamic gesture recognition model fall within the response area, the gesture is considered to be a valid gesture; otherwise, the gesture is considered to be an invalid gesture. The dynamic gesture recognition model input includes the positions and confidence information of 21 key points of the hand in n frames of the same Track ID.

[0010] Furthermore, the method of the present invention includes the following misidentification reduction process:

[0011] 1) Data collection: In a real car, dynamic gestures in the gesture response area are collected through the on-board camera;

[0012] 2) Hand detection: The collected dynamic gesture data is passed through the hand detection module to obtain the position of the gesture area in the image, the upper left corner (x1, y1), the lower right corner (x2, y2) and the hand frame confidence;

[0013] 3) Hand tracking: Since the hand detector can output the hand position, the in-car camera is relatively fixed, and the user's hand position is also relatively fixed during the gesture execution process, the hand area in the image is tracked and given a track ID by using the hand size change relationship, the image similarity relationship, and the relative position relationship of the hand in the image, to ensure that the gesture of the performer can be correctly recognized;

[0014] If the hand has been detected, the key point detection model is input based on the hand image detected in the hand detection stage to obtain the position and confidence of the hand key points;

[0015] The dynamic gesture recognition model is based on a convolutional neural network. The data processed by the hand tracking model and the key point detection model are input into the model for multiple iterative training and model parameter adjustment. The trained dynamic gesture classification model can effectively learn the features of different categories of images and perform recognition. Among them, the dynamic gesture recognition model mainly includes the positions and confidence information of 21 hand key points of the same Track ID n frames.

[0016] 4) Continuous frame recognition: If N (4) consecutive frames are all the same action and the confidence is greater than Score_limit (0.9), then the action is confirmed. Here, N and Score_limit are both thresholds, which can be parameterized and need to be adjusted to appropriate values ​​through experiments;

[0017] 5) Area optimization restriction strategy: If all the key points of the hand in at least one frame of the first N frames of the gesture action output by the dynamic gesture recognition model fall within the response area, the gesture is considered to be a valid gesture; otherwise, the gesture is considered to be an invalid gesture;

[0018] 6) Vehicle application event response: Output the event to the corresponding application interface application to perform functional response to the input event.

[0019] Furthermore, in order to reduce the impact of hand detection on classification accuracy, the gesture frame is expanded outward by 0.5 times. The expansion method is as follows:

[0020] x1'=max(0,x1-(x2-x1)*0.5), y1'=max(0,y1-(y2-y1)*0.5),

[0021] x2'=min(w,x2-(x2-x1)*0.5), y2'=min(h,y2+(y2-y1)*0.5).

[0022] Where w and h are the width and height of the image resolution, x1 and y1 are the coordinates of the upper left corner of the gesture area, and x2 and y2 are the coordinates of the lower right corner.

[0023] According to the location of the expanded area, the gesture images are cropped and saved separately to form a training set, a validation set, and a test set.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. Based on the regional optimization and restriction strategy, the present invention sets a gesture response area in the actual vehicle, which corresponds to the effective camera area range of the vehicle-mounted camera; through gesture recognition in the response area, the input event is output to the corresponding application interface for functional response, thereby reducing the false trigger rate and improving the user's cockpit gesture control function experience; the effective gesture control area in the vehicle is specified to reduce false recognition, which greatly reduces gesture false recognition and improves the user experience.

[0026] 2. The present invention classifies dynamic gestures based on user hand motion information. At the same time, combined with dynamic gesture user usage scenarios, the dynamic gesture response area is increased to reduce the misrecognition of similar gestures such as steering, touching the belly, patting the arm, and unconscious hand waving, and reduce the false trigger rate, thereby improving the user's cockpit gesture control function experience.

[0027] 3. Combined with the user's actual gesture control habits, the first N frames of the gesture action output by the dynamic gesture recognition model are taken. If at least one frame has all the key points of the hand falling within the response area, the gesture is judged to be a valid gesture; otherwise, the gesture is an invalid gesture. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the effective area of ​​the dynamic gesture based on the regional optimization and restriction strategy of the present invention (within the dotted box of the top view of the whole vehicle);

[0029] Figure 2 is an input graph of a dynamic gesture recognition model based on a regional optimization strategy of the present invention;

[0030] Figure 3 It is a flow chart of the present invention for reducing misrecognition based on the application of regional optimization strategy. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described below in conjunction with the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0032] See also Figure 1The present invention provides a gesture misrecognition reduction method based on a regional optimization and restriction strategy. In a real vehicle, the vehicle's central control position and the middle position of the rear row are set as effective areas for dynamic gestures. The dotted box in the figure is the effective area for dynamic gestures. The dynamic gestures in the effective area are collected by setting the position of the vehicle-mounted camera. At the same time, combined with the dynamic gesture user usage scenario, the dynamic gesture response area is increased to reduce the misrecognition of similar gestures such as turning the steering wheel, touching the belly, patting the arms, and unconscious hand waving, reduce the false trigger rate, and thus enhance the user's cockpit gesture control function experience.

[0033] The dynamic gesture collection includes not only positive sample data based on the gesture definition, but also negative sample data that is close to the gesture definition but cannot be responded to according to the user scenario. At the same time, a large amount of natural driving data of vehicles needs to be collected for testing misrecognition.

[0034] In order to improve the robustness of the model, data collection needs to be carried out under different lighting, different angles, different user habits and other scenarios to improve the richness of the data. In addition, the collected data needs to be cleaned and preprocessed. In the actual user usage scenario, the effective control area of ​​gestures in the car is specified to reduce misrecognition, which greatly reduces gesture misrecognition and improves the user experience.

[0035] Furthermore, the present invention classifies dynamic gestures based on the user's hand motion information, including classification and recognition of three categories: left swipe, right swipe, and ignore.

[0036] See also Figure 2 , combined with the user's actual gesture control habits, the first N frames of the gesture action output by the dynamic gesture recognition model, if at least one frame has all the key points of the hand falling within the response area, then the gesture is judged to be a valid gesture; otherwise, this gesture is an invalid gesture. The dynamic gesture recognition model input mainly includes the 21 hand key point positions and confidence information of the same Track ID n frames.

[0037] See also Figure 3 , which is a misidentification reduction process based on the regional optimization strategy of the present invention, specifically includes the following steps:

[0038] 1) Data collection: In a real car, data is collected through the on-board camera. In order to improve the robustness of the model, data collection needs to be carried out under different lighting, different angles, different user habits and other scenarios to improve the richness of the data. In addition, the collected data needs to be cleaned and preprocessed. Among them, the data collection not only includes positive sample data based on the gesture definition, but also needs to consider negative sample data that is close to the gesture definition but cannot respond according to the user scenario. In addition, a large amount of natural driving data of vehicles needs to be collected for testing misrecognition.

[0039] 2) Hand detection: The collected data is passed through the hand detection module to obtain the position of the gesture area in the image, the upper left corner (x1, y1), the lower right corner (x2, y2) and the confidence of the hand frame. In order to reduce the impact of hand detection on the classification accuracy, the gesture frame is expanded outward by 0.5 times. The expansion method is as follows:

[0040] x1'=max(0,x1-(x2-x1)*0.5), y1'=max(0,y1-(y2-y1)*0.5),

[0041] x2'=min(w,x2-(x2-x1)*0.5), y2'=min(h,y2+(y2-y1)*0.5).

[0042] Among them, w and h are the width and height of the image resolution, x1 and y1 are the coordinates of the upper left corner of the gesture area, and x2 and y2 are the coordinates of the lower right corner. According to the position of the area after expansion, the gesture images are cropped and saved separately to form the training set, validation set, and test set.

[0043] 3) Hand tracking: Since the hand detector can output the hand position, the camera in the car is relatively fixed, and the user's hand position is also relatively fixed during the gesture execution process, the hand area in the image is tracked and given a track ID by utilizing the hand size change relationship, the image similarity relationship, and the relative position relationship of the hand in the image, to ensure that the gesture of the performer can be correctly recognized.

[0044] If the hand has been detected, the key point detection model is input based on the hand image detected in the hand detection stage to obtain the position and confidence of 21 hand key points. The distribution of the 21 hand key points is as follows: one at the wrist; one key point at each joint from the base to the fingertip of each finger, that is, 4 key points are defined for each finger (including the thumb).

[0045] The dynamic gesture recognition model is based on a convolutional neural network. The data processed by the hand tracking model and the key point detection model are input into the model for multiple iterations of training and model parameters. The trained dynamic gesture classification model can effectively learn the features of different categories of images and perform recognition. Among them, the dynamic gesture recognition model mainly contains the 21 hand key point positions and confidence information of the same Track ID n frames.

[0046] Because the classification results of a single-frame gesture are not reliable enough, the classification results need to be processed. If N (4) consecutive frames are the same action and the confidence is greater than Score_limit (0.9), then the action is confirmed. Here, N and Score_limit are both thresholds, which can be parameterized and need to be adjusted to appropriate values ​​through experiments.

[0047] 5) Area optimization and restriction strategy: In the actual user usage scenarios, there are many unconscious action features that are basically consistent with the features of dynamic gestures, resulting in gesture misrecognition. Such as the driver turning the steering wheel, touching the belly, waving hands unconsciously while talking, etc. In order to further reduce the misrecognition situation, the present invention combines the user's actual gesture control habits to stipulate the effective gesture control area in the car. Among the first N frames of the gesture action output by the dynamic gesture recognition model, if at least one frame has all the key points of the hand falling within the response area, the gesture is judged to be a valid gesture. Otherwise, this gesture is an invalid gesture. The effective area of ​​​​dynamic gestures mainly includes the center console of the vehicle and the middle of the back row. This strategy greatly reduces the misrecognition of gestures and improves the user experience.

[0048] 6) Vehicle application event response: Output events to the corresponding application interface, such as switching songs, going home, going to work, opening windows, etc. The vehicle application responds to input events functionally.

[0049] Without increasing the hardware cost, the present invention only adds a left-right swipe dynamic gesture response area on the basis of the original left-right swipe dynamic gesture recognition model. If the dynamic gesture recognition model determines that the current action is a left swipe or a right swipe, and all the key points of the hand in at least one of the previous N frames fall within the response area, then the gesture is determined to be a valid gesture intended by the user, and the corresponding instruction is sent to the vehicle end, and the corresponding controller on the vehicle end responds to the corresponding gesture function. Otherwise, the gesture is considered to be an invalid gesture, and the corresponding controller on the vehicle end does not respond to this function.

[0050] In summary, the method for reducing gesture misrecognition based on the regional optimization and restriction strategy of the present invention stipulates the effective control area of ​​gestures in the car to reduce misrecognition in the actual use scenario of the user, greatly reduces gesture misrecognition, and improves the user experience. Based on the user's hand movement information, dynamic gestures are classified. At the same time, the dynamic gesture response area is increased to reduce the misrecognition of similar gestures such as turning the steering wheel, touching the belly, patting the arm, and unconscious hand waving, and reduce the false trigger rate, thereby improving the user's cockpit gesture control function experience. Combined with the user's actual gesture control habits, the first N frames of the gesture action output by the dynamic gesture recognition model, at least one frame of all the key points of the hand falls within the response area, then the gesture is judged to be a valid gesture; otherwise, this gesture is an invalid gesture. The present invention effectively solves many problems in the prior art such as strong interference from the in-car environment, uncertainty of key areas, and non-standardization of user operation gestures, resulting in a high gesture misrecognition rate, poor user experience, and even hidden dangers.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A gesture misrecognition reduction method based on a regional optimization strategy, characterized in that: A gesture response area is set in the actual vehicle, corresponding to the effective camera area range of the vehicle-mounted camera; through gesture recognition in the response area, input events are output to the corresponding application interface for functional response; the gesture response area is a dynamic gesture area; The dynamic gesture includes positive sample data made according to the gesture definition and the proximity gesture definition considered according to the user scenario, and a dynamic gesture recognition model is established; If all the key points of the hand in at least one frame of the first N frames of the gesture action output by the dynamic gesture recognition model fall within the response area, the gesture is judged to be a valid gesture; otherwise, the gesture is an invalid gesture.

2. The gesture misrecognition reduction method based on the regional optimization strategy according to claim 1 is characterized in that: The dynamic gesture area is the center control position and the middle position of the back row of the vehicle; the dynamic gestures in the gesture response area are collected by setting the position of the vehicle camera.

3. The gesture misrecognition reduction method based on the regional optimization strategy according to claim 1 is characterized in that: Perform data cleaning and data preprocessing on the collected dynamic gesture data.

4. The gesture misrecognition reduction method based on the regional optimization strategy according to claim 1 is characterized in that: The dynamic gestures are classified into three categories: left swipe, right swipe and ignore.

5. The gesture misrecognition reduction method based on the regional optimization strategy according to claim 1 is characterized in that: The dynamic gesture recognition model input includes the positions and confidence information of 21 hand key points of n frames with the same Track ID.

6. The gesture misrecognition reduction method based on the regional optimization strategy according to claim 1 is characterized in that: The following false positive reduction process is included: 1) Data collection: In a real car, dynamic gestures in the gesture response area are collected through the on-board camera; 2) Hand detection: The collected dynamic gesture data is passed through the hand detection module to obtain the position of the gesture area in the image, the upper left corner (x1, y1), the lower right corner (x2, y2) and the hand frame confidence; 3) Hand tracking: Track the hand area in the image and give it a track ID to ensure that the gesture of the performer can be correctly recognized. If the hand has been detected, the hand image detected in the hand detection stage is input into the key point detection model to obtain the position and confidence of the hand key points. The dynamic gesture recognition model is based on a convolutional neural network. The data processed by the hand tracking model and the key point detection model are input into the model for multiple iterations of training and model parameters. The trained dynamic gesture classification model can effectively learn the features of different categories of images and perform recognition. Among them, the dynamic gesture recognition model mainly contains the position and confidence information of 21 hand key points of n frames with the same Track ID. 4) Continuous frame recognition: If N (4) consecutive frames are all the same action and the confidence is greater than Score_limit (0.9), then the action is confirmed. Here, N and Score_limit are both thresholds, which can be parameterized and need to be adjusted to appropriate values ​​through experiments. 5) Area optimization restriction strategy: If all the key points of the hand in at least one frame of the first N frames of the gesture action output by the dynamic gesture recognition model fall within the response area, the gesture is considered to be a valid gesture; otherwise, the gesture is considered to be an invalid gesture; 6) Vehicle application event response: Output the event to the corresponding application interface application to perform functional response to the input event.

7. The gesture misrecognition reduction method based on the regional optimization strategy according to claim 6 is characterized in that: In order to reduce the impact of hand detection on classification accuracy, the gesture frame is expanded outward by 0.5 times. The expansion method is as follows: x1'=max0,x1−(x2−x1)*0.5,y1'=max(0,y1−y2−y1*0.5), x2'=min(w,x2−x2−x1*0.5),y2'=min(h,y2+y2−y1*0.5); Where w and h are the width and height of the image resolution, x1 and y1 are the coordinates of the upper left corner of the gesture area, and x2 and y2 are the coordinates of the lower right corner. According to the location of the expanded area, the gesture images are cropped and saved separately to form a training set, a validation set, and a test set.

Images