Method and device for optimizing gesture recognition model, vehicle and storage medium

Abstract

The invention provides a method and device for optimizing a gesture recognition model, a vehicle and a storage medium. In some scenarios, when a user controls a vehicle component through a directional gesture, the gesture direction is consistent with the line-of-sight direction. The method comprises the steps of performing feature extraction on an eye image based on a target sight line estimation model to obtain a first directional feature; and performing feature extraction on the hand image based on the initial gesture recognition model to obtain a second directional feature. And optimizing the initial gesture recognition model based on the first pointing feature and the second pointing feature. In other words, when the fingers make directional gestures, the human eyes can also look towards the corresponding direction to optimize the initial gesture recognition model. The problem that the recognition rate is low due to finger joint flexibility, gesture individual difference and the like during gesture pointing recognition does not exist in sight line direction recognition, and therefore the target gesture recognition model obtained through the method has high recognition accuracy, and the gesture interaction experience of a vehicle user in a cabin can also be improved.

Description

Technical Field

[0001] The present application relates to the field of vehicles, and more specifically, to a method, a device, a vehicle, and a storage medium for optimizing a gesture recognition model in the field of vehicles. Background Art

[0002] With the development of vehicle technology, the use of gesture interaction in the cockpit is gradually becoming a reality and has been applied in some models. Gesture interaction technology allows vehicle users to control various functions of the vehicle through gestures in the cockpit without touching any physical buttons or screens. This interaction method not only improves the convenience of operation, but also provides vehicle users with a more intuitive and natural driving experience.

[0003] In some scenarios, a vehicle user can control the opening or closing of a sunroof in the vehicle by a directional gesture pointing upward; a vehicle user can control the opening or closing of a door on the right side of the vehicle by a directional gesture pointing right.

[0004] However, due to the flexibility of finger joints and individual differences in gestures, some vehicle users bend their fingers slightly when making directional gestures. This will hinder the gesture recognition model from determining the direction of the gesture, affect the accuracy of directional gesture recognition, and thus cause a poor gesture interaction experience for vehicle users in the cabin. Summary of the invention

[0005] The present application provides a method, device, vehicle and storage medium for optimizing a gesture recognition model. The method can obtain a target gesture recognition model with a higher recognition accuracy.

[0006] In a first aspect, a method for optimizing a gesture recognition model is provided, the method comprising: acquiring an eye image and a corresponding hand image, wherein the sight line direction of the eye in the eye image is the same as the gesture direction in the hand image; performing feature extraction on the eye image based on a target sight line estimation model to obtain a first pointing feature, wherein the first pointing feature is used to describe the sight line direction of the eye in the eye image; performing feature extraction on the hand image based on an initial gesture recognition model to obtain a second pointing feature, wherein the second pointing feature is used to describe the gesture direction in the hand image; and optimizing the initial gesture recognition model based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, wherein the target gesture recognition model is used to recognize the gesture direction of a vehicle user in a cockpit.

[0007] In the above technical solution, in some scenarios, when the vehicle user controls the vehicle parts through directional gestures, the gesture direction is consistent with the line of sight direction. The method extracts features from the eye image based on the target line of sight estimation model to obtain a first directional feature; and extracts features from the hand image corresponding to the eye image based on the initial gesture recognition model to obtain a second directional feature. Among them, the first directional feature is used to describe the line of sight direction of the eye in the eye image, and the second directional feature is used to describe the gesture direction in the hand image. Further, based on the first directional feature and the second directional feature, the initial gesture recognition model is optimized to obtain a target gesture recognition model. That is, the method optimizes the initial gesture recognition model by assuming that when the finger makes a directional gesture in some scenarios, the human eye will also look in the corresponding direction. Since the line of sight direction does not have the problem of low recognition rate caused by the flexibility of the finger joints and the individual differences of the gestures during gesture pointing recognition, the method optimizes the target gesture recognition model obtained by the first directional feature and the second directional feature without increasing the complexity of the model, and has high recognition accuracy and robustness. In addition, it can also improve the gesture interaction experience of the vehicle user in the cabin.

[0008] In combination with the first aspect, in some possible implementations, the initial gesture recognition model is optimized based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, including: when eye images and corresponding hand images are collected by image acquisition devices at different positions in the cockpit, the first pointing feature and the second pointing feature are feature aligned to obtain a third pointing feature and a fourth pointing feature; based on the third pointing feature and the fourth pointing feature, the initial gesture recognition model is optimized to obtain the target gesture recognition model.

[0009] In the above technical solution, the eye image and the hand image may be collected by image acquisition devices at different positions, and the collection angles of the eye image and the hand image are different. When the method collects the eye image and the corresponding hand image through image acquisition devices at different positions in the cockpit, the first pointing feature and the second pointing feature are feature aligned, so that the third pointing feature and the fourth pointing feature (two pointing features) obtained have the same corresponding angle of view, and the two pointing features are easier to match. This method can better optimize the initial gesture recognition model by using the fact that when the finger makes a directional gesture (corresponding to the fourth pointing feature), the human eye will also look in the corresponding direction (corresponding to the third pointing feature), and a target gesture recognition model with a higher recognition accuracy can be obtained.

[0010] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the first pointing feature and the second pointing feature are feature aligned to obtain a third pointing feature and a fourth pointing feature, including: direction encoding the first pointing feature and the second pointing feature to obtain the direction of each feature point in the first pointing feature and the direction of each feature point in the second pointing feature; determining a reference direction, which is the direction of the first pointing feature and the second pointing feature after feature alignment; based on the direction of each feature point in the first pointing feature, the direction of each feature point in the second pointing feature and the reference direction, performing a rotation transformation on the first pointing feature and the second pointing feature to obtain the third pointing feature and the fourth pointing feature.

[0011] In the above technical solution, the first pointing feature and the second pointing feature are directional encoded, so that the direction of each feature point in the first pointing feature and the direction of each feature point in the second pointing feature can be obtained; further, the first pointing feature and the second pointing feature are rotated and transformed through the direction of each feature point in the first pointing feature, the direction of each feature point in the second pointing feature, and the reference direction after the first pointing feature and the second pointing feature are aligned, so as to obtain the third pointing feature and the fourth pointing feature. In other words, the method rotates the first pointing feature and the second pointing feature through the difference between the direction of each feature point in the pointing feature before the feature alignment and the reference direction, which can obtain accurate third pointing features and fourth pointing features.

[0012] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the initial gesture recognition model is optimized based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, including: constructing a similarity loss function based on the similarity between the first pointing feature and the second pointing feature; based on the similarity loss function, adjusting the parameters in the initial gesture recognition model to obtain the target gesture recognition model.

[0013] In the above technical solution, the method continuously iteratively adjusts the parameters in the initial gesture recognition model by controlling the second pointing feature extracted by the initial gesture recognition model to be as close as possible to the first pointing feature. That is to say, the parameters in the initial gesture recognition model are adjusted based on the similarity loss function constructed based on the similarity between the first pointing feature and the second pointing feature. When the number of iterations reaches a certain number, the similarity between the first pointing feature and the second pointing feature is very high, and the second pointing feature is closer to the first pointing feature, and the corresponding similarity loss function can reflect the extremely small difference between the first pointing feature and the second pointing feature. In this way, the target gesture recognition model obtained by adjusting the parameters based on the similarity loss function can have a good feature extraction capability in the subsequent process, and extract the pointing feature that is as accurate as the target sight estimation model, and then the target gesture recognition model can classify the accurate gesture pointing based on the pointing feature.

[0014] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, a similarity loss function is constructed based on the similarity between the first directional feature and the second directional feature, including: performing feature fusion of the first directional feature and the second directional feature to obtain a fifth directional feature; and constructing the similarity loss function based on the similarity between the second directional feature and the fifth directional feature.

[0015] In the above technical solution, the method continuously and iteratively adjusts the parameters in the initial gesture recognition model by controlling the second pointing feature extracted by the initial gesture recognition model to be as close as possible to the fifth pointing feature as a standard. The fifth pointing feature is a fusion feature between the first pointing feature and the second pointing feature. The fifth pointing feature has both the feature of the eye's sight direction and the feature of the gesture pointing. Therefore, the feature information covered by the fifth pointing feature is more comprehensive. To a certain extent, it can optimize the recognition accuracy when the directional gesture is blocked but the eyes are not blocked. In this way, when the parameters in the initial gesture recognition model are adjusted based on the similarity loss function constructed based on the similarity between the fifth pointing feature and the second pointing feature, a more accurate target gesture recognition model can be obtained. In this way, the target gesture recognition model obtained by adjusting the parameters based on the similarity loss function can have better feature extraction capabilities in the subsequent process, extract more accurate pointing features with the target sight estimation model, and then the target gesture recognition model can classify more accurate gesture pointing based on the pointing feature.

[0016] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the first pointing feature is feature-fused with the second pointing feature to obtain a fifth pointing feature, including: feature-splicing the first pointing feature and the second pointing feature to obtain the fifth pointing feature; or, based on the contribution of the first pointing feature and the second pointing feature to the determination of the target gesture recognition model, determining a first set of weight coefficients and a second set of weight coefficients; based on the first set of weight coefficients and the second set of weight coefficients, weightedly fusing the first pointing feature with the second pointing feature to obtain the fifth pointing feature.

[0017] In the above technical solution, by performing feature fusion on the first pointing feature and the second pointing feature in a variety of ways, a variety of different fused features (fifth pointing feature) can be obtained, which increases the diversity of fused features. In addition, the feature fusion method of feature splicing can integrate complete feature information from different sources, retain multi-dimensional feature information, and increase the expressiveness of the obtained target gesture recognition model. The feature fusion method of weighted fusion assigns different weights based on the importance of different pointing features, which can highlight certain important features while suppressing less important features. This method helps the target gesture recognition model to better focus on key information, thereby improving the performance and decision quality of the target gesture recognition model.

[0018] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the method also includes: when a vehicle user is present in the vehicle cabin, collecting a hand image of the vehicle user; inputting the hand image of the vehicle user into the target gesture recognition model, and using the target gesture recognition model to recognize the gesture direction in the hand image of the vehicle user to obtain the target direction; when the vehicle user outputs voice information, recognizing the voice information to obtain the target operation that the vehicle user expects the vehicle component corresponding to the target direction to perform; and controlling the vehicle component to perform the target operation.

[0019] In the above technical solution, a process of multi-modal voice gesture interaction by a vehicle user in a vehicle cockpit is described. Specifically, when a vehicle user is present in the vehicle cockpit, the method uses a target gesture recognition model to identify the gesture direction in the hand image of the vehicle user to obtain the target direction. Since the target gesture recognition model is optimized by the corresponding line of sight direction, the target direction is more accurate. Furthermore, when the vehicle user outputs voice information, the method recognizes the voice information to obtain the target operation that the vehicle user expects the target pointing to perform on the vehicle component, thereby realizing the corresponding operation on the vehicle component. That is to say, after the method recognizes the accurate gesture direction through the target gesture recognition model, it further combines the voice information of the vehicle user to control the vehicle component. This can realize the required operation of the vehicle component by the vehicle user and can increase the voice gesture interaction experience of the vehicle user in the cockpit.

[0020] In a second aspect, a device for optimizing a gesture recognition model is provided, the device comprising: an acquisition module, used to acquire an eye image and a corresponding hand image, wherein the sight line direction of the eye in the eye image is the same as the gesture direction in the hand image; a determination module, used to: perform feature extraction on the eye image based on a target sight line estimation model to obtain a first pointing feature, wherein the first pointing feature is used to describe the sight line direction of the eye in the eye image; perform feature extraction on the hand image based on an initial gesture recognition model to obtain a second pointing feature, wherein the second pointing feature is used to describe the gesture direction in the hand image; an optimization module, used to optimize the initial gesture recognition model based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, wherein the target gesture recognition model is used to recognize the gesture direction of the vehicle user in the cockpit.

[0021] In combination with the second aspect, in some possible implementations, the device also includes: an alignment module, which is used to align the first pointing feature and the second pointing feature when collecting eye images and corresponding hand images through image acquisition devices at different positions in the cockpit to obtain a third pointing feature and a fourth pointing feature; the optimization module is specifically used to optimize the initial gesture recognition model based on the third pointing feature and the fourth pointing feature to obtain the target gesture recognition model.

[0022] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the alignment module is specifically used to: perform direction encoding on the first pointing feature and the second pointing feature to obtain the direction of each feature point in the first pointing feature and the direction of each feature point in the second pointing feature; determine a reference direction, which is the direction of the first pointing feature and the second pointing feature after feature alignment; based on the direction of each feature point in the first pointing feature, the direction of each feature point in the second pointing feature and the reference direction, perform rotational transformation on the first pointing feature and the second pointing feature to obtain the third pointing feature and the fourth pointing feature.

[0023] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the device also includes: a construction module, used to construct a similarity loss function based on the similarity between the first pointing feature and the second pointing feature; the optimization module is specifically used to adjust the parameters in the initial gesture recognition model based on the similarity loss function to obtain the target gesture recognition model.

[0024] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the construction module is specifically used to: perform feature fusion of the first directional feature and the second directional feature to obtain a fifth directional feature; and construct the similarity loss function based on the similarity between the second directional feature and the fifth directional feature.

[0025] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the determination module is specifically used to: perform feature splicing of the first pointing feature and the second pointing feature to obtain the fifth pointing feature; or, determine a first set of weight coefficients and a second set of weight coefficients based on the contribution of the first pointing feature and the second pointing feature to determining the target gesture recognition model respectively; based on the first set of weight coefficients and the second set of weight coefficients, perform weighted fusion of the first pointing feature and the second pointing feature to obtain the fifth pointing feature.

[0026] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the device also includes: a collection module, which is used to collect a hand image of the vehicle user when there is a vehicle user in the vehicle cabin; the determination module is also used to: input the hand image of the vehicle user into the target gesture recognition model, and the target gesture recognition model recognizes the gesture direction in the hand image of the vehicle user to obtain the target direction; when the vehicle user outputs voice information, the voice information is recognized to obtain the target operation that the vehicle user expects the vehicle component corresponding to the target direction to perform; the device also includes: a control module, which is used to control the vehicle component to perform the target operation.

[0027] In a third aspect, a vehicle is provided, comprising a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the vehicle executes the method in the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, a computer-readable storage medium is provided, which stores an executable program code. When the executable program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of a scenario of using a vehicle provided in an embodiment of the present application;

[0030] Figure 2 is a schematic flow chart of a method for optimizing a gesture recognition model provided in an embodiment of the present application;

[0031] Figure 3 is a schematic diagram of an image acquisition device at different positions provided in an embodiment of the present application;

[0032] Figure 4 is a schematic diagram for describing the size of a pointing feature provided in an embodiment of the present application;

[0033] Figure 5 is a structural block diagram of an optimized gesture recognition model provided in an embodiment of the present application;

[0034] Figure 6 is a structural schematic diagram of a device for optimizing a gesture recognition model provided in an embodiment of the present application;

[0035] Figure 7 It is a structural schematic diagram of a vehicle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0037] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0038] Figure 1 It is a schematic diagram of a scenario of using a vehicle provided in an embodiment of the present application.

[0039] In some vehicle usage scenarios, when a vehicle user makes a directional gesture, the vehicle can identify the direction of the gesture through a gesture recognition model, and then control the opening or closing of the vehicle component corresponding to the gesture.

[0040] For example, Figure 1 As shown in (a) of FIG. 1 , when the gesture points to the upward direction u, vehicle A can control the sunroof corresponding to the upward direction to open; or, Figure 1 As shown in (b) in the figure, when the gesture points to the right direction d, vehicle B can control the door corresponding to the right direction to close.

[0041] However, due to the flexibility of finger joints and individual differences in gestures, some vehicle users usually bend their fingers slightly when making directional gestures. This will hinder the gesture recognition model from determining the direction of the gesture, affecting the accuracy of directional gesture recognition, and thus resulting in a poor gesture interaction experience for vehicle users in the cabin.

[0042] Figure 2 It is a schematic flowchart of a method for optimizing a gesture recognition model provided in an embodiment of the present application.

[0043] It should be understood that the method for optimizing the gesture recognition model provided in the embodiment of the present application is applied to a vehicle (eg Figure 1 Vehicle A or vehicle B in the vehicle), or, the method for optimizing the gesture recognition model can also be applied to the vehicle controller in the vehicle.

[0044] For example, Figure 2 As shown, the method 200 includes:

[0045] Step 201: The vehicle acquires an eye image and a corresponding hand image, wherein the sight direction of the eye in the eye image is the same as the gesture direction in the hand image.

[0046] It should be understood that in the above step 201, "eye image" refers to the image area including the eyes, and "hand image" refers to the image area including the hands. "Corresponding hand image" means that the hand image and the eye image are from the same user and the corresponding acquisition time is the same.

[0047] It should also be understood that "the sight line direction of the eye in the eye image is the same as the gesture direction in the hand image" in the above step 201 means that the sight line direction of the eye is the same as the gesture direction when the user makes a directional gesture (the same direction as the finger pointing). In addition, the "same" does not necessarily mean that the direction deviation between the sight line direction and the gesture direction is zero. The method allows a small direction deviation between the sight line direction and the gesture direction. When describing the direction in azimuth, the method allows a preset angle deviation between the sight line direction and the gesture direction. In some embodiments, when describing the direction in azimuth and taking the north direction as the reference, the sight line direction is 30 degrees north by east, and the gesture direction is 29 degrees north by east, that is, the preset angle deviation is 1 degree.

[0048] It should also be understood that, generally, image acquisition devices can be installed at different locations in the vehicle cabin. Image acquisition devices at different locations can respectively acquire eye images and hand images; image acquisition devices at the same location can acquire user images to acquire eye images and hand images.

[0049] In some embodiments, the vehicle in step 201 acquires an eye image and a corresponding hand image, including: the vehicle acquires the eye image through an image acquisition device at a first position, and acquires the hand image through an image acquisition device at a second position, the first position and the second position are different; or, the vehicle acquires a user image through an image acquisition device at the same position; the vehicle determines a first key point from the user image, the first key point includes a left corner of the eye, a right corner of the eye, a center of the upper part of the eye, and a center of the lower part of the eye; the vehicle determines the eye boundary based on the first key point; the vehicle crops out the image area surrounded by the eye boundary in the user image, and determines the image area surrounded by the eye boundary as the eye image; and the vehicle determines a second key point from the user image, the second key point includes a wrist joint, a palm, a finger joint, a fingertip, and a finger base; the vehicle determines a hand boundary based on the second key point; the vehicle crops out the image area surrounded by the hand boundary in the user image, and determines the image area surrounded by the hand boundary as the hand image.

[0050] In some embodiments, the image acquisition devices at different positions include those installed in the vehicle in front of the left and right sides of the user corresponding to the user image.

[0051] In some embodiments, the vehicle determines a first key point from the user image, including: the vehicle aligns the user image to adjust the user image to a target size and a target direction; the vehicle extracts features from the aligned user image to obtain human eye features; and the vehicle locates the first key point in the aligned user image based on the human eye features.

[0052] It should be understood that the vehicle can obtain the user's eye features by extracting features from the aligned user image, and the eye features may include the left corner of the eye, the right corner of the eye, the center of the upper part of the eye, and the center of the lower part of the eye. The vehicle can locate the first key point by searching for the positions of the left corner of the eye, the right corner of the eye, the center of the upper part of the eye, and the center of the lower part of the eye in the aligned user image. Therefore, the vehicle can locate the first key point in the aligned user image based on the mouth feature.

[0053] It should also be understood that the process of "the vehicle determines the second key point from the user image" in the above solution is similar to the process of "the vehicle determines the first key point from the user image", except that the former extracts the human eye features from the aligned user image and locates the first key point in the aligned user image based on the human eye features, while the latter extracts the hand features from the aligned user image and locates the second key point in the aligned user image based on the hand features. The hand features include wrist joints, palms, finger joints, fingertips and finger bases.

[0054] In step 202, the vehicle extracts features from the eye image based on a target sight line estimation model to obtain a first directional feature, where the first directional feature is used to describe the sight line direction of the eye in the eye image.

[0055] It should be understood that the "target sight line estimation model" in the above step 202 refers to a sight line estimation model that has been trained by a training image set and tested by a test image set. Wherein, the training image set includes M eye images, and the test image set includes N eye images, and there are corresponding eye areas and corresponding sight line directions in the eye images, and M and N are positive integers greater than 1. In some embodiments, M is greater than or equal to N. The target sight line estimation model can further detect and process the eye images to obtain the eye areas; and further accurately estimate the sight line direction of the eyes in the eye areas. Wherein, the eye area is the image area where the eyes are located, and the difference between the eye image and the eye area is that the eye image includes the eye area. It should also be understood that the target sight line estimation model has the ability to accurately estimate the sight line direction, and the sight line direction does not have problems such as finger joint flexibility and individual differences in gestures during gesture pointing recognition. Therefore, the method assumes that the first pointing feature determined based on the target sight line estimation model is a standard feature.

[0056] Among them, the "target sight line estimation model can further detect and process the eye image to obtain the eye area" in the above scheme can be regarded as detecting the eye area in the eye image through the target detection sub-model in the target sight line estimation model, wherein the "target" is the eye area. The "target sight line estimation model accurately estimates the sight line direction of the eye in the eye area" in the above scheme specifically includes: the target sight line estimation model extracts features from the eye area to obtain pointing features; and then the target sight line estimation model classifies the pointing features to obtain the sight line direction, which can be specifically regarded as determining the sight line direction through the classification sub-model in the target sight line estimation model.

[0057] It should be understood that in some embodiments, the eye detection submodel is a YOLO model. The core of the YOLO model is to convert the target detection task into a regression problem, and directly predict the bounding box coordinates and category probabilities on the feature map of the eye image. This method allows the model to complete the detection of the entire eye image in one forward pass, thereby achieving fast and efficient target detection. It should also be understood that in some embodiments, the classification submodel is a backbone network model. In some embodiments, the backbone network model is a MobileNet model. The core of the MobileNet model is to use deep separable convolution to split the convolution filter into deep convolution and point-by-point convolution. Deep convolution is used to perform spatial transformation on the input feature map, while point-by-point convolution is used to increase the number of channels. This significantly reduces the amount of computation and model size, so that the classification submodel can run efficiently.

[0058] In some embodiments, the vehicle detects the eye area in the eye image through the eye detection sub-model in the target sight line estimation model; the vehicle extracts features of the eye area through the convolution layer in the classification sub-model in the target sight line estimation model to obtain pointing features; the vehicle classifies the pointing features through the fully connected layer in the classification sub-model in the target sight line estimation model to obtain the yaw angle and pitch angle corresponding to the sight line.

[0059] It should be understood that in the above scheme, the "convolution layer" can be regarded as a Conv layer, and the "fully connected layer" can be regarded as an FC layer. In addition, the "yaw angle corresponding to the line of sight" refers to the rotation angle of the eye's line of sight on the horizontal plane relative to the target reference direction (such as straight ahead), and the "pitch angle corresponding to the line of sight" refers to the angle formed by the eye's line of sight and the horizontal line when the eye's line of sight moves up or down. In other words, this method characterizes the line of sight direction by the yaw angle and pitch angle corresponding to the line of sight.

[0060] In some embodiments, the method for determining the target line of sight estimation model in step 202 includes: the vehicle acquires a training image set and a test image set, the training image set includes M eye images and the test image set includes N eye images, and there are corresponding eye areas and corresponding target line of sight directions in the eye images; the vehicle trains a candidate line of sight estimation model based on the M eye images and the first line of sight estimation model; the vehicle tests the target line of sight estimation model based on the candidate line of sight estimation model and the N eye images.

[0061] It should be understood that the “first line of sight estimation model” in the above scheme is the basic line of sight estimation model, which has multiple capabilities such as generating candidate eye areas, extracting directional features and predicting line of sight direction, but these multiple capabilities need to be further improved.

[0062] In some embodiments, the vehicle trains a candidate line of sight estimation model based on the M eye images and the first line of sight estimation model, including: the vehicle extracts features of the eye area in the first eye image of the M eye images through the first line of sight estimation model to obtain a first pointing feature; the vehicle classifies the first pointing feature through the first line of sight estimation model to obtain a first predicted line of sight direction; the vehicle determines a first loss value between the first predicted line of sight direction and a corresponding first target line of sight direction; when the first loss value is greater than a preset loss value, the vehicle adjusts parameters in the first line of sight estimation model to obtain a second line of sight estimation model; the vehicle classifies the M eye images through the second line of sight estimation model The vehicle extracts features from the eye area in the second eye image in the image to obtain a second pointing feature; the vehicle classifies the second pointing feature through the second sight line estimation model to obtain a second predicted sight line direction; the vehicle determines a second loss value between the second predicted sight line direction and the corresponding second target sight line direction; when the second loss value is greater than the preset loss value, the vehicle adjusts the parameters in the second sight line estimation model to obtain a third sight line estimation model; and so on, when the Mth loss value between the Mth predicted sight line direction obtained based on the Mth eye image and the corresponding Mth target sight line direction is less than the preset loss value, the vehicle determines the Mth sight line estimation model as the candidate sight line estimation model.

[0063] In some embodiments, the vehicle tests and obtains the target line of sight estimation model based on the candidate line of sight estimation model and the N eye images, including: the vehicle extracts features of the eye area in each of the N eye images through the candidate line of sight estimation model to obtain multiple directional features; the vehicle classifies the multiple directional features through the candidate line of sight estimation model to obtain multiple test line of sight directions; the vehicle determines the estimation accuracy of the candidate line of sight estimation model based on the multiple test line of sight directions and the corresponding multiple target line of sight directions; when the estimation accuracy is greater than or equal to a probability threshold, the vehicle determines the candidate line of sight estimation model as the target line of sight estimation model; when the estimation accuracy is less than the probability threshold, the vehicle continues to train the candidate line of sight estimation model based on a new training image set.

[0064] In some embodiments, the vehicle determines the estimation accuracy of the candidate line of sight estimation model based on multiple test line of sight directions and corresponding multiple target line of sight directions, including: the vehicle determines the number of directions in the multiple test line of sight directions that are the same as the corresponding multiple target line of sight directions; the vehicle determines the ratio between the number of directions and N as the estimation accuracy.

[0065] In step 203, the vehicle extracts features from the hand image based on the initial gesture recognition model to obtain a second pointing feature, where the second pointing feature is used to describe the gesture pointing in the hand image.

[0066] It should be understood that the "initial gesture recognition model" in the above step 203 is a basic gesture recognition model, and the initial gesture recognition model has multiple capabilities such as generating candidate hand regions, extracting pointing features, and predicting gesture pointing. However, these multiple capabilities may not be sufficient and need to be further improved. In other words, the initial gesture recognition model needs to be optimized to obtain a target gesture recognition model.

[0067] It should also be understood that the method optimizes the initial gesture recognition model based on the fact that when the finger makes a directional gesture, the human eye will also look in the corresponding direction to obtain the target gesture recognition model. The first directional feature is used to describe the direction of the eye's sight (i.e. the direction the human eye looks at), and the second directional feature is used to describe the gesture direction (i.e. the direction the finger points when the finger makes a directional gesture). The second directional feature should be infinitely close to the first directional feature. Therefore, specifically, the method optimizes the initial gesture recognition model based on the first directional feature and the second directional feature to obtain the target gesture recognition model.

[0068] In step 204 , the vehicle optimizes the initial gesture recognition model based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, where the target gesture recognition model is used to recognize the gesture direction of the vehicle user in the cabin.

[0069] It should be understood that the technical solution in the above step 204 can be regarded as a process of optimizing the initial gesture recognition model based on the first pointing feature obtained by feature extraction of the eye image and the second pointing feature obtained by feature extraction of the corresponding hand image. Of course, steps 201 to 204 in the method describe the inventive concept of optimizing the initial gesture recognition model. In addition, optimization mainly refers to adjusting the parameters in the initial gesture recognition model, and the number of optimizations is often large.

[0070] It should also be understood that the target gesture recognition model can further detect and process the hand image to obtain the hand region; and further accurately estimate the gesture direction in the hand region. The hand region is the image region where the hand is located, and the difference between the hand image and the hand region is that the hand image includes the hand region. It should also be understood that the target gesture recognition model has the ability to accurately estimate the gesture direction.

[0071] In some embodiments, the vehicle detects the hand area in the hand image through the hand detection sub-model in the target gesture recognition model; the vehicle extracts features of the hand area through the convolution layer in the classification sub-model in the target gesture recognition model to obtain pointing features; the vehicle classifies the pointing features through the fully connected layer in the classification sub-model in the target gesture recognition model to obtain the yaw angle and pitch angle corresponding to the finger.

[0072] It should be understood that in the above scheme, the "convolution layer" can be regarded as a Conv layer, and the "fully connected layer" can be regarded as an FC layer. In addition, the "yaw angle corresponding to the finger" refers to the rotation angle of the finger on the horizontal plane relative to the target reference direction (such as straight ahead), and the "pitch angle corresponding to the finger" refers to the angle formed by the finger and the horizontal line when the finger moves up or down. In other words, this method characterizes the gesture pointing by the yaw angle and pitch angle corresponding to the finger.

[0073] In some embodiments, the target reference direction is directly ahead, when the yaw angle corresponding to the finger is 0 degrees and the pitch angle corresponding to the finger is 0 degrees, the gesture points to the forward direction; the target reference direction is directly ahead, when the yaw angle corresponding to the finger is 90 degrees and the pitch angle corresponding to the finger is 0 degrees, the gesture points to the right direction; the target reference direction is directly ahead, when the yaw angle corresponding to the finger is 0 degrees and the pitch angle corresponding to the finger is 90 degrees, the gesture points to the upward direction.

[0074] In a possible implementation, the vehicle in step 204 optimizes the initial gesture recognition model based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, including: when collecting eye images and corresponding hand images through image acquisition devices at different positions in the cockpit, the vehicle aligns the first pointing feature and the second pointing feature to obtain a third pointing feature and a fourth pointing feature; the vehicle optimizes the initial gesture recognition model based on the third pointing feature and the fourth pointing feature to obtain the target gesture recognition model.

[0075] It should be understood that in the above scheme, "different positions" refer to multiple different positions relative to the target position in the cockpit with the target position as a reference point. Normally, the image acquisition device in the cockpit is installed on the A-pillar in front of the driver or on the ceiling directly above the driver to monitor the driver's behavior; the image acquisition device can also be installed on the top of the front windshield to monitor all passengers or the situation inside the car; the image acquisition device can also be installed on the B-pillar to obtain images of rear passengers. In addition, "capturing eye images and corresponding hand images through image acquisition devices at different positions in the cockpit" specifically refers to capturing eye images through an image acquisition device at a first position in the cockpit, and capturing corresponding hand images through an image acquisition device at a second position in the cockpit, and the first position is different from the second position.

[0076] It should also be understood that in some scenarios, the direction of the vehicle user's line of sight points to the target vehicle component, and the direction pointed by the user's finger also points to the target vehicle component. However, the first image acquisition device that captures the eye image and the second image acquisition device that captures the corresponding hand image may be in different positions and have different acquisition angles. This may result in different directions being determined based on the eye image and the hand image, that is, the line of sight direction is different from the gesture direction. Therefore, the method needs to overcome the difference between the line of sight direction and the gesture direction caused by the difference between the acquisition angles of image acquisition devices at different positions. Specifically, the method overcomes the difference between the acquisition angles of image acquisition devices at different positions by aligning the first pointing feature used to describe the line of sight direction of the eye in the eye image and the second pointing feature used to describe the gesture direction in the hand image.

[0077] In the above technical solution, the eye image and the hand image may be collected by image acquisition devices at different positions, and the collection angles of the eye image and the hand image are different. When the method collects the eye image and the corresponding hand image through image acquisition devices at different positions in the cockpit, the first pointing feature and the second pointing feature are feature aligned, so that the third pointing feature and the fourth pointing feature (two pointing features) obtained have the same corresponding angle of view, and the two pointing features are easier to match. This method can better optimize the initial gesture recognition model by using the fact that when the finger makes a directional gesture (corresponding to the fourth pointing feature), the human eye will also look in the corresponding direction (corresponding to the third pointing feature), and a target gesture recognition model with a higher recognition accuracy can be obtained.

[0078] In some embodiments, the image acquisition device is a vehicle-mounted camera.

[0079] Figure 3 It is a schematic diagram of an image acquisition device at different positions provided in an embodiment of the present application.

[0080] For example, the first image acquisition device is a first camera c1 for acquiring eye images, and the second image acquisition device is a second camera c2 for acquiring corresponding hand and eye images, and the image acquisition devices at different positions in the cockpit are described. Figure 3 As shown in FIG. 1 , the first camera c1 and the second camera c2 are located at different positions in the cockpit. The first camera c1 collects eye images downward, and the second camera c2 collects hand images forward. In this way, different directions are determined based on the eye image and the hand image, that is, the line of sight is the downward direction, and the direction of the finger is the upward direction u. However, in the case of Figure 3 In the actual scene shown, the vehicle user is clearly pointing at the sunroof, and the line of sight should be upward. Therefore, the method needs to overcome the difference between the acquisition angles of the first camera c1 and the second camera c2 at different positions by aligning the first pointing feature used to describe the line of sight of the eye in the eye image and the second pointing feature used to describe the gesture direction in the hand image.

[0081] In a possible implementation, the vehicle performs feature alignment on the first pointing feature and the second pointing feature to obtain a third pointing feature and a fourth pointing feature, including: the vehicle performs direction encoding on the first pointing feature and the second pointing feature to obtain the direction of each feature point in the first pointing feature and the direction of each feature point in the second pointing feature; the vehicle determines a reference direction, which is the direction after the first pointing feature and the second pointing feature are feature aligned; the vehicle performs rotational transformation on the first pointing feature and the second pointing feature based on the directions of each feature point in the first pointing feature, the directions of each feature point in the second pointing feature and the reference direction to obtain the third pointing feature and the fourth pointing feature.

[0082] It should be understood that "direction encoding the first directional feature and the second directional feature" in the above solution refers to the process of quantizing the directional attributes of the first directional feature and the second directional feature.

[0083] In the above technical solution, the first pointing feature and the second pointing feature are directional encoded, so that the direction of each feature point in the first pointing feature and the direction of each feature point in the second pointing feature can be obtained; further, the first pointing feature and the second pointing feature are rotated and transformed through the direction of each feature point in the first pointing feature, the direction of each feature point in the second pointing feature, and the reference direction after the first pointing feature and the second pointing feature are aligned, so as to obtain the third pointing feature and the fourth pointing feature. In other words, the method rotates the first pointing feature and the second pointing feature through the difference between the direction of each feature point in the pointing feature before the feature alignment and the reference direction, which can obtain accurate third pointing features and fourth pointing features.

[0084] In some embodiments, the reference direction is a direction corresponding to the front of the vehicle.

[0085] It should be understood that the "direction corresponding to the front of the vehicle" in the above solution refers to the straight extension direction directly in front of the vehicle. In some embodiments, when the front of the vehicle is facing due north, the direction corresponding to the front of the vehicle is due north.

[0086] In some embodiments, the vehicle performs a rotational transformation on the first pointing feature and the second pointing feature based on the directions of the feature points in the first pointing feature, the directions of the feature points in the second pointing feature and the reference direction to obtain the third pointing feature and the fourth pointing feature, including: the vehicle determines the angular deviation between the direction of each feature point in the first pointing feature and the reference direction to obtain multiple first angular differences; the vehicle determines the angular deviation between the direction of each feature point in the second pointing feature and the reference direction to obtain multiple second angular differences; the vehicle rotates each feature point in the first pointing feature by a corresponding first angular difference to obtain the third pointing feature; the vehicle rotates each feature point in the second pointing feature by a corresponding second angular difference to obtain the fourth pointing feature.

[0087] In one possible implementation, the vehicle in step 204 optimizes the initial gesture recognition model based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, including: the vehicle constructs a similarity loss function based on the similarity between the first pointing feature and the second pointing feature; the vehicle adjusts the parameters in the initial gesture recognition model based on the similarity loss function to obtain the target gesture recognition model.

[0088] It should be understood that the "similarity loss function" in the above technical solution refers to a function used to measure the similarity between the second directional feature and the first directional feature, and is used to optimize the parameters in the initial gesture recognition model in machine learning and deep learning. The purpose of the similarity loss function is to minimize the difference between the second directional feature and the first directional feature. In other words, it is a process of learning the feature similarity between the second directional feature and the first directional feature.

[0089] In the above technical solution, the method continuously iteratively adjusts the parameters in the initial gesture recognition model by controlling the second pointing feature extracted by the initial gesture recognition model to be as close as possible to the first pointing feature. That is to say, the parameters in the initial gesture recognition model are adjusted based on the similarity loss function constructed based on the similarity between the first pointing feature and the second pointing feature. When the number of iterations reaches a certain number, the similarity between the first pointing feature and the second pointing feature is very high, and the second pointing feature is closer to the first pointing feature, and the corresponding similarity loss function can reflect the extremely small difference between the first pointing feature and the second pointing feature. In this way, the target gesture recognition model obtained by adjusting the parameters based on the similarity loss function can have a good feature extraction capability in the subsequent process, and extract the pointing feature that is as accurate as the target sight estimation model, and then the target gesture recognition model can classify the accurate gesture pointing based on the pointing feature.

[0090] In some embodiments, the similarity loss function is a cosine similarity loss function.

[0091] In some embodiments, the similarity loss function is a cosine similarity loss function, and the vehicle constructs the similarity loss function based on the similarity between the first directional feature and the second directional feature, including: the vehicle constructs the similarity loss function based on the following formula (1);

[0092] L(A,B)=1-cos(A,B) Formula (1)

[0093] Wherein, L(A, B) is the cosine similarity loss function, A is the first directional feature, B is the second directional feature, and cos(A, B) is the cosine similarity between the first directional feature and the second directional feature.

[0094] In some embodiments, the vehicle adjusts the parameters in the initial gesture recognition model based on the similarity loss function to obtain the target gesture recognition model, including: when the function value corresponding to the similarity loss function is greater than the target threshold, the vehicle adjusts the parameters in the initial gesture recognition model to obtain a candidate gesture recognition model; the vehicle obtains a first eye image and a corresponding first hand image, and the sight line direction of the eye in the first eye image is the same as the gesture direction in the first hand image; the vehicle extracts features of the first eye image based on the target sight line estimation model to obtain a sixth pointing feature, and the sixth pointing feature is used to describe the sight line direction of the eye in the first eye image; the vehicle extracts features of the first hand image based on the candidate gesture recognition model to obtain a seventh pointing feature, and the seventh pointing feature is used to describe the gesture direction in the first hand image; the vehicle optimizes the candidate gesture recognition model based on the sixth pointing feature and the seventh pointing feature to obtain the target gesture recognition model.

[0095] In some embodiments, the vehicle optimizes the candidate gesture recognition model based on the sixth pointing feature and the seventh pointing feature to obtain the target gesture recognition model, including: the vehicle constructs a similarity loss function based on the similarity between the sixth pointing feature and the seventh pointing feature; when the function value corresponding to the similarity loss function is less than the target threshold, the vehicle determines the candidate gesture recognition model as the target gesture recognition model.

[0096] It should be understood that when the function value corresponding to the similarity loss function constructed based on the similarity between the sixth pointing feature and the seventh pointing feature is still greater than the target threshold, the vehicle can also obtain a second eye image and a corresponding second hand image, and continue to repeat the steps in the above scheme until the function value corresponding to the similarity loss function constructed based on the similarity between the corresponding two pointing features is less than the target threshold, and the candidate gesture recognition model at this time is determined as the target gesture recognition model.

[0097] In a possible implementation, the vehicle constructs a similarity loss function based on the similarity between the first directional feature and the second directional feature, including: the vehicle performs feature fusion of the first directional feature and the second directional feature to obtain a fifth directional feature; the vehicle constructs the similarity loss function based on the similarity between the second directional feature and the fifth directional feature.

[0098] It should be understood that the "feature fusion" in the above scheme refers to combining the feature information of the first pointing feature and the feature information of the second pointing feature from different sources to obtain a richer pointing feature representation. In this way, the feature information used to optimize the initial gesture recognition model does not completely rely on the first pointing feature (a similarity loss function constructed based on the similarity between the first pointing feature and the second pointing feature, adjusting the parameters in the initial gesture recognition model, and obtaining the target gesture recognition model), but the feature information used to optimize the initial gesture recognition model may rely on the fusion feature between the first pointing feature and the second pointing feature (i.e., the fifth pointing feature, a similarity loss function constructed based on the similarity between the fifth pointing feature and the second pointing feature, adjusting the parameters in the initial gesture recognition model, and obtaining the target gesture recognition model).

[0099] In the above technical solution, the method continuously and iteratively adjusts the parameters in the initial gesture recognition model by controlling the second pointing feature extracted by the initial gesture recognition model to be as close as possible to the fifth pointing feature as a standard. The fifth pointing feature is a fusion feature between the first pointing feature and the second pointing feature. The fifth pointing feature has both the feature of the eye's sight direction and the feature of the gesture pointing. Therefore, the feature information covered by the fifth pointing feature is more comprehensive. To a certain extent, it can optimize the recognition accuracy when the directional gesture is blocked but the eyes are not blocked. In this way, when the parameters in the initial gesture recognition model are adjusted based on the similarity loss function constructed based on the similarity between the fifth pointing feature and the second pointing feature, a more accurate target gesture recognition model can be obtained. In this way, the target gesture recognition model obtained by adjusting the parameters based on the similarity loss function can have better feature extraction capabilities in the subsequent process, extract more accurate pointing features with the target sight estimation model, and then the target gesture recognition model can classify more accurate gesture pointing based on the pointing feature.

[0100] In some embodiments, the vehicle performs feature fusion of the first pointing feature with the second pointing feature to obtain a fifth pointing feature, including: the vehicle performs feature splicing of the first pointing feature with the second pointing feature to obtain the fifth pointing feature; or, the vehicle determines a first set of weight coefficients and a second set of weight coefficients based on the contribution of the first pointing feature and the second pointing feature to determining the target gesture recognition model; based on the first set of weight coefficients and the second set of weight coefficients, performs weighted fusion of the first pointing feature and the second pointing feature to obtain the fifth pointing feature.

[0101] It should be understood that in the above scheme, the "first set of weight coefficients" is the weight coefficient corresponding to the first directional feature when the first directional feature and the second directional feature are weightedly fused, and the "second set of weight coefficients" is the weight coefficient corresponding to the second directional feature when the first directional feature and the second directional feature are weightedly fused. "The contribution of the first directional feature and the second directional feature to the determination of the target gesture recognition model" refers to the role played by the first directional feature and the role played by the second directional feature in the process of determining the target gesture recognition model with a higher recognition accuracy.

[0102] In the above technical solution, by performing feature fusion on the first pointing feature and the second pointing feature in a variety of ways, a variety of different fused features (fifth pointing feature) can be obtained, which increases the diversity of fused features. In addition, the feature fusion method of feature splicing can integrate complete feature information from different sources, retain multi-dimensional feature information, and increase the expressiveness of the obtained target gesture recognition model. The feature fusion method of weighted fusion assigns different weights based on the importance of different pointing features, which can highlight certain important features while suppressing less important features. This method helps the target gesture recognition model to better focus on key information, thereby improving the performance and decision quality of the target gesture recognition model.

[0103] In some embodiments, the vehicle performs feature splicing on the first directional feature and the second directional feature to obtain the fifth directional feature, including: the vehicle determines whether the width and / or length of the first directional feature is the same as that of the second directional feature; when the width and length are the same, the vehicle performs feature splicing on the first directional feature and the second directional feature in the direction of the number of channels to obtain the fifth directional feature; when the widths are different, the vehicle performs a zero-filling operation on the directional feature with a smaller width in the width direction so that the width of the first directional feature is the same as that of the second directional feature; and the vehicle performs a zero-filling operation on the directional feature with the width direction and the directional feature without zero-filling in the direction of the number of channels. Perform feature splicing to obtain the fifth directional feature, the directional feature that is not padded with zeros is the directional feature of the first directional feature and the second directional feature excluding the directional feature padded with zeros in the width direction; and / or, in the case of different lengths, the vehicle performs a zero-padding operation on the directional feature with a smaller length in the length direction so that the length of the first directional feature is the same as the length of the second directional feature; and the vehicle performs feature splicing on the directional feature padded with zeros in the length direction and the directional feature that is not padded with zeros in the direction of the number of channels to obtain the fifth directional feature, the directional feature that is not padded with zeros is the directional feature of the first directional feature and the second directional feature excluding the directional feature padded with zeros in the length direction.

[0104] It should be understood that the size of the directional features (the first directional feature to the seventh directional feature) in this method can be characterized by the length, width and number of channels of the feature. The number of channels is also called depth. For example, Figure 4 The reference pointing feature shown in FIG. 3 has a channel number of 3, which means that the reference pointing feature is composed of features of three channels (feature a, feature b, and feature c). The length and width of features a, b, and c are all 2, which means that the size of each of the three features is 2*2, that is, each feature is composed of 4 elements, specifically, feature a is composed of elements 5, 3, 1, and 2; feature b is composed of elements 2, 6, 3, and 1; feature c is composed of elements 8, 3, 4, and 7.

[0105] In some embodiments, the degree of contribution is positively correlated with the size of the weight coefficient.

[0106] In some embodiments, the vehicle performs weighted fusion of the first directional feature and the second directional feature based on the first set of weight coefficients and the second set of weight coefficients to obtain the fifth directional feature, including: the vehicle determines the fifth directional feature based on the following formula (2);

[0107] C=W1*A+W2*B Formula (2)

[0108] Among them, C is the fifth directional feature, W1 is the first set of weight coefficients, A is the first directional feature, W2 is the second set of weight coefficients, and B is the second directional feature.

[0109] Figure 5 It is a structural block diagram of an optimized gesture recognition model provided in an embodiment of the present application.

[0110] For example, Figure 5 As shown, the vehicle obtains a user image through an on-board camera, and crops an eye image and a hand image from the user image; the vehicle detects and processes the eye image through the target detection submodel in the target sight line estimation model to obtain an eye area; the vehicle detects and processes the hand image through the target detection submodel in the initial gesture estimation model to obtain a hand area; the vehicle extracts features of the eye area through the classification submodel in the target sight line estimation model to obtain a first pointing feature; the vehicle extracts features of the hand area through the classification submodel in the initial gesture estimation model to obtain a second pointing feature; the vehicle constructs a similarity loss function based on the similarity between the first pointing feature and the second pointing feature; the vehicle optimizes the parameters in the initial gesture estimation model based on the similarity loss function to obtain a target gesture estimation model.

[0111] Furthermore, the classification sub-model in the target sight line estimation model classifies the first pointing feature to obtain the sight line direction; and the classification sub-model in the initial gesture estimation model classifies the second pointing feature to obtain the gesture direction.

[0112] In one possible implementation, after obtaining the target gesture recognition model, the method 200 also includes: when there is a vehicle user in the vehicle cabin, the vehicle collects a hand image of the vehicle user; the vehicle inputs the hand image of the vehicle user into the target gesture recognition model, and the target gesture recognition model recognizes the gesture pointing in the hand image of the vehicle user to obtain the target pointing; when the vehicle user outputs voice information, the vehicle recognizes the voice information to obtain the target operation that the vehicle user expects the vehicle component corresponding to the target pointing to perform; the vehicle controls the vehicle component to perform the target operation.

[0113] It should be understood that when the vehicle user controls the vehicle components through directional gestures, the direction of the gesture does not necessarily have to be consistent with the direction of sight. Figure 1When the upward gesture shown in (a) in the figure points to the sunroof, the vehicle user may not be looking at the sunroof. Therefore, in practical applications, the method can only identify the gesture pointing through the target gesture recognition model to determine the vehicle component that the vehicle user wants to control. In addition, there are various control operations for vehicle components. In some embodiments, the vehicle component is an air conditioner, which can control any one of turning the air conditioner on, off, heating up, cooling down, cooling, heating, reducing the wind speed, and increasing the wind speed. Therefore, it is also necessary to recognize the voice information of the vehicle user to determine the operation that the vehicle user expects the target pointing to perform on the corresponding vehicle component.

[0114] In the above technical solution, a process of multi-modal voice gesture interaction by a vehicle user in a vehicle cockpit is described. Specifically, when a vehicle user is present in the vehicle cockpit, the method identifies the gesture direction in the hand image of the vehicle user through a target gesture recognition model to determine the target direction. Since the target gesture recognition model is optimized by the corresponding line of sight direction, the target direction is more accurate. Furthermore, when the vehicle user outputs voice information, the method identifies the voice information to obtain the target operation that the vehicle user expects the target pointing to perform on the vehicle component corresponding to the vehicle component, thereby implementing the corresponding operation on the vehicle component. That is, after the method identifies the accurate gesture direction through the target gesture recognition model, it further combines the voice information of the vehicle user to control the vehicle component. This can realize the required operation of the vehicle user on the vehicle component and can increase the voice gesture interaction experience of the vehicle user in the cockpit.

[0115] In some embodiments, a method for determining the presence of a vehicle user in a vehicle cabin includes any one of the following: when it is detected that a pressure on a vehicle seat is greater than a preset pressure, the vehicle determines that there is a vehicle user in the cabin; when it is detected by a seat belt sensor that a seat belt is being used, the vehicle determines that there is a vehicle user in the cabin; based on the vehicle's identification information, the vehicle obtains from a vehicle server the number of mobile terminals used to access the vehicle server, and when the number is greater than a preset number, the vehicle determines that there is a vehicle user in the cabin.

[0116] It should be understood that the pressure borne by the vehicle seat can be detected by a weight sensor, and the preset pressure is the minimum weight of an adult. In some embodiments, the preset pressure is 40kg. In addition, in some embodiments, the preset amount is 0.

[0117] In some embodiments, the identification information includes any one of a registration account number, an identification code, and a license plate number of the vehicle.

[0118] Figure 6 It is a structural schematic diagram of a device for optimizing a gesture recognition model provided in an embodiment of the present application.

[0119] For example, Figure 6 As shown, the device 600 includes:

[0120] Acquisition module 601: used to acquire an eye image and a corresponding hand image, wherein the sight direction of the eye in the eye image is the same as the gesture direction in the hand image;

[0121] Determination module 602: used for:

[0122] Extracting features of the eye image based on the target sight line estimation model to obtain a first directional feature, where the first directional feature is used to describe the sight line direction of the eye in the eye image;

[0123] Extracting features of the hand image based on the initial gesture recognition model to obtain a second pointing feature, where the second pointing feature is used to describe the gesture pointing in the hand image;

[0124] Optimization module 603: used to optimize the initial gesture recognition model based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, where the target gesture recognition model is used to recognize the gesture direction of the vehicle user in the cockpit.

[0125] Optionally, the device 600 also includes: an alignment module, which is used to align the first pointing feature and the second pointing feature when the eye image and the corresponding hand image are captured by image capture devices at different positions in the cockpit, so as to obtain a third pointing feature and a fourth pointing feature; the optimization module 603 is specifically used to optimize the initial gesture recognition model based on the third pointing feature and the fourth pointing feature to obtain the target gesture recognition model.

[0126] Optionally, the alignment module is specifically used to: perform direction encoding on the first pointing feature and the second pointing feature to obtain the direction of each feature point in the first pointing feature and the direction of each feature point in the second pointing feature; determine a reference direction, which is the direction of the first pointing feature and the second pointing feature after feature alignment; and perform rotational transformation on the first pointing feature and the second pointing feature based on the direction of each feature point in the first pointing feature, the direction of each feature point in the second pointing feature and the reference direction to obtain the third pointing feature and the fourth pointing feature.

[0127] Optionally, the device 600 also includes: a construction module, used to construct a similarity loss function based on the similarity between the first pointing feature and the second pointing feature; the optimization module 603 is specifically used to adjust the parameters in the initial gesture recognition model based on the similarity loss function to obtain the target gesture recognition model.

[0128] Optionally, the construction module is specifically used to: perform feature fusion on the first directional feature and the second directional feature to obtain a fifth directional feature; and construct the similarity loss function based on the similarity between the second directional feature and the fifth directional feature.

[0129] Optionally, the determination module 602 is specifically used to: perform feature splicing of the first pointing feature and the second pointing feature to obtain the fifth pointing feature; or, determine a first set of weight coefficients and a second set of weight coefficients based on the contribution of the first pointing feature and the second pointing feature to determining the target gesture recognition model respectively; and perform weighted fusion of the first pointing feature and the second pointing feature based on the first set of weight coefficients and the second set of weight coefficients to obtain the fifth pointing feature.

[0130] Optionally, the device 600 also includes: a collection module, used to collect a hand image of the vehicle user when there is a vehicle user in the vehicle cabin; the determination module 602 is also used to: input the hand image of the vehicle user into the target gesture recognition model, and the target gesture recognition model recognizes the gesture direction in the hand image of the vehicle user to obtain the target direction; when the vehicle user outputs voice information, the voice information is recognized to obtain the target operation that the vehicle user expects the vehicle component corresponding to the target direction to perform; the device 600 also includes: a control module, used to control the vehicle component to perform the target operation.

[0131] Figure 7 It is a structural schematic diagram of a vehicle provided in an embodiment of the present application.

[0132] For example, Figure 7 As shown, the vehicle 700 includes: a memory 701 and a processor 702, wherein the memory 701 stores an executable program code 703, and the processor 702 is used to call and execute the executable program code 703 to perform a method for optimizing a gesture recognition model.

[0133] In addition, an embodiment of the present application also protects a device, which may include a memory and a processor, wherein the memory stores an executable program code, and the processor is used to call and execute the executable program code to execute a method for optimizing a gesture recognition model provided in an embodiment of the present application.

[0134] In this embodiment, the functional modules of the device can be divided according to the above method example. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0135] In the case of dividing each functional module according to each function, the device may also include an acquisition module, a determination module, an optimization module, an alignment module, a construction module, and a control module, etc. It should be noted that all relevant contents involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0136] It should be understood that the device provided in this embodiment is used to execute the above method for optimizing the gesture recognition model, and thus can achieve the same effect as the above implementation method.

[0137] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to a vehicle, the processing module may be used to control and manage the actions of the vehicle. The storage module may be used to support the vehicle in executing related executable program codes, etc.

[0138] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits shown in conjunction with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0139] In addition, the device provided in the embodiments of the present application may specifically be a chip, a component or a module, and the chip may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a method for optimizing a gesture recognition model provided in the above embodiment.

[0140] This embodiment also provides a computer-readable storage medium, in which executable program code is stored. When the executable program code is run on a computer, the computer executes the above-mentioned related method steps to implement a method for optimizing a gesture recognition model provided in the above embodiment.

[0141] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement a method for optimizing a gesture recognition model provided in the above embodiment.

[0142] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0143] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0144] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for optimizing a gesture recognition model, characterized in that: The method comprises: Acquire an eye image and a corresponding hand image, wherein the sight direction of the eye in the eye image is the same as the hand gesture direction in the hand image; Extracting features of the eye image based on a target sight line estimation model to obtain a first directional feature, where the first directional feature is used to describe a sight line direction of an eye in the eye image; Extracting features of the hand image based on an initial gesture recognition model to obtain a second pointing feature, where the second pointing feature is used to describe the gesture pointing in the hand image; The initial gesture recognition model is optimized based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, where the target gesture recognition model is used to recognize the gesture direction of the vehicle user in the cockpit.

2. The method according to claim 1, characterized in that The optimizing the initial gesture recognition model based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model includes: When the eye image and the corresponding hand image are collected by image acquisition devices at different positions in the cockpit, feature alignment is performed on the first pointing feature and the second pointing feature to obtain a third pointing feature and a fourth pointing feature; The initial gesture recognition model is optimized based on the third pointing feature and the fourth pointing feature to obtain the target gesture recognition model.

3. The method according to claim 2, characterized in that The step of aligning the first pointing feature and the second pointing feature to obtain a third pointing feature and a fourth pointing feature includes: Direction-encoding the first directional feature and the second directional feature to obtain the direction of each feature point in the first directional feature and the direction of each feature point in the second directional feature; Determine a reference direction, where the reference direction is a direction after the first pointing feature and the second pointing feature are aligned; Based on the directions of each feature point in the first pointing feature, the directions of each feature point in the second pointing feature and the reference direction, the first pointing feature and the second pointing feature are rotationally transformed to obtain the third pointing feature and the fourth pointing feature.

4. The method according to claim 1, characterized in that: The optimizing the initial gesture recognition model based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model includes: Based on the similarity between the first directional feature and the second directional feature, construct a similarity loss function; Based on the similarity loss function, the parameters in the initial gesture recognition model are adjusted to obtain the target gesture recognition model.

5. The method according to claim 4, characterized in that The constructing a similarity loss function based on the similarity between the first pointing feature and the second pointing feature includes: Fusing the first directional feature with the second directional feature to obtain a fifth directional feature; Based on the similarity between the second directional feature and the fifth directional feature, the similarity loss function is constructed.

6. The method according to claim 5, characterized in that The step of fusing the first directional feature with the second directional feature to obtain a fifth directional feature includes: The first directional feature and the second directional feature are combined to obtain the fifth directional feature; or, Determining a first set of weight coefficients and a second set of weight coefficients based on the contribution of the first pointing feature and the second pointing feature to determining the target gesture recognition model; Based on the first set of weight coefficients and the second set of weight coefficients, the first directional feature and the second directional feature are weightedly fused to obtain the fifth directional feature.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: When there is a vehicle user in the vehicle cabin, collecting a hand image of the vehicle user; Inputting the hand image of the vehicle user into the target gesture recognition model, and using the target gesture recognition model to recognize the gesture direction in the hand image of the vehicle user to obtain the target direction; In the case where the vehicle user outputs voice information, the voice information is recognized to obtain a target operation that the vehicle user expects the target to perform on a corresponding vehicle component; The vehicle component is controlled to perform the target operation.

8. A device for optimizing a gesture recognition model, characterized in that: The device comprises: An acquisition module, used to acquire an eye image and a corresponding hand image, wherein the sight direction of the eye in the eye image is the same as the gesture direction in the hand image; Identify modules for: Extracting features of the eye image based on a target sight line estimation model to obtain a first directional feature, where the first directional feature is used to describe a sight line direction of an eye in the eye image; Extracting features of the hand image based on an initial gesture recognition model to obtain a second pointing feature, where the second pointing feature is used to describe the gesture pointing in the hand image; The optimization module is used to optimize the initial gesture recognition model based on the first pointing feature and the second pointing feature to obtain a target gesture recognition model, wherein the target gesture recognition model is used to recognize the gesture direction of the vehicle user in the cockpit.

9. A vehicle, characterized in that: The vehicle comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the vehicle executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an executable program code, and when the executable program code is executed, the method according to any one of claims 1 to 7 is implemented.

Images