Method and system for gesture recognition
By detecting hand key points and normalizing coordinates, the problem of high computing power and data demands in existing gesture recognition technology is solved, efficient recognition on the vehicle side is achieved, and recognition accuracy and user experience are improved.
Patent Information
- Application Number
- CN202310546846.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-15
AI Technical Summary
The existing gesture recognition technology requires a large amount of computing power and training data, which is difficult to implement on low-power electronic control units (ECUs), and depends on hand size and position, affecting the recognition accuracy and deployment difficulty.
By detecting the key points of the hand (wrist and finger joint nodes), the distance between the wrist joint nodes and adjacent finger joint nodes is used to normalize the coordinates, reduce the training data needs, and achieve position-view invariance. Reduce computing power requirements and improve identification accuracy.
It realizes efficient gesture recognition on the vehicle side, reduces the computing time and training data requirements, improves the accuracy of the recognition results, and can be directly deployed on the vehicle, improving the user experience.
Smart Images

Figure CN118968540A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, and more particularly, to a method and system for gesture recognition. Background Art
[0002] With the development of advanced driver assistance and autonomous driving, cars are becoming more and more intelligent. As part of car intelligence, human-computer interaction directly affects the user's experience of intelligent driving functions. In human-computer interaction, using gestures to interact with the car computer can be more intelligent and convenient, further improving the user experience.
[0003] However, current solutions for gesture recognition using computer vision technology usually require the input of the entire image to classify the gesture, which requires a lot of computing power and can be difficult to implement for vehicles with low-power electronic control units (ECUs). In addition, when training the gesture classification model, a large amount of image data is also required as training samples for model training, which will significantly increase the training and recognition time and is difficult to deploy directly on the vehicle side.
[0004] Therefore, it is hoped that an improved gesture recognition solution can be provided so that the model requires less training data and is independent of the hand size and its position in the image, thereby significantly reducing the computing time and computing power requirements, improving the accuracy of the recognition results, and can be directly deployed on the vehicle side, thereby further improving the user's driving experience. Summary of the invention
[0005] This disclosure is provided to introduce some concepts in a simplified form that will be further described in the following detailed description. This disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0006] In response to the above problems, according to the first aspect of the present disclosure, a gesture recognition method for a vehicle is provided, the method comprising: acquiring an input image to be recognized; detecting a hand area in the input image; extracting coordinates of hand key points in the hand area, wherein the hand key points include a wrist joint and a plurality of finger joints; taking the wrist joint as the origin, using the distance between the wrist joint and adjacent finger joints to normalize the coordinates of each of the plurality of finger joints relative to the wrist joint; and sending the normalized coordinates of each of the plurality of finger joints relative to the wrist joint to a gesture classification model to recognize gestures.
[0007] In the embodiments of the present disclosure, by detecting the key points of the hand (including the wrist and finger joints) and using the distance between the wrist joints and adjacent finger joints to normalize the coordinate data of each finger key point relative to the wrist joint (as the origin), position-viewpoint invariance can be achieved, so that subsequent gesture recognition results are more accurate and require less training data, and can be directly deployed on the vehicle side with lower computing power requirements.
[0008] According to one embodiment of the present disclosure, the adjacent finger joint point is the metacarpophalangeal joint point of the little finger.
[0009] According to a further embodiment of the present disclosure, detecting the hand area in the input image further includes: feeding the input image into a palm detection model to obtain a hand bounding box including the hand area; and cropping according to the hand bounding box to obtain the hand area.
[0010] According to a further embodiment of the present disclosure, extracting the coordinates of the hand key points in the hand region further includes: predicting the coordinates of the hand key points in the hand region by regression using a key point positioning model.
[0011] According to a further embodiment of the present disclosure, the gesture classification model is trained through machine learning.
[0012] According to a further embodiment of the present disclosure, the gesture classification model is trained in the following manner: acquiring a training sample image; detecting a hand area in the training sample image; extracting the coordinates of hand key points in the hand area, wherein the hand key points include a wrist joint and multiple finger joints; taking the wrist joint as the origin, and using the distance between the wrist joint and adjacent finger joints to normalize the coordinates of each of the multiple finger joints relative to the wrist joint; sending the normalized coordinates of each of the multiple finger joints relative to the wrist joint into the gesture classification model to identify the actual gesture; and training the gesture classification model as a classifier based on the actual gesture and the target gesture corresponding to the training sample image until the model converges.
[0013] According to a further embodiment of the present disclosure, when training the gesture classification model, the target gesture corresponding to the training sample image is custom set.
[0014] According to a second aspect of the present disclosure, a gesture recognition system is provided, the system comprising: an image acquisition device, the image acquisition device being configured to acquire an input image to be recognized; a hand detection device, the hand detection device being configured to detect a hand region in the input image; a hand key point extraction device, the hand key point extraction device being configured to extract coordinates of hand key points in the hand region, wherein the hand key points include a wrist joint point and a plurality of finger joint points; a data normalization device, the data normalization device being configured to normalize the coordinates of each of the plurality of finger joint points relative to the wrist joint point using the distance between the wrist joint point and adjacent finger joint points with the wrist joint point as the origin; and a gesture recognition device, the gesture recognition device being configured to input the normalized coordinates of each of the plurality of finger joint points relative to the wrist joint point into a gesture classification model to recognize a gesture.
[0015] According to one embodiment of the present disclosure, the adjacent finger joint point is the metacarpophalangeal joint point of the little finger.
[0016] According to a further embodiment of the present disclosure, the hand detection device is further configured to: feed the input image into a palm detection model to obtain a hand bounding box including a hand area; and crop the hand area according to the hand bounding box.
[0017] According to a further embodiment of the present disclosure, the hand key point extraction device is further configured to: predict the coordinates of the hand key points in the hand area by regression using a key point positioning model.
[0018] According to a further embodiment of the present disclosure, the gesture classification model is trained in the following manner: acquiring a training sample image; detecting a hand area in the training sample image; extracting the coordinates of hand key points in the hand area, wherein the hand key points include a wrist joint and multiple finger joints; taking the wrist joint as the origin, and using the distance between the wrist joint and adjacent finger joints to normalize the coordinates of each of the multiple finger joints relative to the wrist joint; sending the normalized coordinates of each of the multiple finger joints relative to the wrist joint into the gesture classification model to identify the actual gesture; and training the gesture classification model as a classifier based on the actual gesture and the target gesture corresponding to the training sample image until the model converges.
[0019] According to a further embodiment of the present disclosure, when training the gesture classification model, the target gesture corresponding to the training sample image is custom set.
[0020] According to a third aspect of the present disclosure, a vehicle with a gesture recognition function is provided, comprising: a camera for capturing an input image to be recognized; a gesture recognition system as described in any one of claims 8 to 13; and a vehicle control system, the vehicle control system being configured to perform corresponding vehicle control operations according to the recognized gesture.
[0021] According to a fourth aspect of the present disclosure, a computer-readable storage medium storing instructions is provided, which, when executed, enables a vehicle to perform the method as described in any one of the aforementioned aspects.
[0022] These and other features and advantages will become apparent by reading the following detailed description and by reference to the associated drawings.It is to be understood that the foregoing general description and the following detailed description are illustrative only and are not restrictive of the aspects of what is claimed. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to understand in detail the manner in which the above features of the present disclosure are used, the above briefly summarized contents may be described in more detail with reference to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings only show certain typical aspects of the present disclosure and should not be considered to limit its scope, as the description may allow for other equally effective aspects.
[0024] Figure 1 It is a schematic architecture diagram of a gesture recognition system deployed on a vehicle side according to an embodiment of the present disclosure.
[0025] Figure 2 It is a schematic diagram of hand key point labeling according to an embodiment of the present disclosure.
[0026] Figure 3 It is a schematic diagram of hand gestures and corresponding hand key point annotations according to an embodiment of the present disclosure.
[0027] Figure 4 is an example flow chart of a gesture recognition method according to an embodiment of the present disclosure.
[0028] Figure 5 is an example flowchart of a method for training a gesture classification model according to an embodiment of the present disclosure.
[0029] Figure 6 is an exemplary vehicle supporting gesture recognition according to an embodiment of the present disclosure.
[0030] Figure 7 is a schematic architecture diagram of a gesture recognition system according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0031] The present disclosure is described in detail below in conjunction with the accompanying drawings, and the features of the present disclosure will be further revealed in the following specific description. The term "vehicle" used throughout the specification refers to any type of automobile, including but not limited to cars, vans, trucks, buses, etc.
[0032] Figure 1 FIG. 1 shows a schematic architecture diagram of a gesture recognition system 100 deployed on a vehicle side according to an embodiment of the present disclosure. Figure 1 As shown, the system 100 may include at least an image acquisition device 102 , a hand detection device 104 , a hand key point extraction device 106 , a data normalization device 108 and a gesture recognition device 110 .
[0033] The image acquisition device 102 can acquire an input image to be recognized.
[0034] In one implementation, the image acquisition device 102 may receive captured images (eg, each frame image in a video stream) from a visual sensor (eg, a camera) on the vehicle side for subsequent gesture recognition.
[0035] The hand detection device 104 may detect a hand region in the input image.
[0036] In one embodiment, the hand detection device 104 may feed the input image into a palm detection model to obtain a hand bounding box containing a hand region, and the hand region may be obtained by cropping the hand bounding box. The palm detection model may be trained using a single-stage target detection algorithm SSD. The palm detector is trained here instead of a hand detector because it is much simpler to detect bounding boxes of rigid objects such as palms than to detect fingers with joints. The above SSD algorithm may be optimized by the following operations: (1) using a non-maximum suppression algorithm. Since the palm is a relatively small object, the non-maximum suppression algorithm works well even for the case of self-occlusion of both hands; (2) using an encoder-decoder feature extractor for greater scene context perception; and (3) minimizing the focal loss during training to support a large number of anchor points caused by high-scale variance.
[0037] It is to be understood that any other method known to those skilled in the art may be used to train the hand or palm detection model.
[0038] The hand key point extraction device 106 can extract the coordinates of the hand key points in the hand area, wherein the hand key points include wrist joint points and multiple finger joint points.
[0039] like Figure 2 As shown, Figure 2A schematic diagram of a hand key point annotation 200 according to an embodiment of the present disclosure is shown. The hand key points can be understood as hand skeleton joints, which may include wrist joints (1) and finger joints (20). As shown in 200, the 21 joints marked from 0 to 20 are respectively wrist joints (WRIST), wrist-metacarpal joints (THUMB_CMC), thumb metacarpophalangeal joints (THUMB_MCP), thumb interphalangeal joints (THUMB_IP), thumb fingertip joints (THUMB_TIP), index finger metacarpophalangeal joints (INDEX_FINGER_MCP), index finger proximal interphalangeal joints (INDEX_FINGER_PIP), index finger distal interphalangeal joints (INDEX_FINGER_DIP), index finger fingertip joints (INDEX_FINGER_TIP), middle finger metacarpophalangeal joints (MIDDLE_FINGER_MCP), middle finger proximal interphalangeal joints (MIDDLE_FINGER_MCP), and middle finger proximal interphalangeal joints (MIDDLE_FINGER_DIP). LE_FINGER_PIP), middle finger distal interphalangeal joint (MIDDLE_FINGER_DIP), middle finger tip joint (MIDDLE_FINGER_TIP), ring finger metacarpophalangeal joint (RING_FINGER_MCP), ring finger proximal interphalangeal joint (RING_FINGER_PIP), ring finger distal interphalangeal joint (RING_FINGER_DIP), ring finger tip joint (RING_FINGER_TIP), little finger metacarpophalangeal joint (PINKY_MCP), little finger proximal interphalangeal joint (PINKY_PIP), little finger distal interphalangeal joint (PINKY_DIP), little finger tip joint (PINKY_TIP).
[0040] The above 21 hand key points can be represented by three-dimensional coordinates (x, y, z), where (x, y) are normalized to [0.0, 1.0] by the width and height of the image respectively, and z represents the depth. In this application, the size of z may not be considered (for example, z may be set to a fixed constant).
[0041] In one embodiment, the key point positioning model can be used to predict the coordinates of the above-mentioned hand key points (for example, the three-dimensional coordinates of 21 joint points) through regression. In addition, it can also be predicted whether the hand being detected is the left hand or the right hand. The above-mentioned key point positioning model can be trained using machine learning (for example, neural network, decision tree, etc.).
[0042] It is understandable that any other method known to those skilled in the art may be used to train the key point positioning model.
[0043] Then, return Figure 1The data normalization device 108 can take the wrist joint point as the origin and use the distance between the wrist joint point and the adjacent finger joint points to normalize the coordinates of each finger joint point in the multiple finger joint points relative to the wrist joint point.
[0044] In one embodiment, the adjacent finger joints may be the metacarpophalangeal joints of the little fingers.
[0045] For example, Figure 2 As shown in , the joint point 0 can be taken as the origin, and the distance between the joint point 0 and the joint point 17 can be used to normalize the coordinates of each joint point 1-20 relative to the joint point 0. The above distance can be the Euclidean distance. For example, since the size of z can be ignored, the above joint point 0 (coordinate is (x 0 ,y 0 )) and joint point 17 (coordinate (x 17 ,y 17 The distance d between )) can be calculated as Here, since the wrist joint point is used as the origin, the normalized coordinate data is 20*3 floating point numbers.
[0046] like Figure 3 As shown, Figure 3 FIG. 3 is a schematic diagram showing a hand gesture “OK” and a corresponding hand key point annotation 300 according to an embodiment of the present disclosure. In 300, a hand region in an input image may be detected, and each hand key point (such as Figure 3 The coordinates of the joint points marked in the figure) can then be normalized by using the distance between the wrist joint point and the metacarpophalangeal joint point of the little finger to normalize the coordinates of each finger joint point relative to the wrist joint point for subsequent gesture recognition.
[0047] return Figure 1 The gesture recognition device 110 may input the normalized coordinates of each finger joint point relative to the wrist joint point into the gesture classification model to recognize the gesture.
[0048] In one embodiment, the gesture classification model can be trained by machine learning (e.g., decision tree, etc.). The specific training method can be as follows: Figure 5 Described in further detail.
[0049] In one embodiment, the gesture classification model can be trained based on the custom gestures. In this way, the user can, for example, customize the gestures on the vehicle side to implement specific vehicle functions, which can further improve the user experience.
[0050] Those skilled in the art will appreciate that the gesture recognition system of the present invention may be implemented in the form of hardware or software, and the modules may be merged or combined in any appropriate manner.
[0051] Figure 4 An example flow chart of a gesture recognition method 400 according to an embodiment of the present disclosure is shown.
[0052] The method 400 starts at step 402, where the image acquisition device 102 may acquire an input image to be recognized. The input image may be a static image captured by a vehicle-mounted camera or each frame image in a video stream.
[0053] In step 404 , the hand detection device 104 may detect a hand region in the input image.
[0054] In one embodiment, the hand detection device 104 may send the input image to a hand or palm detection model to obtain a hand bounding box including the hand area, and may crop the hand area according to the hand bounding box.
[0055] In step 406, the hand key point extraction device 106 may extract the coordinates of the hand key points in the hand region, wherein the hand key points include wrist joint points and multiple finger joint points, such as Figure 2 As shown in 200 in FIG. 200 , 21 hand key points are marked.
[0056] In one embodiment, the hand key point extraction device 106 may use the key point positioning model to predict the coordinates of the hand key points in the hand region through regression.
[0057] In step 408, the data normalization device 108 can take the wrist joint point as the origin and use the distance between the wrist joint point and the adjacent finger joint points to normalize the coordinates of each finger joint point in the multiple finger joint points relative to the wrist joint point.
[0058] In other words, the Euclidean distance between the wrist joint and the adjacent finger joints can be used to normalize the distance from each finger joint to the wrist joint in each dimension (i.e., the coordinates of each finger joint relative to the wrist joint as the origin).
[0059] Preferably, the wrist joint (such as Figure 2 The joint point 0 in the middle finger and the metacarpophalangeal joint point of the little finger (such as Figure 2 The coordinates of each finger joint point relative to the wrist joint point are normalized by the distance between the joint points 17) in the figure.
[0060] In step 410 , the gesture recognition device 110 may input the normalized coordinates of each finger joint point in the plurality of finger joint points relative to the wrist joint point into a gesture classification model to recognize the gesture.
[0061] In one embodiment, the gesture classification model may be trained through machine learning, for example, it may be a decision tree model.
[0062] like Figure 3 As shown, in the extraction Figure 3 When obtaining the coordinates of the key hand points in , the distance between the wrist joint point and the metacarpophalangeal joint point of the little finger can be first used to normalize the coordinates of each finger joint point relative to the wrist joint point, and then the normalized coordinate data (for example, 60 floating-point numbers) is sent to the trained gesture classification model to recognize the gesture "OK".
[0063] Therefore, the position-viewpoint invariance can be achieved through the above-mentioned normalization operation, so that the detection of the model does not depend on the direction of the hand, the distance of the hand relative to the camera, and the position of the hand in the image. This makes the subsequent gesture recognition results more accurate and requires less training data, so it can be implemented directly on the vehicle side with lower computing power requirements.
[0064] Figure 5 An example flow chart of a method 500 for training a gesture classification model according to an embodiment of the present disclosure is shown.
[0065] The method 500 starts at step 502, where a training sample image is obtained. The training sample image may be a static image captured by a vehicle-mounted camera or each frame image in a video stream.
[0066] In step 504, a hand region is detected in the training sample image.
[0067] In step 506, the coordinates of the hand key points in the hand region are extracted, wherein the hand key points include a wrist joint point and a plurality of finger joint points.
[0068] In step 508, the coordinates of each finger joint point in the plurality of finger joint points relative to the wrist joint point are normalized using the distance between the wrist joint point and adjacent finger joint points, taking the wrist joint point as the origin.
[0069] Preferably, the adjacent finger joint may be the metacarpophalangeal joint of the little finger.
[0070] In step 510, the normalized coordinates of each finger joint point in the plurality of finger joint points relative to the wrist joint point are sent to a gesture classification model to recognize the actual gesture.
[0071] In step 512, a gesture classification model is trained as a classifier based on the actual gesture and the target gesture corresponding to the training sample image as a label until the model converges.
[0072] In one embodiment, the gesture classification model can be trained based on a decision tree, where entropy can be calculated for each value, and then the value corresponding to the maximum entropy is found to divide the results into two categories, and the cycle is repeated until the model converges or there is no more data to be segmented.
[0073] It can be understood that the above gesture classification model can be trained based on any machine learning method known to those skilled in the art.
[0074] In one embodiment, when training the gesture classification model, the target gesture corresponding to the training sample image may be a custom setting.
[0075] In one example, the predefined gesture categories may include numeric gestures "1-5", gestures "OK", and so on.
[0076] Since only 60 floating-point numbers are input for each training sample when training the model, less computing power is required to train the model and it can be implemented directly on the vehicle side, which significantly reduces the model training time and improves the accuracy of the recognition results, thereby further improving the user experience.
[0077] Figure 6 An exemplary vehicle 600 supporting gesture recognition is shown according to one embodiment of the present disclosure.
[0078] Vehicle 600 may include camera 602 , which may be configured to capture an input image to be recognized and transmit the input image to gesture recognition system 100 for subsequent gesture recognition operations.
[0079] In addition, the vehicle 600 may include Figure 1 The gesture recognition system 100 shown (which can be deployed on a vehicle-mounted computer, for example) can be configured to: obtain an input image to be recognized; detect a hand area in the input image; extract the coordinates of hand key points in the hand area, wherein the hand key points include a wrist joint point and multiple finger joint points; normalize the coordinates of each finger joint point in the multiple finger joint points relative to the wrist joint point using the distance between the wrist joint point and adjacent finger joint points with the wrist joint point as the origin; and send the normalized coordinates of each finger joint point in the multiple finger joint points relative to the wrist joint point to a gesture classification model to recognize gestures.
[0080] In addition, the vehicle 600 may further include a vehicle control system 604, which may be configured to receive a gesture recognition result and perform a corresponding vehicle control operation according to the recognized gesture.
[0081] In an example scenario, when a user uses a car navigation system for navigation, after entering a destination, the navigation system plans a navigation route and asks whether to navigate along the navigation route. The user can make an "OK" gesture (such as Figure 3 As shown), the camera 602 can capture an image containing the gesture and send it to the system 100 to recognize the gesture "OK". Subsequently, the vehicle control system 604 can confirm navigation according to the navigation route based on the gesture.
[0082] Figure 7 Schematic diagram of the gesture recognition system 700 according to an embodiment of the present disclosure is shown. The system 700 may be configured to perform various methods described herein (including, for example, Figure 4 or aspects of the method described in 5).
[0083] like Figure 7 As shown, the system 700 may include one or more processors 702. The one or more processors 702 may include a central processing unit (CPU), which may be a multi-core CPU in some examples. The instructions executed at the CPU may be loaded, for example, from a program memory associated with the CPU or may be loaded from a memory 704. The one or more processors 702 may also include additional processing components customized for specific functions, such as a graphics processing unit (GPU), a digital signal processor (DSP), and a neural processing unit (NPU). In some examples, the one or more processors 702 may be based on an ARM or RISC-V instruction set.
[0084] The system 700 also includes a memory 704. The memory 704 may include RAM, ROM, or a combination thereof. The memory 704 may store computer executable instructions that, when executed by at least one processor 702, cause the at least one processor to perform various functions described herein, including: obtaining an input image to be recognized; detecting a hand region in the input image; extracting the coordinates of a hand key point in the hand region, wherein the hand key point includes a wrist joint point and a plurality of finger joint points; taking the wrist joint point as the origin, using the distance between the wrist joint point and the adjacent finger joint points to normalize the coordinates of each finger joint point in the plurality of finger joint points relative to the wrist joint point; and sending the normalized coordinates of each finger joint point in the plurality of finger joint points relative to the wrist joint point to a gesture classification model to recognize a gesture. In some cases, the memory 704 may include a BIOS in particular, which may control basic hardware or software operations, such as interaction with peripheral components or devices.
[0085] Understandably, Figure 7 This is merely one example of a system, and other systems including fewer, additional, or alternative aspects may also be consistent with the present disclosure.
[0086] The various illustrative blocks and modules described in conjunction with the disclosure herein may be implemented or executed with a general purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration).
[0087] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Other examples and implementations fall within the scope of the present disclosure and the appended claims. For example, due to the nature of software, the functions described herein may be implemented using software executed by a processor, hardware, firmware, hardwiring, or any combination thereof. Features that implement the functions may also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations.
[0088] What has been described above includes examples of various aspects of the claimed subject matter. Of course, it is not possible to describe every conceivable combination of components or methods for the purpose of describing the claimed subject matter, but one of ordinary skill in the art will recognize that many further combinations and permutations of the claimed subject matter are possible. Thus, the disclosed subject matter is intended to encompass all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.
Claims
1. A gesture recognition method for a vehicle, the method comprising: Obtain an input image to be recognized; Detecting a hand region in the input image; Extracting coordinates of key hand points in the hand region, wherein the key hand points include wrist joint points and multiple finger joint points; Taking the wrist joint point as the origin, and using the distance between the wrist joint point and an adjacent finger joint point to normalize the coordinates of each finger joint point in the plurality of finger joint points relative to the wrist joint point; as well as The normalized coordinates of each finger joint point in the plurality of finger joint points relative to the wrist joint point are sent to a gesture classification model to recognize the gesture.
2. The method according to claim 1, characterized in that The adjacent finger joint point is the metacarpophalangeal joint point of the little finger.
3. The method according to claim 1, characterized in that Detecting a hand region in the input image further comprises: Sending the input image to a palm detection model to obtain a hand bounding box containing a hand area; and The hand region is obtained by cropping according to the hand bounding box.
4. The method according to claim 1, characterized in that Extracting the coordinates of the hand key points in the hand area further includes: The key point localization model is used to predict the coordinates of the hand key points in the hand region through regression.
5. The method according to claim 1, characterized in that The gesture classification model is trained through machine learning.
6. The method according to claim 5, characterized in that The gesture classification model is trained in the following way: Obtain training sample images; Detecting a hand region in the training sample image; Extracting coordinates of key hand points in the hand region, wherein the key hand points include wrist joint points and multiple finger joint points; Taking the wrist joint point as the origin, and using the distance between the wrist joint point and an adjacent finger joint point to normalize the coordinates of each finger joint point in the plurality of finger joint points relative to the wrist joint point; Sending the normalized coordinates of each finger joint point of the plurality of finger joint points relative to the wrist joint point into the gesture classification model to recognize the actual gesture; as well as Based on the actual gesture and the target gesture corresponding to the training sample image, the gesture classification model as a classifier is trained until the model converges.
7. The method according to claim 6, characterized in that When training the gesture classification model, the target gesture corresponding to the training sample image is custom set.
8. A gesture recognition system for a vehicle, the system comprising: An image acquisition device, the image acquisition device being configured to acquire an input image to be recognized; a hand detection device configured to detect a hand region in the input image; A hand key point extraction device, the hand key point extraction device is configured to extract the coordinates of the hand key points in the hand area, wherein the hand key points include wrist joint points and multiple finger joint points; a data normalization device, the data normalization device being configured to normalize the coordinates of each finger joint point of the plurality of finger joint points relative to the wrist joint point using the distance between the wrist joint point and adjacent finger joint points, with the wrist joint point as the origin; as well as A gesture recognition device is configured to input the normalized coordinates of each finger joint point of the plurality of finger joint points relative to the wrist joint point into a gesture classification model to recognize a gesture.
9. The system according to claim 8, characterized in that The adjacent finger joint point is the metacarpophalangeal joint point of the little finger.
10. The system according to claim 8, characterized in that The hand detection device is further configured to: Sending the input image to a palm detection model to obtain a hand bounding box containing a hand area; and The hand region is obtained by cropping according to the hand bounding box.
11. The system according to claim 8, characterized in that The hand key point extraction device is further configured to: The key point localization model is used to predict the coordinates of the hand key points in the hand region through regression.
12. The system of claim 8, wherein: The gesture classification model is trained in the following way: Obtain training sample images; Detecting a hand region in the training sample image; Extracting coordinates of key hand points in the hand region, wherein the key hand points include wrist joint points and multiple finger joint points; Taking the wrist joint point as the origin, and using the distance between the wrist joint point and an adjacent finger joint point to normalize the coordinates of each finger joint point in the plurality of finger joint points relative to the wrist joint point; Sending the normalized coordinates of each finger joint point of the plurality of finger joint points relative to the wrist joint point into the gesture classification model to recognize the actual gesture; as well as Based on the actual gesture and the target gesture corresponding to the training sample image, the gesture classification model as a classifier is trained until the model converges.
13. The system of claim 12, wherein: When training the gesture classification model, the target gesture corresponding to the training sample image is custom set.
14. A vehicle with a gesture recognition function, comprising: A camera, the camera being used to capture an input image to be recognized; The gesture recognition system as claimed in any one of claims 8 to 13; as well as A vehicle control system is configured to perform corresponding vehicle control operations according to the recognized gesture.
15. A computer-readable storage medium storing instructions, which, when executed, cause a vehicle to perform the method according to any one of claims 1 to 7.
Citation Information
Cited By
Method for detecting hand joints in glasses equipment, storage medium, electronic equipment and product
CN119672766A
Method and device for learning and detecting hand gestures in a vehicle
WO2026161908A1
