Gesture dataset construction method and gesture dataset construction apparatus

Abstract

The application relates to a gesture dataset construction method and a gesture dataset construction device, wherein the gesture dataset construction method comprises the following steps: acquiring a gesture image sequence; performing gesture region detection on the gesture image sequence to obtain a gesture region sequence; performing three-dimensional gesture key point detection on the gesture image sequence to obtain a three-dimensional gesture key point sequence; performing feature fusion on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information; performing left-right hand determination on the fusion information to obtain a determination result; and constructing a three-dimensional gesture dataset based on the determination result and the fusion information. The application can reduce the dependence on hardware of a sensor device in gesture recognition from the algorithm and software levels, improve the annotation accuracy, reduce the dependence on manual annotation, and finally improve the efficiency of three-dimensional gesture annotation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a gesture dataset construction method and a gesture dataset construction apparatus. BACKGROUND

[0002] In the fields of virtual reality (VR), augmented reality (AR) and human-computer interaction (HCI), three-dimensional gesture recognition technology is crucial. Constructing a three-dimensional gesture dataset with a quality meeting the scene recognition needs to support the training and optimization of a model, so that the model can play a better performance in gesture recognition and obtain a robust, general and accurate recognition result. Therefore, the construction of a three-dimensional gesture dataset is an important pre-step of three-dimensional gesture recognition.

[0003] At present, in a part of related technologies, a multimedia processing framework is used alone to realize gesture detection. This kind of scheme has a high misjudgment rate in the scene of crossing hands. For example, when crossing hands, the misjudgment rate of left and right hands can reach 13.8%, which affects the interactive experience. In addition, the visual scheme is easy to be disturbed by light and background, and it is difficult to provide high-precision three-dimensional spatial coordinates. In another part of related technologies, gesture recognition is performed based on a special sensor device (such as an inertial measurement unit, a data glove or a depth camera). This kind of technology needs to wear additional hardware, which is not suitable for daily natural interaction, so the use scene is limited, in addition, because the sampling rate of part of the sensor devices is low, it is difficult to capture relatively fast gesture actions; the device price of the sensor device is expensive, and it is also difficult to popularize on a large scale. Therefore, in related technologies, for the labeling of three-dimensional gesture data, there is still a problem of low labeling efficiency due to the dependence on hardware devices and low labeling accuracy and the dependence on manual labeling, which ultimately leads to low labeling efficiency.

[0004] At present, there is no effective solution to the problem of low three-dimensional gesture labeling efficiency in related technologies. SUMMARY

[0005] A gesture dataset construction method and a gesture dataset construction apparatus are provided in the embodiment to solve the problem of low three-dimensional gesture labeling efficiency in related technologies.

[0006] In a first aspect, a gesture dataset construction method is provided in the embodiment, comprising:

[0007] obtaining a gesture image sequence;

[0008] performing gesture region detection on the gesture image sequence to obtain a gesture region sequence, and performing three-dimensional gesture key point detection on the gesture image sequence to obtain a three-dimensional gesture key point sequence;

[0009] performing feature fusion on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information;

[0010] performing left-right hand determination on the fusion information to obtain a determination result;

[0011] constructing a three-dimensional gesture dataset based on the determination result and the fusion information.

[0012] In some embodiments, a gesture image sequence is obtained, including:

[0013] obtaining a gesture image sequence captured by a monocular camera.

[0014] In some embodiments, the gesture region sequence and the three-dimensional gesture key point sequence are fused to obtain fusion information, including:

[0015] associating the gesture region sequence and the three-dimensional gesture key point sequence according to position information of each gesture region in the gesture region sequence and key point position statistical information of each group of gesture key points in the three-dimensional gesture key point sequence;

[0016] aligning the gesture region sequence and the three-dimensional gesture key point sequence in a coordinate system after association to obtain fusion information.

[0017] In some embodiments, the gesture region sequence and the three-dimensional gesture key point sequence are associated according to position information of each gesture region in the gesture region sequence and key point position statistical information of each group of gesture key points in the three-dimensional gesture key point sequence, including:

[0018] determining distance information between the position information of each gesture region in the gesture region sequence and the key point position statistical information of each group of gesture key points in the three-dimensional gesture key point sequence, respectively;

[0019] associating each gesture region and each group of gesture key points based on the Hungarian algorithm according to the distance information.

[0020] In some embodiments, the fusion information is subjected to left-right hand determination to obtain a determination result, including:

[0021] determining a biomechanical feature vector according to a metacarpal key point and a wrist key point in the fusion information;

[0022] inputting the biomechanical feature vector into a pre-constructed decision tree model for left-right hand determination to obtain the determination result.

[0023] In some embodiments, a three-dimensional gesture dataset is constructed based on the determination result and the fusion information, including:

[0024] generate, based on the determination result and the fusion information, label data corresponding to each gesture image in the gesture image sequence;

[0025] construct the three-dimensional gesture dataset according to the gesture image sequence and the label data corresponding to each gesture image in the gesture image sequence.

[0026] In some embodiments, the constructing the three-dimensional gesture dataset according to the gesture image sequence and the label data corresponding to each gesture image in the gesture image sequence comprises:

[0027] correct the label data to obtain corrected label data, and obtain a virtual gesture dataset generated by an open-source robot system simulation platform for a preset three-dimensional hand model; the virtual gesture dataset comprises a dynamic gesture sequence and true value label data corresponding to the dynamic gesture sequence;

[0028] fuse the initial dataset composed of the gesture image sequence and the corrected label data with the virtual gesture dataset to obtain the three-dimensional gesture dataset.

[0029] In some embodiments, the gesture region detection on the gesture image sequence to obtain a gesture region sequence comprises:

[0030] detect the gesture region of the gesture image sequence based on a preset target detection model to obtain the gesture region sequence; the target detection model is a model formed by replacing a backbone network in a YOLOv5 model with a lightweight convolutional neural network and adding a channel attention mechanism module.

[0031] In a second aspect, a gesture detection method is provided in the present embodiment, comprising:

[0032] obtain a pre-constructed three-dimensional gesture dataset; the three-dimensional gesture dataset is generated based on the gesture dataset construction method of the first aspect;

[0033] train a gesture detection model based on the three-dimensional gesture dataset;

[0034] detect a gesture in a to-be-detected image according to the gesture detection model to obtain a gesture detection result.

[0035] In a third aspect, a gesture dataset construction apparatus is provided in the present embodiment, comprising an obtaining module, a detecting module, a fusion module, a determination module and a construction module; wherein:

[0036] The obtaining module is configured to obtain a gesture image sequence.

[0037] The detection module is configured to perform gesture region detection on the gesture image sequence to obtain a gesture region sequence, and perform three-dimensional gesture key point detection on the gesture image sequence to obtain a three-dimensional gesture key point sequence.

[0038] The fusion module is configured to perform feature fusion on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information.

[0039] The determination module is configured to perform left-right hand determination on the fusion information to obtain a determination result.

[0040] The construction module is configured to construct a three-dimensional gesture dataset based on the determination result and the fusion information.

[0041] Compared with the related art, the gesture dataset construction method and the gesture dataset construction apparatus are provided in the embodiment. The gesture dataset construction method comprises the following steps: obtaining a gesture image sequence; performing gesture region detection on the gesture image sequence to obtain a gesture region sequence; performing three-dimensional gesture key point detection on the gesture image sequence to obtain a three-dimensional gesture key point sequence; performing feature fusion on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information; performing left-right hand determination on the fusion information to obtain a determination result; and constructing a three-dimensional gesture dataset based on the determination result and the fusion information. The gesture dataset construction method combines gesture region detection and three-dimensional gesture key point detection to obtain fusion information, and performs accurate left-right hand determination based on the fusion information, which can reduce the dependence on hardware of a sensor device in gesture recognition from the algorithm and software levels, improve the accuracy of labeling, reduce the dependence on manual labeling, and finally improve the efficiency of three-dimensional gesture labeling.

[0042] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings illustrated herein are used to provide further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 is an application environment diagram of a gesture dataset construction method according to an embodiment of the present application;

[0045] Figure 2 is a flowchart of a gesture dataset construction method according to an embodiment of the present application;

[0046] Figure 3 is a three-dimensional gesture key point diagram according to an embodiment of the present application;

[0047] Figure 4 is a flow chart of a gesture dataset construction method of some embodiments of the present application;

[0048] Figure 5 is a timing diagram of a gesture dataset construction of some embodiments of the present application;

[0049] Figure 6 is a flow chart of a gesture detection method of an embodiment of the present application;

[0050] Figure 7 is a structural block diagram of a gesture dataset construction apparatus of an embodiment of the present application;

[0051] Figure 8 is an internal structure diagram of a computer device of an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and explained below in conjunction with the drawings and embodiments.

[0053] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the general meaning understood by a person with ordinary skill in the art to which the present application belongs. In the present application, "one", "a", "an", "the", "these" and similar words do not represent a quantitative limitation, and they can be singular or plural. In the present application, the terms "include", "contain", "have" and any variants thereof have the purpose of covering non-exclusive inclusion; for example, a process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connected", "connected", "coupled" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. In the present application, "multiple" means two or more. The association between the associated objects is described by "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. In general, the character " / " represents an "or" relationship between the objects before and after it. In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0054] Figure 1 is an application environment diagram of a gesture dataset construction method of an embodiment of the present application, the gesture dataset construction method provided by the present application can be applied to, for example, Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 obtains a gesture image sequence; gesture region detection is performed on the gesture image sequence to obtain a gesture region sequence; and three-dimensional gesture key point detection is performed on the gesture image sequence to obtain a three-dimensional gesture key point sequence; the gesture region sequence and the three-dimensional gesture key point sequence are fused to obtain fusion information; the left and right hands are determined based on the fusion information to obtain a determination result; and based on the determination result and the fusion information, a three-dimensional gesture dataset is constructed. The constructed three-dimensional gesture dataset is sent to the terminal 102 through the communication network. Among them, the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0055] In this embodiment, a gesture dataset construction method is provided, Figure 2 The flowchart of the gesture dataset construction method of this embodiment is shown in Figure 2 The flowchart includes the following steps:

[0056] Step S210, obtaining a gesture image sequence.

[0057] The gesture image sequence can be a sequence containing several frames of RGB color images, or a sequence of other two-dimensional images suitable for gesture recognition. The gesture image sequence can be obtained by a camera, such as a monocular camera. In this step, the position and posture of the camera when obtaining the gesture image are not limited. For example, a standard computer camera is placed in front of the desktop to capture gesture images; or a camera installed on the top of the computer display or at the position of the rearview mirror of the vehicle is used to capture gesture images from an oblique overhead view of the user's hand operation area; or in some industrial or interactive scenarios, a camera is used to capture gesture images from a side view. The gesture image sequence can also include gesture images from multiple perspectives.

[0058] After obtaining the gesture image sequence, in order to enhance the robustness of gesture recognition to environmental noise, data enhancement can be performed to add data noise to the gesture image sequence. In addition, data enhancement can also be performed based on a generative adversarial network.

[0059] In step S220, gesture region detection is performed on the gesture image sequence to obtain a gesture region sequence, and three-dimensional gesture key point detection is performed on the gesture image sequence to obtain a three-dimensional gesture key point sequence.

[0060] The gesture image sequence can be input into a preset target detection model, and gesture region detection is performed on the gesture image sequence to obtain gesture regions corresponding to each frame of the gesture image sequence, thereby forming the gesture region sequence. For example, in some embodiments, the detection result of the gesture region can include a bounding box of the hand region, a confidence score indicating whether there is a gesture in the bounding box, a specific category of the gesture (left hand or right hand), and a confidence score of the gesture category. The center point coordinates (x, y) and the width and height (w, h) of the bounding box of the hand region in the gesture image are used to accurately determine the position of the hand in the image. The non-maximum suppression algorithm can be used to filter out redundant detection boxes to obtain an accurate gesture bounding box and gesture category. In addition, the palm orientation can also be output. In particular, the target detection model can be based on a YOLOv5 model. The gesture image sequence captured by the camera is input into the trained YOLOv5 model, and the model outputs the detection result of one or more gesture regions through forward propagation. In addition, other models suitable for target detection, such as the YOLOv11n model, can also be used for gesture region detection.

[0061] In addition, the gesture image sequence can also be input into a preset key point detection model to extract three-dimensional gesture key points. The three-dimensional gesture key points corresponding to each frame of the gesture image are combined to form a three-dimensional gesture key point sequence. In particular, the key point detection model can be a multimedia processing framework (MediaPipe Hands framework). Each frame of the gesture image sequence is input into the MediaPipe Hands framework. In the framework, a lightweight palm detector is first set to locate the approximate region of the hand in the entire gesture image. Then, the located region is cropped and sent to the hand key point model (Hand Landmark Model) in the framework. The hand key point model regresses the three-dimensional coordinates (x, y, z) of 21 hand key points, where "x" and "y" are the normalized width and height coordinates of the gesture image, and "z" is the depth coordinate representing the relative distance of the key point to the camera. The smaller the "z" value, the closer the key point is to the camera. Figure 3 A three-dimensional gesture key point diagram for this embodiment is shown in FIG. 2B. Figure 3As shown, the extracted 21 hand key points are key points following the anatomical structure of the human hand, which can specifically include the following key points: 1 key point at the wrist (WRIST), numbered 0; 4 key points at the thumb (THUMB), from the base to the tip, numbered 1 to 4; 4 key points at the index finger (INDEX_FINGER), from the base to the tip, numbered 5 to 8; 4 key points at the middle finger (MIDDLE_FINGER), from the base to the tip, numbered 9 to 12; 4 key points at the ring finger, from the base to the tip, numbered 13 to 16; and 4 key points at the little finger, from the base to the tip, numbered 17 to 20. In addition, three-dimensional key point detection can also be realized based on the hand key point detection function of the open source human pose estimation library, or other trained key point detection models. Preferably, it can be realized based on MediaPipe.

[0062] After obtaining the three-dimensional gesture key points corresponding to each frame of gesture image respectively, the three-dimensional gesture key points corresponding to each frame of gesture image form a group, and then all groups of three-dimensional gesture key points form a three-dimensional gesture key point sequence.

[0063] In step S230, the gesture region sequence and the three-dimensional gesture key point sequence are fused to obtain fusion information.

[0064] Each gesture region in the gesture region sequence is two-dimensional information, and the three-dimensional gesture key point sequence is three-dimensional information. In this step, the gesture region information corresponding to a certain frame of gesture image in the gesture region sequence is fused with the three-dimensional gesture key point corresponding to the same frame of gesture image in the three-dimensional gesture key point sequence. For example, the gesture image sequence includes gesture image A, gesture image B and gesture image C, the gesture region of gesture image A is obtained by detecting the gesture region of gesture image A, and the three-dimensional gesture key point of gesture image A is obtained by detecting the three-dimensional gesture key point of gesture image A. The gesture region and the three-dimensional gesture key point of gesture image A are fused to obtain fusion information. Similarly, the gesture region and the three-dimensional gesture key point of gesture image B are obtained, the gesture region and the three-dimensional gesture key point of gesture image B are fused to obtain fusion information, and the gesture region and the three-dimensional gesture key point of gesture image C are obtained, the gesture region and the three-dimensional gesture key point of gesture image C are fused to obtain fusion information.

[0065] When the gesture region and the three-dimensional gesture key point are fused, the two features can be associated and paired based on the positions of the two features in the gesture image and the coordinate information carried by the two features respectively, and the unified coordinate system of the paired features is completed, so as to complete the feature fusion and obtain the fusion information. The fusion information includes the three-dimensional gesture key point, the gesture region information and the like in the unified coordinate system.

[0066] It can be understood that when multiple hands appear in a gesture image, multiple gesture regions will appear in the gesture image when gesture region detection is performed. In addition, when three-dimensional gesture key point detection is performed on the gesture image, multiple sets of three-dimensional gesture key points will also be detected. For example, for a first hand in the gesture image, there is a set of 21 three-dimensional gesture key points corresponding thereto, and for a second hand in the gesture image, there is also a set of 21 three-dimensional gesture key points corresponding thereto, and so on. Thus, when feature fusion is performed, feature fusion of the gesture region and the three-dimensional gesture key points of the same hand needs to be implemented.

[0067] In step S240, left-right hand determination is performed on the fusion information, and a determination result is obtained.

[0068] A set of rules for left-right hand determination can be constructed for the features contained in the fusion information. Based on the rules, the features in the fusion information are analyzed, and in the case where some conditions in the rules are met, it is determined that a hand in the gesture image is a left hand, and in the case where other conditions in the rules are met, it is determined that a hand in the gesture image is a right hand. Exemplarily, the left-right hand determination can be implemented by constructing a decision tree model, and taking the above rules as the determination logic of the decision tree model. In the left-right hand determination process, left-right hand determination is performed on each hand target in each frame of the gesture image sequence, and a corresponding determination result is obtained.

[0069] In step S250, a three-dimensional gesture dataset is constructed based on the determination result and the fusion information.

[0070] The determination result of the left-right hand and the detection result of the gesture region and the key point in the fusion information are jointly used as the labeling result of each gesture image in the gesture image sequence, and a three-dimensional gesture dataset is constructed according to the labeling result and the corresponding gesture image. In this way, the labeling of the gesture image sequence can be automatically implemented.

[0071] In related technologies, gesture detection is implemented by using a multimedia processing framework. Such a scheme has a high misjudgment rate in scenarios such as hand crossing. For example, when the hands cross, the left-right hand misjudgment rate can reach 13.8%, which affects the interactive experience. In addition, the visual scheme is susceptible to light and background interference, and it is difficult to provide high-precision three-dimensional spatial coordinates. In another part of the related technologies, gesture recognition is performed based on a special sensor device (such as an inertial measurement unit, a data glove, or a depth camera). Such a technology needs to wear additional hardware, which is not suitable for daily natural interaction, and thus the use scenario is limited. In addition, because the sampling rate of part of the sensor devices is low, it is difficult to capture relatively fast gesture actions; the device price of the sensor device is also high, and it is difficult to popularize on a large scale.

[0072] In this embodiment, through the above steps S210 to S250, three-dimensional gesture key point detection is performed by using a deep learning algorithm to estimate three-dimensional information from a two-dimensional image. The features of the three-dimensional gesture key points are fused with the features of the gesture region, and then left and right hand determination is performed based on the fused information, thereby further enhancing the accuracy and robustness of the annotation in the algorithm and software level in the annotation scene involving left and right hand distinction, and reducing the misjudgment rate of left and right hand. The difficulty of obtaining three-dimensional gesture recognition is successfully transformed from the hardware design and manufacturing level to the algorithm design and model training level, thereby breaking the dependence on special hardware. Moreover, the annotation result is automatically generated, which improves the accuracy of three-dimensional gesture annotation, reduces the dependence on manual annotation, and thereby improves the data annotation efficiency.

[0073] Therefore, through the above steps S210 to S250, a gesture image sequence is obtained; gesture region detection is performed on the gesture image sequence to obtain a gesture region sequence; three-dimensional gesture key point detection is performed on the gesture image sequence to obtain a three-dimensional gesture key point sequence; feature fusion is performed on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fused information; left and right hand determination is performed on the fused information to obtain a determination result; and a three-dimensional gesture dataset is constructed based on the determination result and the fused information. By combining gesture region detection and three-dimensional gesture key point detection, the fused information is obtained, and accurate left and right hand determination is completed based on the fused information, which can reduce the dependence on hardware of sensor devices in gesture recognition from the algorithm and software level, improve the annotation accuracy, thereby reduce the dependence on manual annotation, and finally improve the efficiency of three-dimensional gesture annotation.

[0074] In one embodiment, based on the above step S210, the gesture image sequence is obtained, which can specifically include: obtaining a gesture image sequence collected by a monocular camera.

[0075] In this embodiment, a monocular camera is selected as the hardware device for data collection, and the above gesture region detection and three-dimensional gesture key point detection are combined, thereby enabling accurate annotation of gesture images from the algorithm level. In some related technologies, a binocular camera is often used to reconstruct a three-dimensional gesture, and a stereo matching algorithm is used to directly calculate the depth by using the parallax of two cameras in the binocular camera to reconstruct a three-dimensional point cloud. This requires the deployment of two cameras, which has a high hardware cost, and requires strict calibration and maintenance of the two cameras, thereby consuming more computing resources.

[0076] To this end, the embodiment collects a gesture image sequence based on a monocular camera, and then performs subsequent gesture region detection and three-dimensional gesture key point detection based on the gesture image sequence collected by the monocular camera, performs feature fusion, and then performs left and right hand determination, and finally constructs a three-dimensional gesture dataset. The embodiment realizes accurate three-dimensional gesture data construction based on a monocular camera and an algorithm. Therefore, the embodiment can eliminate the dependence on a binocular camera in the related art, reduce the hardware cost, simplify the structure of the entire gesture dataset construction system, and avoid the complex calibration and maintenance required by the multi-camera solution.

[0077] In one embodiment, based on the above step S230, the gesture region sequence and the three-dimensional gesture key point sequence are fused to obtain fusion information, which can specifically include:

[0078] According to the position information of each gesture region in the gesture region sequence and the key point position statistical information of each group of gesture key points in the three-dimensional gesture key point sequence, the gesture region sequence and the three-dimensional gesture key point sequence are associated; and the associated gesture region sequence and three-dimensional gesture key point sequence are aligned in the coordinate system to obtain the fusion information.

[0079] The key point position statistical information can be obtained by statistically processing the three-dimensional coordinates of each key point in the same group of three-dimensional gesture key points. The gesture region corresponding to the same hand in the same frame of gesture image and the three-dimensional gesture key point can be data-associated according to the position information of the gesture region in the gesture image and the key point position statistical information in the frame of gesture image.

[0080] In particular, in some embodiments, the average value of the three-dimensional coordinates of each group (for example, 21 groups) of three-dimensional gesture key points can be calculated to obtain the centroid of the group of three-dimensional gesture key points as the key point position statistical information of the group of three-dimensional gesture key points. The center coordinates of the gesture bounding box in the gesture region are calculated as the position information of the gesture region. Based on an optimization algorithm for data matching, such as the Hungarian algorithm, the above center coordinates and the above centroid are matched to realize the above data association. In this way, the data association of the two features can be realized in a simple and fast calculation manner, while avoiding high computational overhead and time complexity (only one average and one distance calculation are required). For the non-rigid but aggregated target such as the hand, the key point centroid is a relatively stable and representative position statistical feature, which is sufficient to realize the accurate association of the gesture region and the key point.

[0081] Additionally, an Intersection over Union (IOU) method can also be used. According to the (x, y) coordinates of each key point in each set of three-dimensional gesture key points, the smallest bounding box that can wrap all these two-dimensional coordinates is calculated. Then, the IOU value of the smallest bounding box and the bounding box in the gesture region is calculated. Finally, based on an optimization algorithm for solving the data matching problem, such as the Hungarian algorithm, the optimal matching pair is found to maximize the total IOU, thereby realizing the association of the gesture region and the key points.

[0082] In addition, based on the point-in-polygon test method, each set of three-dimensional gesture key points can be traversed to count how many key points fall inside the bounding box of a certain gesture region, obtaining the score between each gesture region and each set of three-dimensional gesture key points (for example, the number of key points falling inside the bounding box is the score). Finally, the matching scheme that maximizes the total score is selected, and thus the data association is completed.

[0083] After completing the data association, the gesture region and the three-dimensional gesture key points can be unified in the coordinate system. For example, the gesture bounding box output by YOLOv5 is a two-dimensional image pixel coordinate, while the three-dimensional gesture key points output by MediaPipe are normalized image coordinates (x, y) and relative depth (z). Therefore, in the alignment process, the (x, y) coordinates of MediaPipe can be multiplied by the actual width and height of the gesture image to convert into a pixel coordinate system consistent with YOLOv5. At the same time, the gesture class label detected by YOLOv5 is assigned to the set of three-dimensional gesture key points output by MediaPipe that successfully match, thereby completing the fusion of information.

[0084] In this embodiment, by accurately matching and fusing the gesture region information and the three-dimensional gesture key point information, more rich multi-dimensional features can be provided for left and right hand judgment, improving the accuracy of left and right hand judgment and reducing the misjudgment rate. Moreover, this embodiment can also solve the coordinate system difference of two-dimensional features (the image coordinate system corresponding to the gesture region and the world coordinate system corresponding to the three-dimensional gesture key points), thereby solving the feature matching ambiguity problem during two-hand interaction.

[0085] In one embodiment, according to the position information of each gesture region in the gesture region sequence and the key point position statistical information of each set of gesture key points in the three-dimensional gesture key point sequence, the gesture region sequence and the three-dimensional gesture key point sequence are associated, which can specifically include:

[0086] Determine distance information between position information of each gesture region in the gesture region sequence and key point position statistical information of each group of gesture key points in the three-dimensional gesture key point sequence; and associate each gesture region with each group of gesture key points based on the Hungarian algorithm according to the distance information.

[0087] When data association is performed on the distance scale based on the position information of the two features in the image, the Hungarian algorithm can be introduced. Specifically, when multiple hand targets are detected in the same frame of gesture image, the detection result corresponding to multiple gesture regions will be output, and thus multiple gesture bounding boxes are included. Similarly, multiple groups of three-dimensional gesture key points will also be generated. In this embodiment, in order to achieve correct pairing between gesture regions and three-dimensional gesture key points belonging to the same hand, the centroid (such as the average value of all 21 point coordinates) of each group of three-dimensional gesture key points can be calculated, and the Euclidean distance between the centroid and the center of the gesture bounding box in the gesture region can be calculated. Based on the Euclidean distance, multiple pairs of features are combined to form a distance matrix, and then the Hungarian algorithm is used to minimize the total distance as the goal to achieve optimal matching between gesture regions and three-dimensional gesture key points, so as to achieve accurate and stable tracking of the gesture target.

[0088] In this embodiment, the Hungarian algorithm is introduced to realize data matching and association, which can further improve the accuracy of data association and matching and reduce the computational overhead.

[0089] In addition, in one embodiment, based on the above step S240, the left-right hand determination is performed on the fusion information, and the determination result can include:

[0090] A biomechanical feature vector is determined according to the metacarpophalangeal joint key point and the wrist key point in the fusion information; and the biomechanical feature vector is input into a pre-constructed decision tree model to perform left-right hand determination, and a determination result is obtained.

[0091] In this embodiment, the metacarpophalangeal joint key point and the wrist key point are selected to construct the biomechanical feature vector from the perspective of anatomy and geometry. Specifically, the wrist key point is defined as the origin, and vector A (V_A) is defined as pointing from the wrist (point 0 in the coordinate system) Figure 3 to the index finger metacarpophalangeal joint (point 5 in the coordinate system), that is, V_A=P5-P0; and vector B (V_B) is defined as pointing from the wrist (point 0 in the coordinate system) Figure 3 to the little finger metacarpophalangeal joint (point 4 in the coordinate system), that is, V_B=P4-P0. Figure 3 Figure 3 ​V_B = P17 - P0. The cross product of V_A and V_B is calculated to obtain a normal vector V_Normal = V_A x V_B perpendicular to the palm plane. According to the right-hand rule, the normal vector has a stable distinction: for the right hand, when the palm is facing the observer, V_Normal usually points to the observer or points out of the screen; for the left hand, when the palm is facing the observer, V_Normal usually points away from the observer or points into the screen. Therefore, V_Normal can be used as a biomechanical feature vector to input into a pre-constructed decision tree model for right-hand or left-hand determination.

[0092] In the embodiment, the wrist is considered as the root of the whole hand and the kinematic base point of all movements, so the wrist is taken as the most ideal and stable coordinate system origin. In addition, the index finger knuckle and the little finger knuckle are considered as the ends of the metacarpal bone structure (metacarpal bone). Compared with the fingertip joint which can be freely bent and has a large range of motion, the positions of the two knuckle joints are relatively fixed relative to the wrist and palm. Regardless of whether the user is making a fist, stretching the palm or making other gestures, the relative positional relationship between the two points and the wrist changes the least. Therefore, the knuckle joints of the index finger knuckle and the little finger knuckle are taken as stable direction points, which are combined with the wrist knuckle to form a biomechanical feature vector, thereby improving the stability of gesture recognition.

[0093] In addition, the index finger knuckle and the little finger knuckle constitute the maximum width of the palm base. With the wrist as the vertex and the two points as the other two vertices, the plane on which the triangle is located can most accurately represent the orientation plane of the palm. If two adjacent knuckles (such as the index finger and the middle finger) are selected, the included angle between the vector formed by the two knuckles and the "wrist-index finger" vector is small, close to collinear. This will cause the calculated cross product normal vector to be sensitive to small key point detection errors, thereby affecting the stability of the detection result. The index finger and the little finger knuckles have the largest span, and the included angle between the vector formed by the two knuckles and the wrist is the largest, so that the cross product calculation result is the most robust. Therefore, the biomechanical feature vector constructed can also maximize the geometric information of the palm and avoid the influence of the collinearity problem on the stability of the detection result.

[0094] After obtaining the above-mentioned biomechanical feature vector, the Z component of V_Normal is checked. The Z component serves as depth information, and the positive and negative signs indicate whether V_Normal is facing or away from the camera. V_Normal can be input into a pre-constructed decision tree model for right-hand or left-hand determination. The decision tree model can be used as a classifier. For each recognized hand target, there is a corresponding biomechanical feature vector, and the decision tree model analyzes the biomechanical feature vector to determine whether the corresponding hand target is "left hand" or "right hand" by executing the following rules, thereby obtaining the corresponding label:

[0095] If the palm is detected to be facing the observer, the hand is considered "right hand" if the Z component of V_Normal is greater than 0, otherwise it is considered "left hand".

[0096] In this process, because the decision tree's judgment logic is extremely simple (only a floating-point comparison), the corresponding computational overhead is small and almost negligible, thus meeting the low latency requirements in practical application scenarios.

[0097] In this embodiment, the robustness and accuracy of left-hand and right-hand determination are enhanced. This, combined with the technical features of fusing gesture regions and 3D gesture key points, further transforms the challenge of 3D gesture detection from the hardware design level to the algorithm level. It breaks through the dependence on dedicated hardware and solves the problem of traditional methods failing to accurately determine left and right hands when the hands rotate or cross. Even if the hand rotates or flips at any angle within the plane, the relative geometric relationship between the vectors A and B (i.e., the directionality of the cross product) remains unchanged, thus ensuring the accuracy and robustness of left-hand and right-hand discrimination under various postures.

[0098] Furthermore, in one embodiment, based on the above step S250, a three-dimensional gesture dataset is constructed based on the determination result and fusion information, which may specifically include:

[0099] Based on the judgment results and fusion information, the annotation data corresponding to each gesture image in the gesture image sequence is generated; based on the gesture image sequence and the annotation data corresponding to each gesture image in the gesture image sequence, a three-dimensional gesture dataset is constructed.

[0100] Specifically, the labels for the left and right hands output by the decision tree, combined with the relevant label information of the gesture regions and the relevant label information of the 3D gesture key points, can be used to construct the annotation data corresponding to each gesture image. Together with the corresponding gesture images, this forms a 3D gesture dataset. Therefore, this embodiment can directly perform preliminary automated annotation on videos captured by real cameras, eliminating the need for manual annotation from scratch, reducing reliance on manual annotation, and improving annotation efficiency.

[0101] In one embodiment, a three-dimensional gesture dataset is constructed based on a sequence of gesture images and the annotation data corresponding to each gesture image in the sequence. Specifically, this may include:

[0102] The labeled data is corrected to obtain corrected labeled data; and a virtual gesture dataset generated by an open-source robot system simulation platform for a preset 3D hand model is obtained; the virtual gesture dataset includes dynamic gesture sequences and ground truth labeled data corresponding to the dynamic gesture sequences; the initial dataset composed of gesture image sequences and corrected labeled data is fused with the virtual gesture dataset to obtain a 3D gesture dataset.

[0103] The result of the automatic labeling described above can be output to a manual labeling staff through a front-end interface or other output forms, and the manual labeling staff reviews and corrects the result of the automatic labeling (for example, corrects the unique identification (ID) jump or left-right hand misjudgment under extreme occlusion). Therefore, compared with manual labeling in the related art, this way of first automatic labeling and then manual correction can improve the labeling efficiency, for example, reducing the labeling time of each frame of gesture image from 5 minutes to 30 seconds. In this way, the corrected labeling data can be obtained.

[0104] In addition, with the help of an open source robot system simulation platform (Gazebo), a pre-set dynamic gesture sequence can be executed on this simulation platform using an accurate three-dimensional hand model, and then the simulation platform will automatically and error-free export all the ground truth labeling data corresponding to the dynamic gesture sequence, for example, including the three-dimensional world coordinate ground truth of 21 three-dimensional gesture key points, accurate left-right hand labels, gesture categories and accurate two-dimensional bounding boxes (gesture regions), thereby constructing an accurate and high-quality virtual gesture dataset.

[0105] The corrected labeling dataset and the gesture image sequence constitute an initial dataset, and then are mixed with the virtual gesture dataset generated by the simulation platform, thereby constituting a final three-dimensional gesture dataset. In this way, in the present embodiment, the zero-error virtual gesture dataset generated by the simulation platform is mixed with the corrected real dataset. The virtual gesture dataset guarantees the coverage of various standard hand postures and extreme angles, and the real dataset introduces the influence of noise such as light changes, skin color differences and complex backgrounds in the real world, thereby constructing a three-dimensional gesture dataset with large scale, high diversity and accurate labeling.

[0106] On the basis of this three-dimensional gesture dataset, the model is trained for gesture recognition, which can make the model not try to guess the missing points during inference, but train a more powerful model to learn to estimate the occluded part according to the visible part. Specifically, on the above simulation platform, various extreme occlusion gestures can be controlled by the hand model. Even if a key point is occluded in the virtual camera view, its three-dimensional space coordinate true value is always known. Therefore, a large amount of data containing more occlusion conditions with accurate three-dimensional labeling is generated. These zero-error virtual data are fused with real-world data corrected by semi-automatic labeling to construct a high-quality mixed dataset. Training or fine-tuning the model with this mixed dataset can enable the model to gradually master the internal constraints of hand anatomy during the learning process. For example, the model can learn that even if the index finger is completely blocked by the thumb, according to the palm posture and the position of other visible fingers, the index finger tip is probably in which spatial range. Therefore, the trained model itself has robustness to occlusion, and can accurately infer the position of the occluded key point according to the context information, thereby reducing the key point loss rate.

[0107] In some embodiments, the fusion of YOLOv5 and MediaPipe can be used as an auxiliary, and YOLOv5 can provide a stable and accurate hand overall bounding box, which is fused with the MediaPipe key point extraction to improve the probability of detecting more visible key points under slight occlusion, and provide more abundant basis for model training. Thus, the embodiment can solve the left and right hand misjudgment problem caused by changes in viewing angle and hand occlusion in a monocular vision system, and reduce the cost of dataset construction.

[0108] In addition, in an embodiment, based on the above step S220, the gesture region detection on the gesture image sequence to obtain the gesture region sequence can include:

[0109] Based on the preset target detection model, the gesture region detection on the gesture image sequence is performed to obtain the gesture region sequence; the target detection model is a model formed after replacing the backbone network in the YOLOv5 model with a lightweight convolutional neural network and adding a channel attention mechanism module.

[0110] In this embodiment, considering that a model with higher accuracy means greater computational complexity and higher latency, which is not suitable for actual interactive application scenarios. Therefore, in this embodiment, a more lightweight model architecture is adopted, that is, a lightweight convolutional neural network (MobileNetV3) is used in combination with a channel attention mechanism (SE) module. Specifically, the backbone network of YOLOv5 is replaced with MobileNetV3. The MobileNet series is a lightweight network designed for mobile terminals and embedded devices, which reduces the computational complexity and parameter quantity by several orders of magnitude through techniques such as deep separable convolution, while ensuring comparable accuracy.

[0111] While being lightweight, the SE (Squeeze-and-Excitation) attention module is added to allow the network to adaptively learn the importance of different feature channels, improving the utilization of computing resources and achieving more significant accuracy improvement with less computational cost.

[0112] In this way, the performance of the overall algorithm can be improved, and redundant calculations can be avoided. Finally, based on this lightweight and efficient model structure, the inference time of the entire process is compressed to within 100 milliseconds, thereby achieving the purpose of smooth interaction and realizing low-latency interaction.

[0113] Figure 4 is a flowchart of a gesture dataset construction method of some embodiments, as shown in Figure 4 The gesture dataset construction method includes the following steps:

[0114] Step S401, an RGB image input is obtained to obtain a gesture image sequence. The gesture dataset construction method is applicable to scenarios including an industrial operation scenario of excavator driver gesture control of a mechanical arm, a medical scenario of a surgeon gesture operation of a three-dimensional organ model for preoperative planning, and an intelligent cockpit scenario of driver gesture control of a vehicle-mounted entertainment system. The hardware configuration can be a monocular camera with a model number of "Logitech C920 (1080P@30fps)" and a processor with a model number of "NVIDIA Jetson Xavier NX". The RGB image can be obtained by the monocular camera.

[0115] Step S402, gesture region detection is performed on the gesture image sequence to obtain a gesture region sequence. For example, gesture region detection is performed based on YOLOv5.

[0116] Step S403, three-dimensional gesture key point extraction is performed on the gesture image sequence to obtain a three-dimensional gesture key point sequence. For example, key point extraction is performed based on MediaPipe. The execution of step S402 and step S403 can be performed in parallel.

[0117] At step S404, the output results of step S402 and step S403 are fused to obtain fusion information. The specific fusion process can refer to the above embodiments, which will not be described here.

[0118] At step S405, left and right hand determination is performed on the fusion information. For example, left and right hand determination is realized based on a decision tree.

[0119] At step S406, a three-dimensional gesture dataset is constructed.

[0120] The above steps S401 to S406, through the combination of gesture region detection and three-dimensional gesture key point detection, obtain fusion information, and complete accurate left and right hand determination based on the fusion information, which can reduce the dependence of gesture recognition on sensor hardware from the algorithm and software levels, and improve the accuracy of labeling, thereby reducing the dependence on manual labeling, and finally improving the efficiency of three-dimensional gesture labeling.

[0121] Figure 5 A timing diagram is constructed for the gesture dataset of some embodiments, as shown in Figure 5 The monocular camera transmits a gesture image sequence to the YOLOv5 model and the MediaPipe framework, respectively. The YOLOv5 model outputs a gesture region sequence, and the MediaPipe framework outputs a three-dimensional gesture key point sequence; the gesture region sequence and the three-dimensional gesture key point sequence are fused (data correlation and coordinate system alignment) to obtain fusion information, and then an anatomical constraint decision tree (i.e., the above biomechanical feature vector is input into the decision tree for left and right hand determination) is used to obtain left and right hand determination results. Finally, combined with the structured information such as the coordinates, categories, left and right hand labels of the gesture region, the three-dimensional gesture dataset is constructed, and then output through the output end.

[0122] In this embodiment, a gesture detection method is also provided. Figure 6 A flowchart of the gesture detection method of this embodiment is shown in Figure 6 The gesture detection method can specifically include the following steps:

[0123] At step S601, a pre-constructed three-dimensional gesture dataset is obtained; the three-dimensional gesture dataset is generated based on the gesture dataset construction method provided in the above embodiments.

[0124] Step S602, based on the three-dimensional gesture dataset, a gesture detection model is trained; wherein the gesture dataset construction method provided in the above embodiments can adapt to any mainstream, data-driven gesture recognition algorithm, such as a spatiotemporal feature-based algorithm, such as a three-dimensional convolutional neural network (3D CNN) combined with a long short-term attention network (LSTM) or a self-attention (Transformer) model. The above three-dimensional gesture dataset can provide three-dimensional coordinate data with a time sequence relationship for such a network. In addition, it can also adapt to an algorithm based on a graph neural network (GCN), which regards the key points of the hand as nodes of a graph and the skeleton as edges to learn the structural features of the gesture. The above three-dimensional gesture dataset directly provides the three-dimensional coordinates of the nodes, which is an ideal input for training the GCN. In addition, it can also adapt to an algorithm that directly classifies poses, such as for static gesture recognition, the single-frame three-dimensional key point coordinates in the above three-dimensional gesture dataset can be directly used as features and input into a support vector machine (SVM), a multilayer perceptron (MLP), or a simple classification network. In addition, it also adapts to an algorithm that needs to be pre-trained or fine-tuned. Any existing gesture recognition model can use the above three-dimensional gesture dataset for fine-tuning if it is desired to improve its performance in complex scenarios such as occlusion, multiple perspectives, and crossed hands.

[0125] Step S603, according to the gesture detection model, gesture detection is performed on the to-be-detected image to obtain a gesture detection result.

[0126] The above steps S601 to S603 use three-dimensional gestures with high annotation efficiency and high annotation accuracy for gesture recognition, which can improve the accuracy and efficiency of gesture recognition.

[0127] In the embodiments, a gesture dataset construction device is also provided, which is used to implement the above embodiments and preferred embodiments, and details are not repeated. The terms "module", "unit", "sub-unit" and the like used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and contemplated.

[0128] Figure 7 is a structural block diagram of the gesture dataset construction device 70 of the embodiments, as Figure 7 shown, the gesture dataset construction device 70 includes an acquisition module 71, a detection module 72, a fusion module 73, a determination module 74, and a construction module 75; wherein:

[0129] The module 71 is used to acquire a sequence of gesture images; the detection module 72 is used to detect gesture regions in the sequence of gesture images to obtain a sequence of gesture regions; and to detect three-dimensional gesture key points in the sequence of gesture images to obtain a sequence of three-dimensional gesture key points; the fusion module 73 is used to fuse the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information; the determination module 74 is used to determine left and right hands based on the fusion information to obtain a determination result; and the construction module 75 is used to construct a three-dimensional gesture dataset based on the determination result and the fusion information.

[0130] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0131] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0132] In one embodiment, a computer device is provided, which may be a terminal. Figure 8 This is an internal structural diagram of the computer device in this embodiment. (As shown...) Figure 8 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for constructing a gesture dataset. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0133] Those skilled in the art can understand that Figure 8 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0135] Step one, obtaining a gesture image sequence;

[0136] Step two, performing gesture region detection on the gesture image sequence to obtain a gesture region sequence, and performing three-dimensional gesture key point detection on the gesture image sequence to obtain a three-dimensional gesture key point sequence;

[0137] Step three, performing feature fusion on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information;

[0138] Step four, performing left and right hand determination on the fusion information to obtain a determination result;

[0139] Step five, constructing a three-dimensional gesture data set based on the determination result and the fusion information.

[0140] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0141] Step one, obtaining a gesture image sequence;

[0142] Step two, performing gesture region detection on the gesture image sequence to obtain a gesture region sequence, and performing three-dimensional gesture key point detection on the gesture image sequence to obtain a three-dimensional gesture key point sequence;

[0143] Step three, performing feature fusion on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information;

[0144] Step four, performing left and right hand determination on the fusion information to obtain a determination result;

[0145] Step five, constructing a three-dimensional gesture data set based on the determination result and the fusion information.

[0146] In one embodiment, a computer program product is provided, comprising a computer program, and the computer program is executed by a processor to implement the following steps:

[0147] Step one, obtaining a gesture image sequence;

[0148] Step two, gesture region detection is performed on the gesture image sequence to obtain a gesture region sequence, and three-dimensional gesture key point detection is performed on the gesture image sequence to obtain a three-dimensional gesture key point sequence;

[0149] Step three, feature fusion is performed on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information;

[0150] Step four, left and right hand determination is performed on the fusion information to obtain a determination result;

[0151] Step five, based on the determination result and the fusion information, a three-dimensional gesture dataset is constructed.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0153] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of each method. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0154] It should be understood that the specific embodiments described herein are merely used to explain this application, but not to limit it. According to the embodiments provided in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0155] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are information and data authorized by the user or fully authorized by all parties.

[0156] It is apparent that the drawings depicted are only a few exemplifying or embodiment of the present application and can be applied to other similar situations without paying creative labor by a person of ordinary skill in the art. In addition, it is understood that although the work done in the development process can be complex and long, some design, manufacture or production changes made according to the technical content disclosed in the present application by a person of ordinary skill in the art are only routine technical means and should not be regarded as insufficient disclosure of the present application.

[0157] The word "embodiment" in the present application means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The presence of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean independence or alternative to other embodiments. It is clear or implicitly understood by a person of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0158] The above-described embodiments only express several implementation manners of the present application, which are described in detail and specifically, but cannot be understood as a limitation on the scope of patent protection. It should be noted that for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for constructing a gesture dataset, characterized in that, The method comprises the following steps: obtaining a gesture image sequence; performing gesture region detection on the gesture image sequence to obtain a gesture region sequence; performing three-dimensional gesture key point detection on the gesture image sequence to obtain a three-dimensional gesture key point sequence; performing feature fusion on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information; performing left-right hand determination on the fusion information to obtain a determination result; based on the determination result and the fusion information, constructing a three-dimensional gesture dataset; wherein: performing left-right hand determination on the fusion information to obtain a determination result, comprising: determining a biomechanical feature vector according to the metacarpophalangeal key point and the wrist key point in the fusion information; inputting the biomechanical feature vector into a pre-constructed decision tree model to perform left-right hand determination to obtain the determination result.

2. The gesture dataset construction method of claim 1, wherein, obtaining a gesture image sequence, comprising: obtaining a gesture image sequence collected by a monocular camera.

3. The gesture dataset construction method of claim 1, wherein, performing feature fusion on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information, comprising: associating the gesture region sequence and the three-dimensional gesture key point sequence according to the position information of each gesture region in the gesture region sequence and the key point position statistical information of each group of gesture key points in the three-dimensional gesture key point sequence; performing coordinate system alignment on the associated gesture region sequence and the three-dimensional gesture key point sequence to obtain the fusion information.

4. The gesture dataset construction method of claim 3, wherein, associating the gesture region sequence and the three-dimensional gesture key point sequence according to the position information of each gesture region in the gesture region sequence and the key point position statistical information of each group of gesture key points in the three-dimensional gesture key point sequence, comprising: determining the distance information between the position information of each gesture region in the gesture region sequence and the key point position statistical information of each group of gesture key points in the three-dimensional gesture key point sequence, respectively; based on the distance information, associating each gesture region and each group of gesture key points based on the Hungarian algorithm.

5. The gesture dataset construction method of claim 1, wherein, based on the determination result and the fusion information, constructing a three-dimensional gesture dataset, comprising: based on the determination result and the fusion information, generating label data corresponding to each gesture image in the gesture image sequence; based on the gesture image sequence and the label data corresponding to each gesture image in the gesture image sequence, constructing the three-dimensional gesture dataset.

6. The gesture dataset construction method of claim 5, wherein, based on the gesture image sequence and the label data corresponding to each gesture image in the gesture image sequence, constructing the three-dimensional gesture dataset, comprising: correcting the label data to obtain corrected label data; and obtaining a virtual gesture dataset generated by an open-source robot system simulation platform for a preset three-dimensional hand model; the virtual gesture dataset comprises a dynamic gesture sequence and true value label data corresponding to the dynamic gesture sequence; fusing an initial dataset composed of the gesture image sequence and the corrected label data with the virtual gesture dataset to obtain the three-dimensional gesture dataset.

7. The gesture dataset construction method of claim 1, wherein, performing gesture region detection on the gesture image sequence to obtain a gesture region sequence, comprising: Based on a preset target detection model, gesture region detection is performed on the gesture image sequence to obtain a gesture region sequence; the target detection model is a model formed after replacing a backbone network in a YOLOv5 model with a lightweight convolutional neural network and adding a channel attention mechanism module.

8. A gesture detection method, characterized by, The method comprises the following steps: obtaining a pre-constructed three-dimensional gesture dataset; the three-dimensional gesture dataset is generated based on the gesture dataset construction method in any one of claims 1 to 7; training a gesture detection model based on the three-dimensional gesture dataset; performing gesture detection on a to-be-detected image according to the gesture detection model to obtain a gesture detection result.

9. A gesture dataset construction apparatus characterized by comprising: The method comprises the following steps: an obtaining module, a detection module, a fusion module, a determination module and a construction module; wherein: the obtaining module is configured to obtain a gesture image sequence; the detection module is configured to perform gesture region detection on the gesture image sequence to obtain a gesture region sequence, and perform three-dimensional gesture key point detection on the gesture image sequence to obtain a three-dimensional gesture key point sequence; the fusion module is configured to perform feature fusion on the gesture region sequence and the three-dimensional gesture key point sequence to obtain fusion information; the determination module is configured to perform left-right hand determination on the fusion information to obtain a determination result; the construction module is configured to construct a three-dimensional gesture dataset based on the determination result and the fusion information; wherein: performing left-right hand determination on the fusion information to obtain a determination result comprises: determining a biomechanics feature vector according to a metacarpophalangeal joint key point and a wrist key point in the fusion information; inputting the biomechanics feature vector into a pre-constructed decision tree model to perform left-right hand determination and obtain the determination result.

Images