A 3D gesture data generation method, apparatus, device and computer program product

Abstract

The embodiment of the application discloses a 3D gesture data generation method, device, equipment and computer program product, belongs to the technical field of computer vision, and solves the problem that each frame image needs to be processed twice by a convolutional neural network when the handedness is judged in the prior art, and the calculation resources are wasted. Including, obtaining initial gesture data corresponding to a hand video frame output by a backbone network; wherein the initial gesture data at least includes one of an initial handedness label and an initial handedness confidence; determining a data scene category corresponding to the initial gesture data through a preset scene detection mechanism; wherein the data scene category at least includes one of a data fuzzy scene and a data clear scene; based on the data scene category, whether to trigger an auxiliary network is selected to output final gesture data through the backbone network and / or the auxiliary network.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device and computer program product for generating 3D gesture data. Background Technology

[0002] Gesture recognition technology using RGB cameras has become a research hotspot in the field of human-computer interaction due to its high versatility. Especially in the application of smart glasses, recognizing hand movements captured by smart glasses can improve the accuracy of AR / VR interaction or sign language recognition and translation.

[0003] MediaPipe, a commonly used technology, is an open-source, cross-platform machine learning application framework that can estimate 3D key points of the hand from 2D video in real time. The MediaPipe process includes determining hand dominance (i.e., left or right hand) and outputs a corresponding confidence score.

[0004] However, MediaPipe's dominant hand determination has significant shortcomings in certain scenarios. First, when the hand pose is nearly symmetrical or the hand is partially occluded, the confidence level of the dominant hand determination output by the backbone network drops significantly, leading to unreliable results. Second, during rapid hand flips or continuous movements, the dominant hand determination of the backbone network may jump back and forth between "left hand" and "right hand" within several consecutive frames, causing instability in the output data stream and severely interfering with the logical judgments of upper-layer applications.

[0005] Currently, to improve the accuracy of chirality determination, a convolutional neural network is usually added. That is, regardless of whether MediaPipe's own determination is accurate, each frame of the image needs to undergo secondary processing by an additional convolutional neural network, resulting in a serious waste of computing resources. Summary of the Invention

[0006] This application provides a 3D gesture data generation method, apparatus, device, and computer program product to solve the following technical problem: In the prior art, when performing chirality judgment, each frame of image needs to undergo secondary processing by a convolutional neural network, resulting in a serious waste of computing resources.

[0007] The embodiments of this application adopt the following technical solutions:

[0008] This application provides a 3D gesture data generation method. It includes: acquiring initial gesture data corresponding to hand video frames output by a backbone network; wherein the initial gesture data includes at least one of an initial hand-righting label and an initial hand-righting confidence level; determining the data scene category corresponding to the initial gesture data through a preset scene detection mechanism, including: comparing the hand-righting confidence level of the current frame hand image corresponding to the hand video frame with a preset confidence threshold to obtain a confidence detection result; if the confidence detection result meets the preset confidence conditions, performing label jump detection on the hand-righting labels of multiple frames of hand images within a preset time window to obtain a label stability detection result; determining the data scene category corresponding to the initial gesture data based on the confidence detection result and / or the label stability detection result; wherein the data scene category includes at least one of a data blurry scene and a data clear scene; and selecting whether to trigger an auxiliary network based on the data scene category to output the final gesture data through the backbone network and / or the auxiliary network.

[0009] In one implementation of this application, the data scene category corresponding to the initial gesture data is determined based on the confidence level detection result and / or the label stability detection result. Specifically, this includes: when the confidence level detection result indicates that the hand advantage confidence of the current frame hand image is less than a preset confidence threshold, the data scene category is determined to be a data fuzzy scene; when the confidence level detection result indicates that the hand advantage confidence of the current frame hand image is not less than a preset confidence threshold, and the label stability detection result indicates that the label has jumped, the data scene category is determined to be a data fuzzy scene; when the confidence level detection result indicates that the hand advantage confidence of the current frame hand image is not less than a preset confidence threshold, and the label stability detection result indicates that the label has not jumped, the data scene category is determined to be a data clear scene.

[0010] In one implementation of this application, before determining the data scene category corresponding to the initial gesture data through a pre-set scene detection mechanism, the method further includes: drawing a hand detection box based on the hand key point coordinates output by the backbone network; concatenating the hand detection boxes corresponding to different chirality labels in chronological order to obtain hand detection box trajectories corresponding to different chirality labels; determining reference trajectory data in the historical database whose similarity difference with the hand detection box trajectory meets preset conditions; wherein the reference trajectory data includes at least one of a reference confidence threshold and a reference time window; obtaining the change data of the hand key points corresponding to the current hand video frame; and adjusting the reference confidence threshold and the reference time window based on the change data.

[0011] In one implementation of this application, the reference confidence threshold and reference time window are adjusted based on the changing data, specifically including: constructing a confidence data adjustment set based on the changing data; wherein the confidence data adjustment set includes at least one of the following: the degree of occlusion of hand key points, the distinction between hand key points and background, and the dispersion of hand key points; generating a first adjustment factor based on the confidence data adjustment set and the first adjustment coefficients corresponding to different data in the confidence data adjustment set; adjusting the reference confidence threshold according to the first adjustment factor and the confidence adjustment ratio; constructing a time window data adjustment set based on the changing data; wherein the time window data adjustment set includes at least one of the following: the movement speed of hand key points, the displacement of hand key points, and the coordinate fluctuation value corresponding to hand key points; generating a second adjustment factor based on the time window data adjustment set and the second adjustment coefficients corresponding to different data in the time window data adjustment set; and adjusting the reference time window according to the second adjustment factor and the time window adjustment ratio.

[0012] In one implementation of this application, before selecting whether to trigger the auxiliary network based on the data scene category, the method further includes: performing horizontal flipping processing on the current frame image sample in the acquired hand video data sample; performing corresponding horizontal flipping processing on the adjacent frame images of the current frame image sample based on the temporal relationship of the hand video data samples; adding timestamp information to the enhanced image sample data after flipping, sorting the enhanced image samples according to the timestamp information, and constructing a hand image sample sequence; wherein, the chirality labels corresponding to the image samples before and after flipping are opposite; and training a pre-set neural network based on the hand video data sample and the hand image sample sequence to obtain the auxiliary network.

[0013] In one implementation of this application, based on the data scene type, it is selected whether to trigger an auxiliary network to output the final gesture data through the auxiliary network and / or the backbone network. Specifically, this includes: when the data scene is determined to be clear, the initial gesture data output by the backbone network is used as the final gesture data; when the data scene is determined to be blurry, the hand image corresponding to the hand video frame is input into the auxiliary network, and the final gesture data is output through the auxiliary network; wherein, the gesture data output by the auxiliary network includes at least one of the corrected chirality label and the corrected chirality confidence.

[0014] In one implementation of this application, after the final gesture data is output through the auxiliary network, the method further includes: correcting the initial gesture data using the final gesture data; integrating the corrected gesture data with the hand key point data output by the backbone network; and transmitting the integrated data to the camera terminal.

[0015] This application provides a 3D gesture data generation device, comprising: a data acquisition unit for acquiring initial gesture data corresponding to hand video frames output by a backbone network; wherein the initial gesture data includes at least one of an initial hand-righting label and an initial hand-righting confidence level; a data detection unit for determining the data scene category corresponding to the initial gesture data through a preset scene detection mechanism, including: comparing the hand-righting confidence level of the current frame hand image corresponding to the hand video frame with a preset confidence threshold to obtain a confidence detection result; if the confidence detection result meets the preset confidence condition, performing label jump detection on the hand-righting labels of multiple frames of hand images within a preset time window to obtain a label stability detection result; and determining the data scene category corresponding to the initial gesture data based on the confidence detection result and / or the label stability detection result; wherein the data scene category includes at least one of a data blurry scene and a data clear scene; and a data output unit for selecting whether to trigger an auxiliary network based on the data scene category, so as to output the final gesture data through the backbone network and / or the auxiliary network.

[0016] This application provides a 3D gesture data generation device, including at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to: acquire initial gesture data corresponding to a hand video frame output from a backbone network; wherein the initial gesture data includes at least one of an initial dominant hand label and an initial dominant hand confidence score; and determine the data scene category corresponding to the initial gesture data through a preset scene detection mechanism, including: determining the dominant hand of the current frame hand image corresponding to the hand video frame. The confidence level is compared with a preset confidence threshold to obtain a confidence detection result. If the confidence detection result meets the preset confidence conditions, label jump detection is performed on the hand advantage labels of multiple frames of hand images within a preset time window to obtain label stability detection results. Based on the confidence detection results and / or label stability detection results, the data scene category corresponding to the initial gesture data is determined. The data scene category includes at least one of data blurry scene and data clear scene. Based on the data scene category, it is selected whether to trigger the auxiliary network to output the final gesture data through the backbone network and / or auxiliary network.

[0017] This application provides a computer program product that, when run, can execute the steps of the 3D gesture data generation method as described in any of the preceding embodiments.

[0018] The above-mentioned technical solutions adopted in this application embodiment can achieve the following beneficial effects: This application embodiment constructs a pre-set scene detection mechanism to divide the initial gesture data output by the backbone network into data corresponding to either blurry or clear scenes. This allows for the selection of different gesture data generation strategies based on different scenes, avoiding misjudgments caused by using a single processing method in different scenarios. Secondly, based on the data scene category, this application embodiment directly uses the backbone network output results in clear data scenes, reducing computational costs; in blurry data scenes, an auxiliary network is enabled to correct the gesture data output by the backbone network, improving the accuracy and reliability of gesture data in complex environments and achieving a balance between performance and efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0020] Figure 1 This is a schematic diagram of the system architecture that can be applied to the embodiments of this application;

[0021] Figure 2 A flowchart of a 3D gesture data generation method provided in this application embodiment;

[0022] Figure 3 A flowchart illustrating a method for determining data scene categories provided in this application embodiment;

[0023] Figure 4 A flowchart for outputting final data is provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the overall process of a 3D gesture data generation method provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of a 3D gesture data generation device provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of a 3D gesture data generation device provided in an embodiment of this application.

[0027] Figure label:

[0028] 101: Camera terminal, 102: Communication network, 103: Backbone network, 104: Scene detection module, 105: Auxiliary network, 106: Cloud server. Detailed Implementation

[0029] This application provides a method, apparatus, and device for generating 3D gesture data.

[0030] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0031] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0032] Figure 1 This is a schematic diagram of the system architecture that can be applied to the embodiments of this application, such as... Figure 1 As shown, the 3D gesture data generation method can be applied to, for example... Figure 1 In the illustrated environment, the application environment may include a camera terminal 101, a communication network 102, and a cloud server 106. The cloud server 106 further includes a backbone network 103, a scene detection module 104, and an auxiliary network 105. The camera terminal 101 is a terminal device equipped with a camera, such as smart glasses, a smart cockpit, or a smart home device. For example, with smart glasses, the camera can capture images of the user's hands during interaction for sign language recognition and translation, or enable precise two-handed operations in virtual space, such as grasping, placing, or manipulating virtual interfaces. With a smart cockpit, the camera can capture the driver's or passenger's left and right hand gestures to achieve contactless control of the central control screen, air conditioning, music, etc. With a smart home, the camera can capture the user's left and right hand gestures to control lights, curtains, televisions, and other smart home devices.

[0033] It should be noted that the camera set on the camera terminal 101 in this embodiment can be a monocular RGB camera, a binocular RGB camera, or a multi-lens RGB camera, depending on the characteristics of the terminal device. When generating 3D gesture data, the camera is used to acquire hand video images and send the acquired hand video images to the cloud server 106.

[0034] Furthermore, the camera terminal 101 sends the acquired hand video images to the backbone network 103 of the cloud server 106 via the communication network 102. The backbone network 103 processes each frame of the hand image and outputs initial gesture data, which includes at least hand key point coordinates, dominant hand label, and dominant hand confidence score. The scene detection module 104 detects the initial gesture data. If it meets the preset confidence score and dominant hand label stability conditions, it directly outputs the gesture data through the backbone network 103 and sends the initial gesture data to the camera terminal 101. If the preset reliability conditions and / or the chirality label stability conditions are not met, the auxiliary network 105 is activated, the current hand image is input into the lightweight auxiliary network 105, and the chirality label output by the backbone network 103 is replaced with the chirality label output by the auxiliary network 105. Finally, the chirality label output by the auxiliary network 105 and the hand key point coordinates output by the backbone network 103 are used to construct the final gesture data, which is then sent to the camera terminal 101.

[0035] Figure 2 A flowchart of a 3D gesture data generation method provided in this application embodiment is shown below. Figure 2 As shown, the 3D gesture data generation method includes the following steps S201-S203:

[0036] S201. Obtain the initial gesture data corresponding to the hand video frames output by the backbone network.

[0037] In one implementation of this application, a hand video frame is first acquired via a camera terminal and then uploaded to the backbone network. Initial gesture data is then output via the backbone network. The following detailed explanation uses binocular smart glasses as the camera terminal:

[0038] First, the dual cameras built into the binocular smart glasses capture real-time images of the user's hand from different perspectives, quickly obtaining video frames containing rich depth information and color details. After acquiring the video frames, the binocular smart glasses use their built-in data transmission modules, such as Wi-Fi or Bluetooth, to upload the hand video frames to the backbone network in real time. In this embodiment, the backbone network is MediaPipe, an open-source, cross-platform machine learning application framework.

[0039] The backbone network is used to perform 3D gesture estimation on each frame of the acquired hand image, and obtains data such as the coordinates of multiple 3D key points of the hand, the initial dominant hand label, and the initial dominant hand confidence.

[0040] Specifically, MediaPipe's 3D gesture data generation model can locate 21 key points of the hand, such as fingertips and knuckles. Each key point's coordinates contain spatial position information in three dimensions: X, Y, and Z, thus constructing a three-dimensional spatial pose model of the hand. Secondly, while acquiring the coordinates of the 3D key points of the hand, the dominant hand recognition branch within the MediaPipe model analyzes the input image. This branch determines the dominant hand by learning the structural differences and movement patterns of the left and right hands in 3D space. The model outputs a dominant hand label and a confidence score based on the degree of feature matching. For example, when the model has a high degree of matching for the right-hand features and conforms to the right-hand pattern distribution in the training data, the dominant hand label is output as "right," and a high confidence score (e.g., 0.95) is assigned, indicating the model's certainty about the judgment result.

[0041] S202. The data scene category corresponding to the initial gesture data is determined through a preset scene detection mechanism.

[0042] In one implementation of this application, after obtaining the initial chirality label and initial chirality confidence from the backbone network, the initial chirality label and initial chirality confidence are detected respectively, so as to determine whether the auxiliary network needs to be started based on the detection results. Figure 3 A flowchart of a data scene category determination method provided in this application embodiment is shown below. Figure 3 As shown, the method for determining data scenario categories includes the following steps: S301-S303:

[0043] 301. Compare the confidence level of the hand image in the current frame corresponding to the hand video frame with the preset confidence threshold to obtain the confidence detection result.

[0044] The preset confidence threshold in this embodiment is a target value calculated based on a large amount of historical data, used to measure the reliability of the gesture recognition result in the current frame. The extracted initial chirality confidence is compared with the preset confidence threshold. If the chirality confidence is greater than or equal to the threshold, it indicates that the gesture recognition result in the current frame has high reliability. If it is less than the threshold, it indicates that the gesture recognition result in the current frame has low reliability. This yields the confidence detection result, which preliminarily reflects the quality of the gesture data in the current frame.

[0045] For example, the preset confidence threshold T_conf = 0.85 in the embodiments of the present application. The initial handedness confidence C_current = 0.95 is obtained. At this time, the confidence detection result is T_conf (0.85) <= C_current (0.95), so the confidence detection passes. If the current frame confidence C_current = 0.60, and the confidence detection result is C_current (0.60) < T_conf (0.85) at this time, then the confidence detection fails.

[0046] 302. When the confidence detection result meets the preset confidence condition, perform label jump detection on the handedness labels of multiple frames of hand images within the preset time window to obtain the label stability detection result.

[0047] When the confidence detection result is greater than or equal to the preset confidence threshold, label stability detection is performed. If the confidence detection result is less than the preset confidence threshold, label stability detection is not required, and at this time, the auxiliary network is directly triggered to correct the initial hand data.

[0048] Specifically, the embodiments of the present application preset a time window, and the size of this window can be flexibly adjusted according to the actual application scenario, such as the speed of gesture actions, system response requirements, etc. Within the time window, collect the handedness labels of multiple frames of hand images and perform detection by analyzing the changes of these labels. If the handedness labels between adjacent frames change frequently, that is, within the time window, the number of times the handedness label changes is greater than the preset change number threshold, it is determined that a label jump occurs, indicating that the gesture recognition result is unstable during this period. If the number of times the handedness label changes is not greater than the preset change number threshold, it is determined that no label jump occurs, and thus the label stability detection result is obtained. Among them, the preset change number threshold can be 10% - 20% of the total number of frames of hand images within the time window, or it can be adjusted according to the actual application situation, and the embodiments of the present application do not limit this. Through label stability detection, the fluctuation of the gesture recognition result within a period of time can be effectively captured to assist in judging the reliability of the data.

[0049] For example, the embodiments of the present application set a time window W_time with a size of 5 frames, and check the handedness labels of the last 5 frames stored within W_time. Assuming that the label stability detections are all "left", it indicates that no label jump occurs, and at this time, the label stability detection passes.

[0050] 303. Based on the confidence detection result and / or the label stability detection result, determine the data scenario category corresponding to the initial gesture data.

[0051] In one implementation of this application, when the confidence level detection result indicates that the confidence level of the hand image in the current frame is less than a preset confidence threshold, the data scene category is determined to be a data fuzzy scene. Low confidence level implies that the backbone network has a significant possibility of error in recognizing the hand advantage of the current frame. Whether due to image blurring, occlusion, or difficulty in feature extraction due to complex hand poses, this indicates poor data quality. Therefore, without referring to the label stability detection result, the data scene category is directly determined to be a data fuzzy scene.

[0052] In one implementation of this application, when the confidence test result shows that the hand image's right-hand confidence is not less than a preset confidence threshold, and the label stability test result shows that the label has changed, the data scene category is determined to be a data fuzzy scene. Frequent label changes indicate that the gesture recognition result is significantly unstable, possibly due to environmental interference, rapid and irregular hand movements, or other factors that cause contradictory judgments by the model between different frames. At this point, the data scene category is marked as a data fuzzy scene, and based on this determination, an auxiliary network is prepared to be invoked to correct the initial gesture data.

[0053] In one implementation of this application, when the confidence test result is that the confidence of the hand image in the current frame is not less than a preset confidence threshold, and the label stability test result is that the label has not jumped, the data scene category is determined to be a clear data scene.

[0054] In clear scenarios, this indicates that the current gesture recognition results are not only reliable in single frames but also remain stable over time, demonstrating good image quality and model recognition performance. After classifying this data scenario as a clear scenario, the initial gesture data output by the backbone network is directly used as the final gesture data, skipping the auxiliary network correction stage. This improves data processing efficiency, saves computational resources, and enables rapid response to user actions.

[0055] In one implementation of this application, when detecting initial gesture data, the pre-set confidence threshold and preset time window are dynamically adjusted based on the accuracy of the detection results and the hand movement state in the hand video frame, to make the detection results more accurate. Specifically, the process is as follows: A hand detection box is drawn based on the hand keypoint coordinates output by the backbone network. The hand detection boxes corresponding to different chirality labels are concatenated in chronological order to obtain the hand detection box trajectories corresponding to different chirality labels. Reference trajectory data that meet preset conditions in the historical database and whose similarity difference with the hand detection box trajectory is determined; wherein the reference trajectory data includes at least one of a reference confidence threshold and a reference time window. Change data of the hand keypoints corresponding to the current hand video frame is obtained, and the reference confidence threshold and reference time window are adjusted based on the change data.

[0056] Specifically, after the backbone network outputs the coordinates of the hand keypoints, the boundary of the hand detection box is determined based on this coordinate information. That is, the maximum and minimum x-coordinates, maximum and minimum y-coordinates of all keypoints in the image coordinate system are first found. A rectangle is then constructed using these four extreme points, ensuring it completely encloses the hand keypoints. For example, if the minimum x-coordinate of a hand keypoint is 50 pixels and the maximum is 200 pixels, and the minimum y-coordinate is 80 pixels and the maximum is 300 pixels, then a rectangle with the top-left corner at (50, 80) and the bottom-right corner at (200, 300) is drawn as the hand detection box. After drawing, the detection box information is associated and stored with the corresponding hand video frame and the dominant hand label.

[0057] Secondly, hand detection boxes corresponding to different chirality labels are collected chronologically. For the same chirality label, hand detection boxes from adjacent frames are connected sequentially to form a continuous trajectory. Furthermore, if the hand detection box in a frame shifts due to slight jitter, a more reasonable intermediate position can be calculated using linear interpolation based on the positions of the detection boxes in preceding and following frames, making the trajectory more closely match the actual hand movement path. Finally, the hand detection box trajectories corresponding to different chirality labels are obtained, thus presenting the changes in the movement trajectory of the left and right hands in the video sequence.

[0058] Next, the generated hand detection bounding box trajectory is compared with trajectory data in the historical database. Algorithms such as Euclidean distance are used to determine the difference between the current trajectory and historical trajectories. When the difference between two trajectories is less than a difference threshold, they are considered similar. The reference confidence threshold and reference time window corresponding to the reference trajectory data that meet this difference threshold are extracted and used as the data basis for adjusting the current preset confidence threshold and preset time window.

[0059] Finally, for the current hand video frame, the change data of hand key points are further determined. This includes calculating the movement speed, displacement, and coordinate distribution fluctuations of the key points. For example, if the coordinate displacement of the index fingertip key point between two adjacent frames is 10 pixels, with a time interval of 0.03 seconds, its movement speed is approximately 333 pixels per second. Simultaneously, the overall changes of multiple key points are determined, forming a change dataset reflecting the dynamic characteristics of hand movements. Based on the acquired hand key point change data, the reference confidence threshold and reference time window are adjusted.

[0060] In one implementation of this application, the specific process of adjusting the reference confidence threshold is as follows: A confidence data adjustment set is constructed based on the changing data; wherein the confidence data adjustment set includes at least one of the following: the degree of occlusion of hand key points, the distinguishability between hand key points and the background, and the degree of dispersion of hand key points. A first adjustment factor is generated based on the confidence data adjustment set and the first adjustment coefficients corresponding to different data points in the confidence data adjustment set. The reference confidence threshold is adjusted according to the first adjustment factor and the confidence adjustment ratio.

[0061] Specifically, the degree of occlusion of hand keypoints is determined by calculating the proportion of occluded keypoints to the total number of keypoints. An image segmentation algorithm is used to separate the hand region from the background, and the differences in color, texture, and other features between the two are calculated to determine the distinguishability between the hand keypoints and the background. The variance of all keypoint coordinates is calculated; a larger variance indicates a higher degree of dispersion, thus determining the dispersion of hand keypoints. At least one of the above features is selected, and their values ​​are combined to form a confidence data adjustment set, which serves as the basis for adjusting the reference confidence threshold. It should be noted that the confidence data adjustment set in this embodiment may include other relevant data in addition to the above features, and this embodiment does not impose any limitations on this.

[0062] Furthermore, a corresponding first adjustment coefficient is assigned to each data item in the confidence data adjustment set. These coefficients are pre-set based on historical data training or expert experience and are used to measure the importance of each data item to the confidence threshold adjustment. For example, if the first adjustment coefficient for the occlusion degree of hand key points is 0.4, the discrimination coefficient with the background is 0.3, the dispersion coefficient is 0.3, and the currently calculated occlusion degree is 0.2, the discrimination difference is 0.6, and the dispersion variance is 0.8, then the first adjustment factor is generated by weighted summation: 0.4 × 0.2 + 0.3 × 0.6 + 0.3 × 0.8 = 0.58. Finally, the first adjustment factor is multiplied by the preset confidence adjustment ratio to obtain the specific adjustment amount. Assuming the initial reference confidence threshold is 0.8, the confidence adjustment ratio is 0.1, and the calculated first adjustment factor is 0.4, then the adjustment amount is 0.4 × 0.1 = 0.04. When the first adjustment factor is greater than 0, it indicates that the current hand state requires a higher confidence threshold to reduce false positives. The reference confidence threshold is increased by an adjustment amount, i.e., 0.8 + 0.04 = 0.84. When the first adjustment factor is less than 0, the threshold is decreased to avoid missed detections. Through this calculation and adjustment process, the reference confidence threshold is made to fit the recognition requirements of the current hand movement scene.

[0063] In one implementation of this application, the specific process of adjusting the reference time window is as follows: A time window data adjustment set is constructed based on the changing data; wherein the time window data adjustment set includes at least one of the following: hand keypoint movement speed, hand keypoint displacement, and coordinate fluctuation value corresponding to the hand keypoint. A second adjustment factor is generated based on the time window data adjustment set and the second adjustment coefficients corresponding to different data within the time window data adjustment set. The reference time window is adjusted according to the second adjustment factor and the time window adjustment ratio.

[0064] Specifically, when calculating the movement speed of hand keypoints, the coordinates of each keypoint in two adjacent frames are obtained. The displacement is calculated using the Euclidean distance formula, and then divided by the frame interval to obtain the movement speed of a single keypoint. Finally, the average speed of all keypoints is calculated. For the displacement amount, the cumulative displacement of all keypoints from their starting position to their current position within a certain time range is statistically analyzed. The coordinate fluctuation value is obtained by calculating the standard deviation of the keypoint coordinates over multiple consecutive frames; a larger standard deviation indicates more severe coordinate fluctuations.

[0065] Furthermore, a second adjustment coefficient is assigned to each data item in the time window data adjustment set. The second adjustment factor is calculated using the formula: Second Adjustment Factor = Movement Speed ​​ / Maximum Speed ​​Reference Value × Corresponding Coefficient + Displacement Amount / Maximum Displacement Reference Value × Corresponding Coefficient + Coordinate Fluctuation Value / Maximum Fluctuation Value Reference Value × Corresponding Coefficient. This factor reflects the current hand movement timing characteristics' need for time window adjustment. Finally, the second adjustment factor is multiplied by the preset time window adjustment ratio to obtain the adjustment amount. Assuming the initial reference time window value is 10 frames, the time window adjustment ratio is 0.15, and the calculated second adjustment factor is 0.5, then the adjustment amount is 0.5 × 0.15 = 0.075. When the second adjustment factor is greater than 0, it means the hand movement changes frequently or lasts for a long time, requiring an extended reference time window, i.e., 10 + 0.075 × rounded up, approximately 11 frames. When the second adjustment factor is less than 0, the time window is shortened to improve processing efficiency. Through this adjustment process, the time window length is made to match the duration and frequency of hand movements, optimizing gesture data processing performance.

[0066] S203. Based on the data scenario category, select whether to trigger the auxiliary network to output the final gesture data through the backbone network and / or auxiliary network.

[0067] In one implementation of this application, the current frame image sample in the acquired hand video data sample is horizontally flipped. Based on the temporal relationship of the hand video data samples, the adjacent frame images of the current frame image sample are horizontally flipped accordingly. Timestamp information is added to the enhanced image sample data after flipping, and the enhanced image samples are sorted according to the timestamp information to construct a hand image sample sequence; wherein the chirality labels corresponding to the image samples before and after flipping are opposite. Based on the hand video data samples and the hand image sample sequence, a pre-set neural network is trained to obtain an auxiliary network.

[0068] Specifically, the auxiliary network in this application embodiment is a lightweight convolutional neural network with minimal computational cost, specifically designed for high-precision left-hand and right-hand classification. Here, a lightweight network refers to a type of neural network model with few parameters, low computational complexity, and fast inference speed. Its design goal is to run efficiently on devices with limited computing resources while maintaining acceptable accuracy. Common lightweight networks include MobileNet and ShuffleNet.

[0069] Furthermore, when constructing the auxiliary network, after taking the current frame image sample from the hand video data sample, a horizontal flipping operation is initiated. For example, a flipping function from an image processing library can be used to flip the image around the vertical center line. During the flipping process, the positions of each pixel in the image are swapped according to the rule of horizontal symmetry. After the flipping is completed, the left and right directions of the original image are completely reversed, forming a new image sample that is a mirror image of the original image in terms of content. Based on the temporal sequence of the hand video data samples, the adjacent frames of the current frame image sample are determined. For these adjacent frames, the horizontal flipping operation is also performed to ensure that the flipping processing of all related frames is consistent. Moreover, during the flipping process, the temporal logic of the video is followed to ensure the continuity of actions and temporal consistency between adjacent frames after the flipping.

[0070] Furthermore, timestamps are added to all image sample data after horizontal flipping enhancement. The timestamps record the temporal position of each image sample in the original video sequence, ensuring the temporal order of the samples is traceable. After adding the timestamps, all enhanced image samples are sorted in ascending or descending order based on the timestamp information, constructing an ordered sequence of hand image samples. In this sequence, the image samples are arranged sequentially according to time, forming a complete and logically coherent set of image sequences. Because the image samples are horizontally flipped, their corresponding chirality also changes mirror-imagely. For example, if the original image sample's chirality label is "left," after horizontal flipping, the corresponding chirality label is updated to "right." This label correction mechanism ensures the consistency between image samples and their chirality labels, avoiding label errors caused by image flipping.

[0071] Finally, the original hand video data samples and the hand image sample sequence constructed through the above processing are used as training data and input into the pre-built neural network. During training, the pre-built neural network uses a deep learning algorithm to continuously adjust its internal parameters based on the input image samples and their corresponding hand dominance labels. By learning hand features, movement patterns, and hand dominance rules in the image samples, the pre-built neural network gradually optimizes its model structure and parameter configuration to improve its ability to recognize and process gesture data. After multiple rounds of iterative training, training stops when the performance indicators of the pre-built neural network on the training and validation sets reach the preset standards, resulting in the finally trained auxiliary network.

[0072] Figure 4 A flowchart for outputting final data is provided as an embodiment of this application, such as... Figure 4 As shown, the final data output includes the following steps: S401-S402:

[0073] S401. In the case of a scenario where the data is clear, the initial gesture data output by the backbone network shall be used as the final gesture data.

[0074] When the data scenario received by the gesture data processing decision module is a clear data scenario, the initial gesture data output by the backbone network, including the initial chirality label and initial chirality confidence, is directly marked as the final gesture data. At this time, the results output by the backbone network under clear data conditions have high accuracy and reliability, requiring no additional correction processing, thereby reducing computational resource consumption, quickly transmitting gesture data to subsequent application stages, and improving response efficiency.

[0075] S402. In the case of a scene with blurred data, the hand image corresponding to the hand video frame is input into the auxiliary network. The initial gesture data is corrected by the gesture data output by the auxiliary network to obtain the final gesture data.

[0076] If the judgment result indicates a blurred data scene, the hand image data is input into a pre-trained auxiliary network. The multi-layered neural network structure within the auxiliary network performs deep extraction and analysis of image features. Using the gesture feature patterns and rules learned during its training, the auxiliary network re-identifies and judges information such as hand posture, key point positions, and dominant hand position in the input image. During the recognition process, the network uses components such as convolutional layers and pooling layers to extract local and global features of the image, and then performs feature fusion and classification prediction through fully connected layers. Finally, it outputs high-confidence gesture data containing corrected dominant hand position labels and corrected dominant hand position confidence. For example, {"left": 0.99, "right": 0.01} indicates that the auxiliary network has corrected the original judgment.

[0077] After the auxiliary network outputs the corrected gesture data, the corrected data output by the auxiliary network overwrites the initial data. For example, the decision result "left" from the auxiliary network is adopted and used to replace the "right" label output by the backbone network in this frame.

[0078] Finally, the corrected chirality label "left" is integrated with the 3D keypoint data generated by the backbone network and packaged for output to the camera terminal.

[0079] Figure 5 This is a schematic diagram illustrating the overall process of a 3D gesture data generation method provided in an embodiment of this application. Figure 5 As shown, after receiving hand video frames from the camera terminal, the data is sent to the backbone network, which outputs initial gesture data. Next, this initial gesture data is input to the scene detection module. This module detects the hand-right confidence and the stability of the hand-right label in the initial gesture data. If the detection meets the requirements, it is determined to be a clear scene, and the initial gesture data output from the backbone network is used as the final gesture data. If the detection does not meet the requirements, it is determined to be a blurred scene. In this case, the initial gesture data is input to the auxiliary network, which uses a lightweight convolutional neural network to re-output the hand image's hand-right confidence and hand-right label to correct the initial gesture data output from the backbone network. The corrected gesture data is then output.

[0080] Figure 6 This is a schematic diagram of a 3D gesture data generation device provided in an embodiment of this application. Figure 6 As shown, the 3D gesture data generation device includes:

[0081] The data acquisition unit acquires the initial gesture data corresponding to the hand video frames output by the backbone network; wherein, the initial gesture data includes at least one of the initial hand dominance label and the initial hand dominance confidence.

[0082] The data detection unit determines the data scene category corresponding to the initial gesture data through a preset scene detection mechanism. This includes: comparing the hand-righting confidence of the current frame hand image corresponding to the hand video frame with a preset confidence threshold to obtain a confidence detection result; if the confidence detection result meets the preset confidence conditions, performing label jump detection on the hand-righting labels of multiple frames of hand images within a preset time window to obtain a label stability detection result; and determining the data scene category corresponding to the initial gesture data based on the confidence detection result and / or the label stability detection result. The data scene category includes at least one of a blurred data scene or a clear data scene.

[0083] The data output unit, based on the data scenario category, selects whether to trigger the auxiliary network to output the final gesture data through the backbone network and / or the auxiliary network.

[0084] In one implementation of this application, the data scene category corresponding to the initial gesture data is determined based on the confidence level detection result and / or the label stability detection result, specifically including:

[0085] When the confidence test result is that the confidence of the hand image in the current frame is less than the preset confidence threshold, the data scene category is determined to be a data fuzzy scene.

[0086] When the confidence test result is that the confidence of the hand image in the current frame is not less than the preset confidence threshold, and the label stability test result is that the label jumps, the data scene category is determined to be a data fuzzy scene.

[0087] If the confidence test result is that the confidence of the hand image in the current frame is not less than the preset confidence threshold, and the label stability test result is that the label has not jumped, then the data scene category is determined to be a clear data scene.

[0088] In one implementation of this application, before determining the data scene category corresponding to the initial gesture data through a pre-set scene detection mechanism, the method further includes:

[0089] Draw the hand detection box based on the coordinates of the key hand points output by the backbone network;

[0090] The hand detection boxes corresponding to different chiral labels are spliced ​​together in chronological order to obtain the hand detection box trajectories corresponding to different chiral labels.

[0091] In the historical database, reference trajectory data that meet the preset conditions for similarity difference with the hand detection box trajectory are identified; wherein, the reference trajectory data includes at least one of the reference confidence threshold and the reference time window.

[0092] Obtain the change data of key hand points corresponding to the current hand video frame;

[0093] Based on the changing data, the reference confidence threshold and the reference time window are adjusted respectively.

[0094] In one implementation of this application, the reference confidence threshold and the reference time window are adjusted based on the changing data, specifically including:

[0095] Based on the change data, a confidence data adjustment set is constructed; wherein the confidence data adjustment set includes at least one of the following: the degree of occlusion of hand key points, the distinction between hand key points and background, and the degree of dispersion of hand key points;

[0096] A first adjustment factor is generated based on the confidence data adjustment set and the first adjustment coefficients corresponding to different data in the confidence data adjustment set.

[0097] The reference confidence threshold is adjusted based on the first adjustment factor and the confidence adjustment ratio.

[0098] Based on the changing data, a time window data adjustment set is constructed; wherein, the time window data adjustment set includes at least one of the following: hand key point movement speed, hand key point displacement, and the coordinate fluctuation value corresponding to the hand key point;

[0099] A second adjustment factor is generated based on the time window data adjustment set and the second adjustment coefficients corresponding to different data in the time window data adjustment set.

[0100] The reference time window is adjusted based on the second adjustment factor and the time window adjustment ratio.

[0101] In one implementation of this application, before selecting whether to trigger the auxiliary network based on the data scenario category, the method further includes:

[0102] The current frame image sample in the acquired hand video data sample is horizontally flipped;

[0103] Based on the temporal relationship of hand video data samples, the adjacent frame images of the current frame image sample are subjected to corresponding horizontal flipping processing.

[0104] Timestamp information is added to the enhanced image sample data after flipping, and the enhanced image samples are sorted according to the timestamp information to construct a hand image sample sequence; wherein, the chirality labels of the image samples before and after flipping are opposite;

[0105] An auxiliary network is obtained by training a pre-set neural network based on hand video data samples and hand image sample sequences.

[0106] In one implementation of this application, based on the data scenario type, it is selected whether to trigger the auxiliary network, so as to output the final gesture data through the auxiliary network and / or the backbone network, specifically including:

[0107] In scenarios where the data is clear, the initial gesture data output by the backbone network is used as the final gesture data.

[0108] In cases where the data is ambiguous, the hand image corresponding to the hand video frame is input into the auxiliary network, and the auxiliary network outputs the final gesture data. The gesture data output by the auxiliary network includes at least one of the corrected dominant label and the corrected dominant confidence.

[0109] In one implementation of this application, after outputting the final gesture data through the auxiliary network, the method further includes:

[0110] The initial gesture data is corrected using the final gesture data;

[0111] The corrected gesture data and the hand key point data output by the backbone network are integrated, and the integrated data is sent to the camera terminal.

[0112] Figure 7 This is a schematic diagram of the structure of a 3D gesture data generation device provided in an embodiment of this application. Figure 7 As shown, a 3D gesture data generation device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-described 3D gesture data generation methods.

[0113] This application also provides a computer program product, which, when run, can perform the steps of the 3D gesture data generation method as described in any of the preceding embodiments.

[0114] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0115] The above descriptions are merely embodiments of this application and are not intended to limit the scope of this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions in the embodiments of this application.

Claims

1. A method for generating 3D gesture data, characterized in that, The method includes: Obtain the initial gesture data corresponding to the hand video frames output by the backbone network; wherein, the initial gesture data includes at least one of the initial hand supremacy label and the initial hand supremacy confidence. The data scene category corresponding to the initial gesture data is determined through a preset scene detection mechanism, including: comparing the hand-righting confidence of the current frame hand image corresponding to the hand video frame with a preset confidence threshold to obtain a confidence detection result; if the confidence detection result meets the preset confidence conditions, performing label jump detection on the hand-righting labels of multiple frames of hand images within a preset time window to obtain a label stability detection result; and determining the data scene category corresponding to the initial gesture data based on the confidence detection result and / or the label stability detection result; wherein the data scene category includes at least one of data blurry scene and data clear scene. Based on the data scenario category, select whether to trigger the auxiliary network to output the final gesture data through the backbone network and / or the auxiliary network; The step of determining the data scene category corresponding to the initial gesture data based on the confidence level detection result and / or the label stability detection result specifically includes: When the confidence detection result is that the confidence of the hand image in the current frame is less than the preset confidence threshold, the data scene category is determined to be a data fuzzy scene. When the confidence detection result is that the confidence of the hand image in the current frame is not less than the preset confidence threshold, and the label stability detection result is that the label jumps, the data scene category is determined to be a data fuzzy scene. When the confidence detection result is that the confidence of the hand image in the current frame is not less than the preset confidence threshold, and the label stability detection result is that the label has not jumped, the data scene category is determined to be a clear data scene. The step of selecting whether to trigger the auxiliary network based on the data scenario type, so as to output the final gesture data through the auxiliary network and / or the backbone network, specifically includes: In the case where the data is determined to be clear, the initial gesture data output by the backbone network is used as the final gesture data; In the case of a data ambiguity scenario, the hand image corresponding to the hand video frame is input into the auxiliary network, and the final gesture data is output through the auxiliary network; wherein, the gesture data output by the auxiliary network includes at least one of corrected dominant label and corrected dominant confidence.

2. The method according to claim 1, characterized in that, Before determining the data scene category corresponding to the initial gesture data through a preset scene detection mechanism, the method further includes: Based on the coordinates of the key hand points output by the backbone network, draw the hand detection box; The hand detection boxes corresponding to different chiral labels are spliced ​​together in chronological order to obtain the hand detection box trajectories corresponding to different chiral labels. In the historical database, reference trajectory data that meet the preset conditions in terms of similarity difference with the trajectory of the hand detection box are identified; wherein, the reference trajectory data includes at least one of a reference confidence threshold and a reference time window. Obtain the change data of key hand points corresponding to the current hand video frame; Based on the changed data, the reference confidence threshold and the reference time window are adjusted respectively.

3. The method according to claim 2, characterized in that, The adjustment of the reference confidence threshold and the reference time window based on the changed data specifically includes: Based on the aforementioned change data, a confidence data adjustment set is constructed; wherein, the confidence data adjustment set includes at least one of the following: the degree of occlusion of hand key points, the distinguishability of hand key points from the background, and the degree of dispersion of hand key points; Based on the confidence data adjustment set and the first adjustment coefficients corresponding to different data in the confidence data adjustment set, a first adjustment factor is generated; The reference confidence threshold is adjusted according to the first adjustment factor and the confidence adjustment ratio; Based on the aforementioned change data, a time window data adjustment set is constructed; wherein, the time window data adjustment set includes at least one of the following: hand key point movement speed, hand key point displacement, and coordinate fluctuation value corresponding to the hand key point; A second adjustment factor is generated based on the time window data adjustment set and the second adjustment coefficients corresponding to different data in the time window data adjustment set. The reference time window is adjusted according to the second adjustment factor and the time window adjustment ratio.

4. The method according to claim 1, characterized in that, Before selecting whether to trigger the auxiliary network based on the data scenario category, the method further includes: The current frame image sample in the acquired hand video data sample is horizontally flipped; Based on the temporal relationship of the hand video data samples, the adjacent frame images of the current frame image sample are subjected to corresponding horizontal flipping processing; Timestamp information is added to the enhanced image sample data after flipping, and the enhanced image samples are sorted according to the timestamp information to construct a hand image sample sequence; wherein, the chirality labels corresponding to the image samples before and after flipping are opposite; The auxiliary network is obtained by training a pre-set neural network based on the hand video data samples and the hand image sample sequence.

5. The method according to claim 4, characterized in that, After outputting the final gesture data through the auxiliary network, the method further includes: The initial gesture data is corrected using the final gesture data; The corrected gesture data and the hand key point data output by the backbone network are integrated, and the integrated data is sent to the camera terminal.

6. A 3D gesture data generation device, characterized in that, The device includes: The data acquisition unit acquires the initial gesture data corresponding to the hand video frames output by the backbone network; wherein, the initial gesture data includes at least one of the initial hand dominance label and the initial hand dominance confidence. The data detection unit determines the data scene category corresponding to the initial gesture data through a preset scene detection mechanism, including: comparing the hand-righting confidence of the current frame hand image corresponding to the hand video frame with a preset confidence threshold to obtain a confidence detection result; if the confidence detection result meets the preset confidence conditions, performing label jump detection on the hand-righting labels of multiple frames of hand images within a preset time window to obtain a label stability detection result; determining the data scene category corresponding to the initial gesture data based on the confidence detection result and / or the label stability detection result; wherein the data scene category includes at least one of data blurry scene and data clear scene; the determination of the data scene category corresponding to the initial gesture data based on the confidence detection result and / or the label stability detection result specifically includes: when the confidence detection result is that the hand-righting confidence of the current frame hand image is less than the preset confidence threshold, determining the data scene category as a data blurry scene; ... The data scene category is determined to be a data fuzzy scene when the confidence level of the current frame hand image is not less than a preset confidence threshold and the label stability detection result is that the label jumps; when the confidence level detection result is that the confidence level of the current frame hand image is not less than a preset confidence threshold and the label stability detection result is that the label does not jump; the data scene category is determined to be a data clear scene. Based on the data scene type, the step of selecting whether to trigger an auxiliary network to output final gesture data through the auxiliary network and / or the backbone network specifically includes: when the data scene is determined to be clear, using the initial gesture data output by the backbone network as the final gesture data; when the data scene is determined to be fuzzy, inputting the hand image corresponding to the hand video frame into the auxiliary network, and outputting the final gesture data through the auxiliary network; wherein the gesture data output by the auxiliary network includes at least one of corrected hand advantage label and corrected hand advantage confidence. The data output unit, based on the data scenario category, selects whether to trigger the auxiliary network to output the final gesture data through the backbone network and / or the auxiliary network.

7. A 3D gesture data generation device, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to: perform the 3D gesture data generation method according to any one of claims 1-5.

8. A computer program product, characterized in that, Running the computer program product can perform the steps of the 3D gesture data generation method as described in any one of claims 1 to 5.

Images