Gesture recognition rotation locking method, device, equipment and storage medium
By acquiring hand videos and calculating 3D key points of the hand, and locking finger joints to generate a virtual hand, the problem of interaction interruption caused by hand occlusion is solved, and the stability of gesture recognition and the flexibility of the system are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU SHIYUAN ELECTRONICS CO LTD
- Filing Date
- 2023-03-06
- Publication Date
- 2026-08-04
AI Technical Summary
Existing gesture recognition methods cannot accurately identify key hand points when the hand is severely obscured, leading to interruptions in user interaction.
By acquiring hand videos, extracting 2D key points and projecting them into 3D space, calculating the rotation of finger joints and wrist, determining whether the wrist is in a rotating position, locking finger joints, and generating a virtual hand to maintain interaction.
It effectively solves the problem that the hand cannot maintain stability and interact with objects during rotation, and improves the flexibility of modification and computational efficiency of the gesture recognition system.
Smart Images
Figure CN118609198B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to a rotation locking method, apparatus, device and storage medium for gesture recognition. Background Technology
[0002] When extracting hand key points in AR / VR gesture recognition, significant occlusion can occur due to changes in hand movements, leading to inaccurate or unpredictable predictions of hand key points by the neural network, thus impacting user experience. When users interact with virtual objects in the scene, such as picking up, rotating, or placing objects, inaccurate key point predictions often prevent interaction, or even cause the AR / VR virtual hand to disappear, frequently interrupting the user's interaction with the object.
[0003] Therefore, existing gesture recognition methods cannot accurately identify key points of the hand when the hand is severely obscured, leading to gesture recognition anomalies. Summary of the Invention
[0004] The main objective of this application is to provide a rotation locking method, apparatus, device, and storage medium for gesture recognition, aiming to solve the technical problem that the prior art cannot accurately identify key points of the hand when the hand is severely obscured.
[0005] To achieve the aforementioned objectives, this application proposes a rotation locking method based on gesture recognition, the method comprising:
[0006] Obtain a video of your hand;
[0007] Extract the key points of the hand in each frame of the hand video to obtain the 2D key points of the hand;
[0008] The 2D key points of the hand are reprojected into 3D space to calculate the 3D key points of the hand;
[0009] Based on the aforementioned 3D key points of the hand, calculate the rotation amount of each finger joint and the wrist rotation amount;
[0010] Based on the wrist rotation amount, determine whether the wrist is in a rotating position in the current frame;
[0011] If the wrist is in a rotating position in the current frame, obtain the rotation amount of the finger joints in the historical frames and lock the finger joints;
[0012] The wrist rotation amount and the finger joint rotation amount from the historical frame are applied to the template hand to obtain a virtual hand.
[0013] Further, the step of determining whether the wrist is in a rotating position in the current frame based on the wrist rotation amount includes:
[0014] Obtain the set camera coordinate system, left wrist coordinate system, and right wrist coordinate system;
[0015] Based on the wrist rotation amount, calculate the rotation judgment coefficients of the left and right wrists relative to the camera coordinate system;
[0016] Based on the rotation judgment coefficient, determine whether the left and right wrists in the current frame are in a rotating position.
[0017] Furthermore, the step of obtaining the finger joint rotation amount from historical frames and locking the finger joint if the wrist is in a rotated position in the current frame includes:
[0018] Get the distance between the tip of the thumb and the tip of the index finger in the current frame;
[0019] When the distance is less than the first preset distance, it is determined that the thumb and index finger are pinched together, the finger joint unlock count is reset to zero, and the five finger joints are locked.
[0020] Further, after the step of obtaining the distance between the tip of the thumb and the tip of the index finger in the current frame, the following steps are included:
[0021] When the distance is greater than a first preset distance and the distance is less than a second preset distance, obtain the historical distance between the thumb tip and the index finger tip corresponding to a first preset number of frames before the current frame, wherein the second preset distance is greater than the first preset distance;
[0022] Determine whether the historical distance is less than the first preset distance;
[0023] If the historical distance is less than the first preset distance, then count the number of times the historical distance is less than the first preset distance;
[0024] Determine whether the number of times is equal to or greater than a first preset number of times;
[0025] If the number of times is equal to or greater than the first preset number of times, the finger joint unlock count is reset to zero, and the five finger joints are locked.
[0026] If the number of attempts is less than the first preset number of attempts, the finger joint unlock count is reset to zero, and three finger joints are locked, wherein the three finger joints are the joints of the middle finger, ring finger, and little finger.
[0027] Further, after the step of obtaining the distance between the tip of the thumb and the tip of the index finger in the current frame, the following steps are included:
[0028] When the distance is greater than the second preset distance, it is determined that the thumb and index finger are separated, and the unlock count of the finger joint is incremented by one.
[0029] Further, after the step of determining whether the wrist is in a rotating position in the current frame based on the wrist rotation amount, the method includes:
[0030] Determine whether the finger joints were locked in the previous frame of the current frame;
[0031] If the finger joints were locked in the previous frame and the wrist is in a non-rotating position in the current frame, then the finger joint unlock count is incremented by one.
[0032] Furthermore, after the step of incrementing the finger joint unlock count, the following steps are included:
[0033] Determine whether the finger joint unlock count is greater than or equal to the first preset number;
[0034] If the finger joint unlock count is greater than or equal to the first preset number, then the finger joint lock is released;
[0035] If the unlock count of the finger joint is less than the first preset number, then the three finger joints are locked.
[0036] This application also provides a gesture recognition rotation locking device, the device comprising:
[0037] The acquisition module is used to acquire hand videos;
[0038] The 2D key point extraction module is used to extract the key points of the hand in each frame of the hand video to obtain the 2D key points of the hand.
[0039] The 3D key point calculation module is used to reproject the 2D key points of the hand into 3D space and calculate the 3D key points of the hand.
[0040] The rotation calculation module is used to calculate the rotation of each finger joint and the rotation of the wrist based on the 3D key points of the hand.
[0041] The rotation determination module is used to determine whether the wrist is in a rotating position in the current frame based on the amount of wrist rotation.
[0042] The rotation locking module is used to lock the finger joints if the wrist is in a rotating position in the current frame by obtaining the rotation amount of the finger joints in the historical frames.
[0043] The virtual hand generation module is used to apply the wrist rotation amount and the historical frame finger joint rotation amount to the template hand to obtain a virtual hand.
[0044] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.
[0045] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.
[0046] This application provides a rotation locking method for gesture recognition. By employing a rotation locking logic, it effectively solves the problems of hand instability and the inability to maintain interaction with objects during rotation. Rotation locking allows for flexible decision-making regarding when and what historical data to use, rather than rigidly relying on past frames. Unlike filtering, which requires continuous operation, rotation locking only intervenes when the wrist is detected to be rotating and the hand is interacting with a virtual object. Therefore, in most cases, only a very small amount of computational power is needed to determine whether locking is necessary. Furthermore, as an independent module, rotation locking does not affect the operation and updates of other modules. Even if other modules are optimized or replaced later, the system can quickly adapt to the new modules without modifying the interfaces, improving the overall flexibility of the gesture recognition system. Attached Figure Description
[0047] Figure 1 This is a schematic flowchart of a gesture recognition rotation locking method according to an embodiment of this application;
[0048] Figure 2 This is a schematic block diagram of the structure of a gesture recognition rotation locking device according to an embodiment of this application;
[0049] Figure 3 This is a schematic diagram of 2D key points of the hand for gesture recognition according to an embodiment of this application;
[0050] Figure 4 This is a schematic diagram of finger joints for gesture recognition according to an embodiment of this application;
[0051] Figure 5 This is a schematic diagram of the camera coordinate system for gesture recognition according to an embodiment of this application;
[0052] Figure 6 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.
[0053] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0055] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.
[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0057] Reference Figure 1 This invention provides a rotation locking method for gesture recognition, including steps S1-S7, specifically:
[0058] S1. Obtain hand video;
[0059] S2. Extract the key points of the hand in each frame of the hand video to obtain the 2D key points of the hand;
[0060] Hand videos record the process of hand movements. Key points of the hand are extracted from each frame of the hand video image to obtain 2D hand key points. Extraction of 2D hand key points can be achieved using a neural network model. In this application, each frame of the hand video image is input into a neural network model, which outputs 21 key point heatmaps, such as... Figure 3 As shown, the positions of 21 2D keypoints of the hand were obtained through heatmaps, with each number representing a 2D keypoint of the hand.
[0061] S3. Reproject the 2D key points of the hand into 3D space and calculate the 3D key points of the hand;
[0062] Epipolar geometry triangulation reprojects 2D keypoints into 3D space. To achieve this, two or more cameras are typically needed to photograph the same object. For example... Figure 5As shown, in this application, two cameras, one for the left eye and one for the right eye, are used to capture video of the hand.
[0063] S4. Calculate the rotation of each finger joint and the rotation of the wrist based on the 3D key points of the hand.
[0064] Calculating the rotation of each finger joint and wrist by using 3D keypoints of the hand can be achieved using various traditional optimization algorithms, such as Gaussian-Newtown, Levenberg-Marquardt (LM), and Stochastic Gradient Descend (SGD). These algorithms share a common feature: they require a custom loss function, which is then minimized through optimization. The LM optimization algorithm can be used, where L1 loss is defined as the triangulated 3D coordinates of the 21 keypoints, and L2 loss is the application of the rotation to the coordinates of the 21 keypoints after the template is created. Minimizing the loss yields the optimal finger joint and wrist rotation. The template hand can be a custom template hand or the classic Mano hand; this application uses a custom template hand. In this application, as... Figure 4 As shown, a total of 21 rotational quantities were calculated, each including the rotation of 20 finger joints and the rotation of the wrist joint. The rotation of the 20 finger joints was expressed using the Rodrigues rotation formula, and the wrist movement was represented by a 3*4 rotation-translation matrix.
[0065] S5. Based on the wrist rotation amount, determine whether the wrist is in a rotating position in the current frame;
[0066] Based on the amount of wrist rotation, and the positional relationship between the camera coordinates used to capture the hand video and the coordinates of the left and right hands, it can be determined whether the wrist is in a rotating position. In this application, the left-eye camera coordinates are used as a reference, such as... Figure 5 As shown. The left wrist coordinate system is defined as follows: the thumb is perpendicular to the palm, with the thumb direction being the positive Z direction; the palm is extended, with the four fingers pointing in the positive X direction; and the direction perpendicular to the back of the hand is the positive Y direction. The right wrist coordinate system is defined as follows: the thumb is perpendicular to the palm, with the thumb direction being the positive Z direction; the palm is extended, with the four fingers pointing in the negative X direction; and the direction perpendicular to the back of the hand is the positive Y direction.
[0067] S6. If the wrist is in a rotating position in the current frame, obtain the rotation amount of the finger joints in the historical frames and lock the finger joints.
[0068] S7. Apply the wrist rotation amount and the historical frame finger joint rotation amount to the template hand to obtain a virtual hand.
[0069] When wrist rotation is detected, the finger joint rotation values from historical frames are used to lock the finger joints at that moment. The specific historical frame's finger joint rotation value used can be set according to the actual situation. If there are more than one historical frame before the current frame, the finger joint rotation value can also be locked as the average of the rotation values of the corresponding joints in all historical frames. When locking finger joints, the distance between the thumb tip and index finger tip in the current frame determines whether to lock three or five finger joints. This method ensures that even if hand rotation causes partial occlusion and the neural network's judgment is inaccurate when extracting 2D key points, the fingers will not be released, thus preventing the interruption of interaction with the object. After obtaining the wrist rotation and finger joint rotation values, these values are applied to the template hand to re-observe the 3D coordinates of each key point, generating a virtual hand. Finally, the virtual hand, including finger and wrist movements, is displayed on the screen of an AR / VR device.
[0070] As described above, the rotation-locking logic effectively solves the problem of hand instability and interaction with objects during rotation. Rotation locking allows for flexible decision-making regarding when and what historical data to use, rather than relying on static data from past frames. Unlike filtering, which requires continuous operation, rotation locking only intervenes when the wrist is detected to be rotating and the hand is interacting with the virtual object. Therefore, in most cases, only a very small amount of computational power is needed to determine whether locking is necessary. Furthermore, as an independent module, rotation locking does not affect the operation and updates of other modules. Even if other modules are optimized or replaced later, the system can quickly adapt to the new modules without modifying the interface, improving the overall flexibility of the gesture recognition system.
[0071] In one embodiment, the step of determining whether the wrist is in a rotating position in the current frame based on the wrist rotation amount includes:
[0072] S501. Obtain the set camera coordinate system, left wrist coordinate system, and right wrist coordinate system;
[0073] S502. Based on the wrist rotation amount, calculate the rotation judgment coefficients of the left and right wrists relative to the camera coordinate system;
[0074] S503. Based on the rotation judgment coefficient, determine whether the left and right wrists in the current frame are in a rotating position.
[0075] The wrist rotation is represented by a 3x3 rotation matrix R, and the rotation judgment coefficient P is calculated:
[0076] P = <1, 0, 0><0, 0, 1> T
[0077] All multiplications involved in the rotation determination coefficient formula are matrix multiplications. <1, 0, 0> is a 1×3 matrix, R is a 3×3 matrix, and <0, 0, 1> T is a 3×1 matrix. The rotation determination coefficient P is a scalar.
[0078] For the left hand, when 0.5 < P < 0.55, the left hand wrist is in the rotation position; for the right hand, when -0.65 < P < 0.4, the right hand wrist is in the rotation position.
[0079] In one embodiment, the step of obtaining the historical finger joint rotation amount of the historical frame and locking the finger joints if the wrist of the current frame is in the rotation position includes:
[0080] S601. Obtain the distance between the thumb tip and the index finger tip of the current frame;
[0081] S602. When the distance is less than the first preset distance, determine that the thumb and the index finger are pinched, clear the unlocking count of the finger joints, and lock the five finger joints.
[0082] The set first preset distance is the boundary condition for the interaction between the hand and the virtual object. When the distance between the thumb tip and the index finger tip is less than the first preset distance, it is determined that the thumb and the index finger are in the pinched state. At this time, the virtual object can be grabbed by the hand and interact with the hand. Obtain the historical finger joint rotation amount of the five fingers of the historical frame, and lock the five finger joints according to the historical finger joint rotation amount of the five fingers of the historical frame. Setting the unlocking count is to prevent frequent and incorrect unlocking due to the error of the neural network. The first preset distance can be set according to the actual situation, for example, set to 6.5 cm.
[0083] In one embodiment, after the step of obtaining the distance between the thumb tip and the index finger tip of the current frame, it includes:
[0084] S603. When the distance is greater than the first preset distance and less than the second preset distance, obtain the historical distances between the thumb tip and the index finger tip corresponding to the first preset number of frames before the current frame, where the second preset distance is greater than the first preset distance;
[0085] S604. Determine whether the historical distance is less than the first preset distance;
[0086] S605. If the historical distance is less than the first preset distance, calculate the number of times the historical distance is less than the first preset distance;
[0087] S606. Determine whether the number of times is equal to or greater than the first preset number of times;
[0088] S607. If the number of times is equal to or greater than the first preset number of times, clear the unlocking count of the finger joints and lock the five finger joints;
[0089] S608. If the number of times is less than the first preset number of times, the finger joint unlock count is reset to zero, and three finger joints are locked, wherein the three finger joints are the joints of the middle finger, ring finger, and little finger.
[0090] The first preset number, the first preset number of times, and the second preset distance can be set according to actual conditions. The first preset number is greater than the first preset number of times, and the second preset distance is greater than the first preset distance. The set second preset distance is the boundary condition for hand-virtual object interaction. The first preset number is the number of frames, and one frame includes a corresponding historical distance between the tip of the thumb and the tip of the index finger. The first preset number of times is the unlock count. In this embodiment, when the number of times the historical distance is less than the first preset distance is greater than the first preset number of times, it is determined to reduce finger joint locking. For example, the first preset number is 4, the first preset number of times is 3, and the second preset distance is 7.5 cm. For example, when the distance between the thumb tip and index finger tip is greater than 6.5 cm and less than 7.5 cm, the historical distances between the thumb tip and index finger tip in the four frames preceding the current frame are retrieved. If the historical distance in any one frame is less than 6.5 cm, the number of times the historical distance is less than 6.5 cm is counted. If there are three or four instances (i.e., three or four frames), the finger joint unlock count is reset to zero, and the rotation amount of the five-finger historical frames is retrieved. The five-finger joints are locked based on the rotation amount of the five-finger historical frames. If there are one or two instances (i.e., one or two frames), the rotation amount of the three-finger historical frames is retrieved. The three-finger joints are locked based on the rotation amount of the three-finger historical frames, where the three-finger joints are the joints of the middle, ring, and little fingers. When the distance between the thumb tip and index finger tip in the current frame is greater than a first preset distance and less than a second preset distance, the neural network is more likely to misjudge the distance between the thumb tip and index finger tip. This method can prevent false locking due to misjudgment by the neural network.
[0091] In one embodiment, after the step of obtaining the distance between the tip of the thumb and the tip of the index finger in the current frame, the following steps are included:
[0092] S609. When the distance is greater than the second preset distance, it is determined that the thumb and index finger are separated, and the finger joint unlock count is incremented by one.
[0093] The second preset distance is set as the boundary condition for interaction between the hand and the virtual object. This second preset distance can be set according to actual conditions, for example, 7.5 cm. When the distance between the tip of the thumb and the tip of the index finger is greater than the second preset distance, it is determined that the thumb and index finger are separated in the current frame, and the finger joint unlock count is incremented. In this embodiment, the finger joint unlock count must be greater than or equal to the first preset number before the finger joint can be unlocked, which can prevent misjudgment of the distance between the tip of the thumb and the index finger due to neural network errors leading to false unlocks.
[0094] In one embodiment, after the step of determining whether the wrist is in a rotating position in the current frame based on the wrist rotation amount, the following is included:
[0095] S504. Determine whether the finger joints were locked in the previous frame of the current frame.
[0096] S505. If the finger joints were locked in the previous frame of the current frame, and the wrist is in a non-rotating position in the current frame, then the finger joint unlock count is incremented by one.
[0097] When the wrist changes from a rotated position to a non-rotated position in the current frame, the neural network determines that it can accurately extract the 2D key points of the hand in the non-rotated position, and therefore increments the finger joint unlock count. In this embodiment, the finger joint unlock count must be greater than or equal to a first preset number before the finger joint can be unlocked, which can prevent false locking due to misjudgment of wrist position.
[0098] In one embodiment, after the step of incrementing the finger joint unlock count by one, the following is included:
[0099] S506. Determine whether the finger joint unlock count is greater than or equal to the first preset number;
[0100] S507. If the finger joint unlock count is greater than or equal to the first preset number, then the finger joint lock is released.
[0101] S508. If the unlock count of the finger joint is less than the first preset number, then the three finger joints are locked.
[0102] Finger joint unlock count must be greater than or equal to a first preset number before a finger joint can be unlocked. This prevents false locking caused by misjudgment in the current frame. If the finger joint unlock count is less than the first preset number, the historical frame finger joint rotation values of the middle, ring, and little fingers are retrieved. Based on these historical frame finger joint rotation values, the joints of the middle, ring, and little fingers are locked to prevent false locking that could cause the fingers to release and interrupt interaction with virtual objects.
[0103] Reference Figure 2 This is a structural block diagram of a gesture recognition rotation locking device according to an embodiment of this application. The device includes:
[0104] Module 100 is used to acquire hand videos;
[0105] The 2D key point extraction module 200 is used to extract the key points of the hand in each frame of the hand video to obtain the 2D key points of the hand.
[0106] The 3D key point calculation module 300 is used to reproject the 2D key points of the hand into 3D space and calculate the 3D key points of the hand.
[0107] The rotation calculation module 400 is used to calculate the rotation of each finger joint and the rotation of the wrist based on the 3D key points of the hand.
[0108] The rotation determination module 500 is used to determine whether the wrist is in a rotating position in the current frame based on the amount of wrist rotation.
[0109] The rotation locking module 600 is used to lock the finger joints by obtaining the rotation amount of the finger joints in the historical frames if the wrist is in a rotating position in the current frame.
[0110] The virtual hand generation module 700 is used to apply the wrist rotation amount and the historical frame finger joint rotation amount to the template hand to obtain a virtual hand.
[0111] In one embodiment, the gesture recognition rotation locking device further includes:
[0112] The rotation determination submodule is used to obtain the set camera coordinate system, left wrist coordinate system, and right wrist coordinate system; calculate the rotation determination coefficients of the left and right wrists relative to the camera coordinate system based on the wrist rotation amount; and determine whether the left and right wrists are in a rotating position in the current frame based on the rotation determination coefficients.
[0113] In one embodiment, the gesture recognition rotation locking device further includes:
[0114] The first locking module is used to obtain the distance between the tip of the thumb and the tip of the index finger in the current frame; when the distance is less than the first preset distance, it is determined that the thumb and index finger are pinched together, the finger joint unlock count is cleared to zero, and the five finger joints are locked.
[0115] In one embodiment, the gesture recognition rotation locking device further includes:
[0116] The second locking module is used to: obtain the historical distance between the thumb tip and index finger tip corresponding to a first preset number of frames prior to the current frame when the distance is greater than a first preset distance and the distance is less than a second preset distance; determine whether the historical distance is less than the first preset distance; if the historical distance is less than the first preset distance, calculate the number of times the historical distance is less than the first preset distance; determine whether the number is equal to or greater than a first preset number; if the number is equal to or greater than the first preset number, reset the finger joint unlock count to zero and lock five finger joints; if the number is less than the first preset number, reset the finger joint unlock count to zero and lock three finger joints, wherein the three finger joints are the joints of the middle finger, ring finger, and little finger.
[0117] In one embodiment, the gesture recognition rotation locking device further includes:
[0118] The first unlocking counting module is used to determine that the thumb and index finger are separated when the distance is greater than the second preset distance, and to increment the unlocking count of the finger joint by one.
[0119] In one embodiment, the gesture recognition rotation locking device further includes:
[0120] The second unlocking counting module is used to determine whether the finger joints were locked in the previous frame of the current frame; if the finger joints were locked in the previous frame of the current frame, and the wrist is in a non-rotating position in the current frame, then the finger joint unlocking count is incremented by one.
[0121] In one embodiment, the gesture recognition rotation locking device further includes:
[0122] The unlocking module is used to determine whether the unlock count of the finger joint is greater than or equal to the first preset number; if the unlock count of the finger joint is greater than or equal to the first preset number, the finger joint is unlocked; if the unlock count of the finger joint is less than the first preset number, the three finger joints are locked.
[0123] Reference Figure 6 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores usage data such as those related to the rotation locking method for gesture recognition. The network interface allows communication with external terminals via a network connection. Furthermore, the computer device may also include input devices and a display screen. When the aforementioned computer program is executed by a processor to implement a flipped image text recognition method, it includes the following steps: acquiring a hand video; extracting key points of the hand in each frame of the hand video to obtain 2D key points of the hand; reprojecting the 2D key points of the hand into 3D space to calculate 3D key points of the hand; calculating the rotation amount of each finger joint and the rotation amount of the wrist based on the 3D key points of the hand; determining whether the wrist is in a rotating position in the current frame based on the wrist rotation amount; if the wrist is in a rotating position in the current frame, acquiring the finger joint rotation amounts of historical frames and locking the finger joints; applying the wrist rotation amount and the finger joint rotation amounts of historical frames to a template hand to obtain a virtual hand.
[0124] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.
[0125] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements a rotation locking method for gesture recognition, including the following steps: acquiring a hand video; extracting key points of the hand in each frame of the hand video to obtain 2D key points of the hand; reprojecting the 2D key points of the hand into 3D space to calculate 3D key points of the hand; calculating the rotation amount of each finger joint and the rotation amount of the wrist based on the 3D key points of the hand; determining whether the wrist is in a rotating position in the current frame based on the wrist rotation amount; if the wrist is in a rotating position in the current frame, acquiring the finger joint rotation amounts of historical frames and locking the finger joints; applying the wrist rotation amount and the finger joint rotation amounts of historical frames to a template hand to obtain a virtual hand.
[0126] It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.
[0127] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAM bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0128] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.
[0129] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A rotation locking method for gesture recognition, characterized in that, The method includes: Obtain a video of your hand; Extract the key points of the hand in each frame of the hand video to obtain the 2D key points of the hand; The 2D key points of the hand are reprojected into 3D space to calculate the 3D key points of the hand; Based on the aforementioned 3D key points of the hand, calculate the rotation amount of each finger joint and the wrist rotation amount; Based on the wrist rotation amount, determine whether the wrist is in a rotating position in the current frame; If the wrist is in a rotating position in the current frame, obtain the rotation amount of the finger joints in the historical frames and lock the finger joints; The wrist rotation amount and the finger joint rotation amount from the historical frame are applied to the template hand to obtain a virtual hand.
2. The rotation locking method for gesture recognition according to claim 1, characterized in that, The step of determining whether the wrist is in a rotating position in the current frame based on the wrist rotation amount includes: Obtain the set camera coordinate system, left wrist coordinate system, and right wrist coordinate system; Based on the wrist rotation amount, calculate the rotation judgment coefficients of the left and right wrists relative to the camera coordinate system; Based on the rotation judgment coefficient, determine whether the left and right wrists in the current frame are in a rotating position.
3. The rotation locking method for gesture recognition according to claim 1, characterized in that, The step of obtaining the finger joint rotation amount from historical frames and locking the finger joint if the wrist is in a rotated position in the current frame includes: Get the distance between the tip of the thumb and the tip of the index finger in the current frame; When the distance is less than the first preset distance, it is determined that the thumb and index finger are pinched together, the finger joint unlock count is reset to zero, and the five finger joints are locked.
4. The rotation locking method for gesture recognition according to claim 3, characterized in that, After the step of obtaining the distance between the tip of the thumb and the tip of the index finger in the current frame, the following steps are included: When the distance is greater than a first preset distance and the distance is less than a second preset distance, obtain the historical distance between the thumb tip and the index finger tip corresponding to a first preset number of frames before the current frame, wherein the second preset distance is greater than the first preset distance; Determine whether the historical distance is less than the first preset distance; If the historical distance is less than the first preset distance, then count the number of times the historical distance is less than the first preset distance; Determine whether the number of times is equal to or greater than a first preset number of times; If the number of times is equal to or greater than the first preset number of times, the finger joint unlock count is reset to zero, and the five finger joints are locked. If the number of attempts is less than the first preset number of attempts, the finger joint unlock count is reset to zero, and three finger joints are locked, wherein the three finger joints are the joints of the middle finger, ring finger, and little finger.
5. The rotation locking method for gesture recognition according to claim 4, characterized in that, After the step of obtaining the distance between the tip of the thumb and the tip of the index finger in the current frame, the following steps are included: When the distance is greater than the second preset distance, it is determined that the thumb and index finger are separated, and the unlock count of the finger joint is incremented by one.
6. The rotation locking method for gesture recognition according to claim 4, characterized in that, After the step of determining whether the wrist is in a rotating position in the current frame based on the wrist rotation amount, the following steps are included: Determine whether the finger joints were locked in the previous frame of the current frame; If the finger joints were locked in the previous frame and the wrist is in a non-rotating position in the current frame, then the finger joint unlock count is incremented by one.
7. The rotation locking method for gesture recognition according to claim 5 or 6, characterized in that, After the step of incrementing the finger joint unlock counter, the following steps are included: Determine whether the finger joint unlock count is greater than or equal to the first preset number; If the finger joint unlock count is greater than or equal to the first preset number, then the finger joint lock is released; If the unlock count of the finger joint is less than the first preset number, then the three finger joints are locked.
8. A gesture recognition rotation locking device, characterized in that, The device includes: The acquisition module is used to acquire hand videos; The 2D key point extraction module is used to extract the key points of the hand in each frame of the hand video to obtain the 2D key points of the hand. The 3D key point calculation module is used to reproject the 2D key points of the hand into 3D space and calculate the 3D key points of the hand. The rotation calculation module is used to calculate the rotation of each finger joint and the rotation of the wrist based on the 3D key points of the hand. The rotation determination module is used to determine whether the wrist is in a rotating position in the current frame based on the amount of wrist rotation. The rotation locking module is used to lock the finger joints if the wrist is in a rotating position in the current frame by obtaining the rotation amount of the finger joints in the historical frames. The virtual hand generation module is used to apply the wrist rotation amount and the historical frame finger joint rotation amount to the template hand to obtain a virtual hand.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.