Biomechanical modeling and normalized training data generation method and device for human hand posture estimation, equipment and medium

Abstract

The invention relates to a biomechanical modeling and normalized training data generation method and device for human hand posture estimation, equipment and a medium. The method comprises the following steps: acquiring target three-dimensional hand posture data of a target object; for each target object, acquiring reference static biomechanical characteristics and dynamic biomechanical characteristics corresponding to each target three-dimensional hand posture of the target object; for each target three-dimensional hand posture, determining a three-dimensional joint coordinate of the target three-dimensional hand posture and a two-dimensional joint coordinate corresponding to the three-dimensional joint coordinate based on the reference static biomechanical characteristics and the dynamic biomechanical characteristics; wherein the three-dimensional joint coordinates represent the spatial positions of the hand joint points; and determining hand posture training data based on the three-dimensional joint coordinates and the corresponding two-dimensional joint coordinates. By adopting the method, the individual biomechanical structure difference can be eliminated, and the number of training data can be expanded more accurately and reasonably.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning models, and in particular to a method, apparatus, computer device, and computer-readable storage medium for biomechanical modeling and normalized training data generation for human hand posture estimation. Background Art

[0002] 3D hand gesture estimation technology has important application prospects in fields such as virtual reality (VR), augmented reality (AR), human-computer interaction, and sign language recognition. In recent years, methods based on deep neural networks have become the mainstream research. However, the performance of such methods is highly dependent on the quality and distribution consistency of the training data.

[0003] The currently widely used 3D hand posture datasets have the problem of significant differences in hand biomechanical characteristics (such as bone length, palm structure, etc.). Summary of the Invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment and computer-readable storage medium for biomechanical modeling and normalized training data generation for human hand posture estimation that can eliminate differences in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for biomechanical modeling and normalized training data generation for human hand posture estimation, the method comprising:

[0006] Obtaining three-dimensional hand posture data of each target object;

[0007] For each of the target objects, reference static biomechanical features and dynamic biomechanical features corresponding to each of the target three-dimensional hand postures of the target object are obtained; wherein the reference static biomechanical features correspond to the target objects one-to-one, and the reference static biomechanical features are determined by normalizing the static biomechanical features of each of the target three-dimensional hand postures of each of the target objects; the static biomechanical features are used to characterize the features of the palm corresponding to the target three-dimensional hand posture; and the dynamic biomechanical features are used to characterize the spatial angles of the fingers corresponding to the target three-dimensional hand posture;

[0008] For each target three-dimensional hand posture, determining, based on the reference static biomechanical features and the dynamic biomechanical features, the three-dimensional joint coordinates of the target three-dimensional hand posture and the two-dimensional joint coordinates corresponding to the three-dimensional joint coordinates; wherein the three-dimensional joint coordinates represent the spatial positions of the hand joint points;

[0009] Based on each of the three-dimensional joint coordinates and the corresponding two-dimensional joint coordinates, human hand posture training data is determined.

[0010] In one embodiment, obtaining reference static biomechanical features corresponding to each target three-dimensional hand posture of the target object includes:

[0011] Calculating static biomechanical features of each target three-dimensional hand posture of the target object according to each target three-dimensional hand posture; the static biomechanical features including root bone length, angle between root bones, and palmar curvature;

[0012] The reference static biomechanical characteristics corresponding to the target object are determined according to the root bone length, the angle between the root bones, and the palm surface curvature of each target three-dimensional hand posture of the target object.

[0013] In one embodiment, the calculating, based on each of the target three-dimensional hand postures of the target object, the static biomechanical characteristics of each of the target three-dimensional hand postures includes:

[0014] For each target three-dimensional hand posture of the target object, obtaining root bone vectors of the palm corresponding to the target three-dimensional hand posture;

[0015] Calculating the Euclidean distance based on each of the root bone vectors to obtain the root bone length corresponding to the target three-dimensional hand posture;

[0016] Calculating the angles between the root bone vectors of adjacent root bones using an arc cosine function to obtain the angles between the root bones corresponding to the target three-dimensional hand posture; wherein the number of the angles between the root bones is less than the number of the root bone vectors;

[0017] Based on the root bone vectors of adjacent root bones, basis vectors are determined, and based on the basis vectors, the palm curvature corresponding to the target three-dimensional hand posture is determined; wherein the basis vectors represent standard orthogonal vectors of adjacent root bone vectors.

[0018] In one embodiment, obtaining each target three-dimensional hand posture of the target object includes:

[0019] Obtaining initial three-dimensional hand poses of the target object;

[0020] For each of the initial three-dimensional hand postures, obtaining an orthogonal reference coordinate system corresponding to the initial three-dimensional hand posture; the orthogonal reference coordinate system is a coordinate system with the palm center joint of each initial three-dimensional hand posture as the origin and the palm surface normal vector as the target direction;

[0021] Each of the initial three-dimensional hand postures is subjected to a root node position transformation and rotation in the corresponding orthogonal reference coordinate system to obtain each of the target three-dimensional hand postures.

[0022] In one embodiment, the dynamic biomechanical features include flexion and extension angles and abduction angles of finger bone vectors; obtaining the dynamic biomechanical features corresponding to each target three-dimensional hand posture of the target object includes:

[0023] For each target three-dimensional hand posture of the target object, obtaining each bone vector of each finger corresponding to the target three-dimensional hand posture and a local reference coordinate system where each finger is located; wherein the local reference coordinate system is determined based on a reference plane where the root bone vector of the palm is located;

[0024] For each finger, the flexion angle and the abduction angle of the finger bone vector are obtained in the local reference coordinate system where each finger is located; wherein the angle of each flexion angle is within the flexion angle reference range; and the angle of each abduction angle is within the abduction angle reference range.

[0025] In one embodiment, the reference static biomechanical features include a reference root bone length, a reference angle between root bones, and a reference palmar curvature; the dynamic biomechanical features include a flexion angle and an abduction angle of a finger bone vector; and determining the three-dimensional joint coordinates of the target three-dimensional hand posture and the two-dimensional joint coordinates corresponding to the three-dimensional joint coordinates based on the reference static biomechanical features and the dynamic biomechanical features includes:

[0026] Determining a vector code based on the flexion and extension angles of each finger bone vector and the abduction angles of each finger bone vector corresponding to the target three-dimensional hand posture, as well as the reference root bone length, the angle between the reference root bones, the reference palm surface curvature, and the surface direction constraint of the palm corresponding to the target three-dimensional hand posture;

[0027] Determine three-dimensional joint coordinates based on the vector code; wherein the three-dimensional joint coordinates represent the spatial position of the joint point;

[0028] Performing data enhancement on the three-dimensional joint coordinates through affine transformation to obtain different types of three-dimensional hand postures;

[0029] Each type of 3D hand posture is projected through view projection to obtain the 2D joint coordinates of each type of 3D hand posture.

[0030] In a second aspect, the present application provides a posture estimation model training method, comprising:

[0031] Obtaining the hand posture training data generated by the above method;

[0032] The three-dimensional posture estimation model is trained using the human hand posture training data; wherein the three-dimensional posture estimation model is used to process the two-dimensional joint coordinates in the human hand posture training data to obtain three-dimensional joint coordinates.

[0033] In a third aspect, the present application provides a biomechanical modeling and normalized training data generation device for human hand posture estimation, the device comprising:

[0034] A first acquisition module is used to acquire three-dimensional hand gesture data of each target object;

[0035] A second acquisition module is configured to acquire, for each target object, reference static biomechanical features and dynamic biomechanical features corresponding to each target three-dimensional hand posture of the target object; wherein the reference static biomechanical features correspond one-to-one to the target object, and the reference static biomechanical features are determined by normalizing the static biomechanical features of each target three-dimensional hand posture of each target object; the static biomechanical features are used to characterize features of the palm corresponding to the target three-dimensional hand posture; and the dynamic biomechanical features are used to characterize spatial angles of fingers corresponding to the target three-dimensional hand posture;

[0036] a first determination module configured to determine, for each target three-dimensional hand posture, three-dimensional joint coordinates of the target three-dimensional hand posture and two-dimensional joint coordinates corresponding to the three-dimensional joint coordinates based on the reference static biomechanical features and the dynamic biomechanical features; wherein the three-dimensional joint coordinates represent the spatial positions of the hand joint points;

[0037] The second determination module is used to determine the human hand posture training data based on each of the three-dimensional joint coordinates and the corresponding two-dimensional joint coordinates.

[0038] In a fourth aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0039] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0040] The above-mentioned biomechanical modeling and normalized training data generation method, device, computer equipment and computer-readable storage medium for human hand posture estimation obtain reference static biomechanical features by normalizing static biomechanical features to eliminate the negative effects of data annotation differences and individual structural differences; only retain the differences in dynamic biomechanical features to enhance the model's sensitivity to posture changes and robustness to individual structural differences; through the reference static biomechanical features and dynamic biomechanical features of the target three-dimensional hand posture, the target three-dimensional hand posture data can be expanded to obtain the three-dimensional joint coordinates and corresponding two-dimensional joint coordinates of the target three-dimensional hand posture that conforms to biomechanics; so as to obtain human hand posture training data with reasonable quantity, scale and diversity.

[0041] Due to the significant differences in the biomechanical characteristics of the hands, the 2D-3D mapping relationship learned by the model is unstable, which limits the generalization performance of the model. The above method eliminates the differences in the biomechanical characteristics of the hands and expands the diversity of the human hand posture training data. When the posture estimation model is trained using the diverse expanded training data, on the one hand, the accuracy and generalization performance of the posture estimation model can be improved; on the other hand, the posture estimation model outputs three-dimensional joint coordinates, reducing the dependence on manually labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 Schematic diagram of a process for biomechanical modeling and normalized training data generation for hand posture estimation in one embodiment;

[0044] Figure 2 This is a schematic diagram of hand gesture annotation in one embodiment. Figure 2 (a) is a schematic diagram of joint point marking. Figure 2 (b) is a schematic diagram of the annotation of the root bone vector of the palm and the finger bone vector of the fingers; Figure 2 (c) is a schematic diagram of the angle between the root bone vectors; Figure 2 (d) in the figure is a schematic diagram of basis vectors; Figure 2 (e) is a schematic diagram of the local reference coordinate system;

[0045] Figure 3 FIG1 is a schematic diagram of a process for calculating static biomechanical characteristics of each target three-dimensional hand posture in one embodiment;

[0046] Figure 4 A schematic diagram of a process for obtaining target three-dimensional hand postures of a target object in one embodiment;

[0047] Figure 5 Schematic diagram of normalization of biomechanical features of initial three-dimensional hand posture in one embodiment;

[0048] Figure 6 1. A schematic diagram of a process for determining three-dimensional joint coordinates and corresponding two-dimensional joint coordinates based on reference static biomechanical features and dynamic biomechanical features in one embodiment;

[0049] Figure 7 Schematic diagram of the 2D-3D mapping model training architecture generated based on biomechanical modeling and normalized training data;

[0050] Figure 8 1 is a structural block diagram of a device for biomechanical modeling and normalized training data generation for human hand posture estimation in one embodiment;

[0051] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] In one embodiment, Figure 1 As shown, a method for biomechanical modeling and normalized training data generation for human hand posture estimation is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps S102 to S108. Among them:

[0054] Step S102: Acquire the three-dimensional hand posture data of each target object.

[0055] Optionally, the terminal obtains different target three-dimensional hand gestures of Alice, Bob, and Charlie at different times, such as fist clenching, making a V sign, etc. The target three-dimensional hand gestures are normalized data.

[0056] Step S104 : For each target object, obtain reference static biomechanical features and dynamic biomechanical features corresponding to each target three-dimensional hand posture of the target object.

[0057] The reference static biomechanical features correspond to the target objects one by one, and the reference static biomechanical features are determined by normalizing the static biomechanical features of each target three-dimensional hand posture of each target object.

[0058] Static biomechanical features are used to characterize the palm features corresponding to the target 3D hand posture. The palm contains a set of joints The study found that individual palmar structures are stable in time and space, which can be used as static biomechanical characteristics.

[0059] Dynamic biomechanical features are used to characterize the spatial angles of the fingers corresponding to the target three-dimensional hand posture.

[0060] like Figure 2 As shown in (a), including joint points to The root joint is , Represents the root node before coordinate transformation, Represents the root node and joint point after coordinate transformation Defined as the central joint of the palm. The parent node is defined as ; Bone vector Defined as the parent joint j p(i+1) Point to its child joint j (i+1) The calculation method of the vector is shown in formula (1).

[0061] Formula (1)

[0062] The root bone vector set is represented as ,correspond Figure 2 The solid line part in (b); b0=j1-j p1 ; b4=j5-j p5 ; b8=j9-j p9 ; b 12 =j 13 -j p13 ; b 16 =j 17 -j p17 ; where j p1 Represents the parent node of joint point j1, j p5 Represents the parent node of joint point j5, j p9 Represents the parent node of joint point j9, j p13 Represents joint point j 13 The parent node of j p17 Represents joint point j 17 The parent node of is node j0.

[0063] The morphology of the fingers Figure 2 As shown by the dotted lines in (b), the nodes corresponding to the thumb bone vector are {j1 j2 j3 j4}; the nodes corresponding to the index finger bone vector are {j5 j6 j7 j8}; the nodes corresponding to the middle finger bone vector are {j9 j10 j11 j12}; the nodes corresponding to the ring finger bone vector are {j13 j14 j15 j16}; and the nodes corresponding to the little finger bone vector are {j17 j18 j19 j20}.

[0064] like Figure 2 As shown in (c), the angle between the root bone vectors, such as the angle between the root bone vector b0 and the root bone vector b4, is recorded as The angle between the root bone vector b4 and the root bone vector b8 is denoted as ; Root bone vector b8 and root bone vector b 12 The angle between ; root bone vector b 12 and the root bone vector b 16 The angle between .

[0065] like Figure 2 As shown in (d), the orthogonal vectors of adjacent root bones are defined as , e0=norm(b0×b4), the orthogonal vectors of b0 and b4 are normalized to obtain the basis vector e0; similarly, e1=norm(b4×b8), the orthogonal vectors of b4 and b8 are normalized to obtain the basis vector e1; based on b8 and b 12 The basis vector e3 can be obtained; based on b 12 and b 16 The basis vector e4 can be obtained.

[0066] like Figure 2 As shown in (e), each finger builds an independent local reference coordinate system The establishment of the local reference coordinate system takes the palmar surface as the reference plane root bone, are considered to be roughly in the same plane.

[0067] Optionally, the terminal obtains, for each target object, reference static biomechanical features and dynamic biomechanical features corresponding to each target three-dimensional hand posture of the target object. For example, reference static biomechanical features and dynamic biomechanical features corresponding to each target three-dimensional hand posture of target object Alice, reference static biomechanical features and dynamic biomechanical features corresponding to each target three-dimensional hand posture of target object Bob, and reference static biomechanical features and dynamic biomechanical features corresponding to each target three-dimensional hand posture of target object Charlie.

[0068] Furthermore, by unifying the static biomechanical features of the same target object with the reference static biomechanical features, the only differences between different target 3D hand postures of the same target object remain in the dynamic biomechanical features. During training, the "changes" observed by the model are solely due to changes in posture, without being affected by individual structural differences.

[0069] Step S106 , for each target 3D hand posture, based on the reference static biomechanical features and the dynamic biomechanical features, determine the 3D joint coordinates of the target 3D hand posture and the 2D joint coordinates corresponding to the 3D joint coordinates.

[0070] Among them, the three-dimensional joint coordinates represent the spatial position of the hand joint points.

[0071] Optionally, the terminal combines the reference static biomechanical features and dynamic biomechanical features of each target three-dimensional hand posture to obtain biomechanical features corresponding to different target three-dimensional hand postures, which are recorded as [reference static biomechanical features, dynamic biomechanical features]. The terminal combines the reference static biomechanical features and dynamic biomechanical features of each target three-dimensional hand posture into a complete code. Terminal pair encoding Processing, such as weakly supervised processing, is performed to obtain the 3D joint coordinates and the corresponding 2D joint coordinates for each target 3D hand pose. By modeling the individual hand structure and motion constraints, a large number of physically feasible 3D joint coordinates and their corresponding 2D joint coordinates can be automatically generated without additional manual annotation.

[0072] Step S108 : determining the hand posture training data based on each three-dimensional joint coordinate and the corresponding two-dimensional joint coordinate.

[0073] Optionally, the terminal records the obtained three-dimensional joint coordinates as 3D joint coordinates and the corresponding two-dimensional joint coordinates as 2D joint coordinates, and splices them into a posture pair, which can be recorded as [2D joint coordinates, 3D joint coordinates] to obtain multiple posture pairs.

[0074] It should be noted that there is no limit on the number of target objects, nor on the number of target three-dimensional hand gestures.

[0075] In the above-mentioned biomechanical modeling and normalized training data generation method for hand posture estimation, the static biomechanical features are normalized to obtain reference static biomechanical features to eliminate the negative effects of data annotation differences and individual structural differences; only the differences in dynamic biomechanical features are retained to enhance the model's sensitivity to posture changes and robustness to individual structural differences; through the reference static biomechanical features and dynamic biomechanical features of the target three-dimensional hand posture, the target three-dimensional hand posture data can be expanded to obtain the three-dimensional joint coordinates and corresponding two-dimensional joint coordinates of the target three-dimensional hand posture that conforms to biomechanics; so as to obtain human hand posture training data with a reasonable quantity, scale and diversity.

[0076] In an exemplary embodiment, obtaining reference static biomechanical features corresponding to each target three-dimensional hand posture of the target object includes: calculating the static biomechanical features of each target three-dimensional hand posture according to each target three-dimensional hand posture of the target object; wherein the static biomechanical features include root bone length, angle between root bones, and palm surface curvature; determining the reference static biomechanical features corresponding to the target object according to the root bone length, angle between root bones, and palm surface curvature of each target three-dimensional hand posture of the target object.

[0077] The root bone length is recorded as , referred to as , the angle between the root bones is recorded as , the palmar curvature is recorded as The reference root bone length is recorded as ; The angle between the reference roots is recorded as ; The reference palm curvature is recorded as .

[0078] Optionally, the terminal obtains the root bone length corresponding to each target 3D hand posture of the target object Alice , the angle between the root bones , palmar curvature The root bone length corresponding to each target 3D hand posture of the target object Bob , the angle between the root bones , palmar curvature The root bone length corresponding to each target 3D hand posture of the target object Charlie , the angle between the root bones , palmar curvature .

[0079] Furthermore, the terminal counts the static biomechanical characteristics of all target three-dimensional hand postures of each target object. Since the measurement of a single target three-dimensional hand posture may be noisy or affected by slight posture, taking the average of the static biomechanical characteristics of all target three-dimensional hand postures of each target object can more robustly estimate its true, inherent body structure and reduce measurement errors and posture interference.

[0080] Optionally, the reference static biomechanical features include reference root bone length, reference angle between root bones and reference palm surface curvature. The terminal calculates the mean of the root bone lengths corresponding to each target three-dimensional hand posture of the target object to obtain the reference root bone length of the target object; calculates the mean of the angles between the root bones corresponding to each target three-dimensional hand posture of the target object to obtain the reference angle between the root bones of the target object; calculates the mean of the reference palm surface curvatures corresponding to each target three-dimensional hand posture of the target object to obtain the reference palm surface curvature of the target object. The terminal calculates the static biomechanical features corresponding to each target three-dimensional hand posture of the target object Replaced with reference static biomechanical features .

[0081] It should be noted that all target 3D hand postures of the same target object now have exactly the same reference static biomechanical characteristics. The reference static biomechanical characteristics of different target objects are different.

[0082] In this embodiment, by unifying the static biomechanical features of the same target object and replacing them with reference static biomechanical features, the differences in body structure between different target objects, i.e., individuals, in the training data are eliminated, thereby forcing the model to focus on learning the dynamic changes of the posture itself.

[0083] In an exemplary embodiment, Figure 3 As shown, according to each target 3D hand posture of the target object, the static biomechanical characteristics of each target 3D hand posture are calculated, including the following steps S302 to S308.

[0084] Step S302 : for each target 3D hand posture of the target object, obtain each root bone vector of the palm corresponding to the target 3D hand posture.

[0085] Among them, the root bone vector set is expressed as ,correspond Figure 2 The solid line part in (b). Each target 3D hand pose corresponds to 5 root bone vectors.

[0086] Optionally, the terminal traverses each target object, obtains all target three-dimensional hand postures of each target object, and obtains five root bone vectors of the palm corresponding to each target three-dimensional hand posture.

[0087] Step S304: Calculate the Euclidean distance based on each root bone vector to obtain the root bone length corresponding to the target three-dimensional hand posture.

[0088] Optionally, the terminal calculates the Euclidean distance based on each bone vector, such as 、 、 、 and , we get the five root bone lengths corresponding to the target three-dimensional hand posture, and the set of each root bone length is expressed as .

[0089] Step S306: Calculate the angle between the root bone vectors of adjacent root bones using an arc cosine function to obtain the angle between the root bones corresponding to the target three-dimensional hand posture.

[0090] Among them, the number of angles between the root bones is 4, which is less than the number of root bone vectors, 5. and Angular distance Calculated using the arc cosine function, as shown in formula (2).

[0091] Formula (2)

[0092] Optionally, the terminal calculates the angle between the root bone vectors of adjacent root bones using the arc cosine function through formula (3) to obtain the angle between the root bones corresponding to the target three-dimensional hand posture. For example, the angle between the root bone vector b0 and the root bone vector b4 is recorded as The angle between the root bone vector b4 and the root bone vector b8 is denoted as ; Root bone vector b8 and root bone vector b 12 The angle between ; root bone vector b 12 and the root bone vector b 16 The angle between .

[0093] Formula (3)

[0094] Step S308: determining basis vectors based on the root bone vectors of the adjacent root bones, and determining the palm curvature corresponding to the target three-dimensional hand posture based on the basis vectors.

[0095] The basis vectors represent the standard orthogonal vectors of adjacent root bone vectors, and the palmar curvature represents the curvature relationship between adjacent bone surfaces.

[0096] Optionally, the terminal determines the basis vector according to the root bone vectors of the adjacent root bones. For example, the orthogonal vectors of b0 and b4 are normalized to obtain the basis vector e0. Similarly, e1=norm(b4×b8), the orthogonal vectors of b4 and b8 are normalized to obtain the basis vector e1. Based on b8 and b 12 The basis vector e3 can be obtained; based on b 12 and b 16 The basis vector e4 can be obtained. The terminal determines the palm curvature corresponding to the target 3D hand posture using each basis vector. Since there are four basis vectors in a target 3D hand posture, there are three palm curvatures. The palm curvature is calculated as shown in formula (4).

[0097] Formula (4)

[0098] In this embodiment, static biomechanical characteristics can be calculated using the root bone vectors corresponding to a target three-dimensional hand posture, including the root bone length, the angle between the root bone vectors, and the palm curvature.

[0099] In an exemplary embodiment, Figure 4 As shown, obtaining each target 3D hand posture of the target object includes steps S402 to S406.

[0100] Step S402: Acquire each initial three-dimensional hand posture of the target object.

[0101] The different initial 3D hand postures are different hand gestures from different objects at different times. The target object is a specific person.

[0102] Optionally, the terminal collects multiple public three-dimensional (3D) hand datasets, such as the STB, RHD, FreiHAND, and HO-3D hand datasets. These hand datasets cover a variety of acquisition environments, camera parameters, and hand pose distributions. These hand datasets include different initial 3D hand poses from different subjects at different times. Hand datasets typically include initial 3D hand poses corresponding to multiple different targets. For example, the terminal collects different initial 3D hand poses of Alice, Bob, and Charlie at different times, such as fists and V signs.

[0103] Step S404 : For each initial three-dimensional hand posture, obtain an orthogonal reference coordinate system corresponding to the initial three-dimensional hand posture.

[0104] Among them, the orthogonal reference coordinate system is a coordinate system with the palm center joint of each initial three-dimensional hand posture as the origin and the palm normal vector as the target direction such as the z-axis. The orthogonal reference coordinate system is denoted as The normal vector is perpendicular to the palm surface and best represents the direction of the palm in the initial three-dimensional hand posture. That is, the origin is located at the center joint of the palm. , the Z axis coincides with the palm normal vector, the X axis is parallel to the extension direction of the metacarpal bone, and the Y axis is calculated by cross product to ensure that the three axes are orthogonal.

[0105] The central joints of the palm, for example Figure 2 The joints in Defined as the central joint of the palm, Figure 5 The lowest joint of the middle finger, measured from top to bottom. The palmar center joint is the core of the hand's skeleton, connecting the wrist and fingers. Using it as the origin best represents the "center" of the hand.

[0106] Optionally, for each initial 3D hand pose data, the terminal obtains an orthogonal reference coordinate system corresponding to each initial 3D hand pose. The initial 3D hand pose data has different absolute coordinates (x, y, z) in 3D space, and the palm may be facing up, down, tilted, or rotated. Therefore, the orientation of the palm plane in space is unique for each initial 3D hand pose data. The terminal obtains an orthogonal reference coordinate system corresponding to each initial 3D hand pose. The orthogonal reference coordinate system is a coordinate system with the palm center joint of each initial 3D hand pose as the origin and the palm normal vector as the target direction, such as the z-axis.

[0107] Step S406 , performing a position transformation and rotation of the root node of each initial three-dimensional hand posture in a corresponding orthogonal reference coordinate system to obtain each target three-dimensional hand posture.

[0108] Optionally, the terminal translates the palm center of each initial 3D hand pose data to the origin of the world coordinate system (0,0,0), eliminating absolute position differences in space for the initial 3D hand pose data. By rotating the orthogonal reference coordinate system for each initial 3D hand pose data, the Z axes (calculated based on their own normal vectors) of all initial 3D hand pose data are aligned (e.g., pointing in the Z+ direction of the world coordinate system), eliminating absolute differences in the palm orientation of each initial 3D hand pose data.

[0109] Furthermore, by converting all the initial 3D hand pose data to their respective local coordinate systems (with the origin at the palm of the hand and the Z axis perpendicular to the palm surface), pose standardization is achieved. In this way, a "thumbs-up" gesture, no matter where it appears in the room or which direction the palm faces in the initial 3D hand pose data, has very similar joint coordinate relative relationships in the normalized coordinate system. Figure 5 As shown, an initial 3D hand posture data is normalized in the corresponding orthogonal reference coordinate system, such as position transformation and rotation, that is, the root node After changing and rotating to Get the target 3D hand pose.

[0110] Optionally, the initial 3D hand posture data is a fist, and the terminal converts the palm center joint Translate to the origin of the coordinate system and rotate the entire hand so that the normal vector coincides with the Z axis. is fixed at the origin, and the root node is Figure 2 in Get the new coordinate value, and finally get The target 3D hand pose is normalized in coordinates.

[0111] In this embodiment, the standardization of the initial three-dimensional hand posture is achieved by converting all the initial three-dimensional hand posture data into their respective local coordinate systems (with the origin at the palm of the hand and the Z axis perpendicular to the palm surface of the hand).

[0112] In an exemplary embodiment, the dynamic biomechanical features include the angles of flexion and extension and the angles of abduction of the finger bone vectors; the dynamic biomechanical features corresponding to each target three-dimensional hand posture of the target object are obtained, including: for each target three-dimensional hand posture of the target object, obtaining each bone vector of each finger corresponding to the target three-dimensional hand posture and the local reference coordinate system where each finger is located; wherein the local reference coordinate system is determined based on the reference plane where the root bone vector of the palm is located; for each finger, obtaining the angles of flexion and extension and the angles of abduction of the finger bone vector in the local reference coordinate system where each finger is located.

[0113] The angle of each flexion and extension angle is recorded as The angle of each abduction angle is recorded as Located within the abduction angle reference range. Flexion and extension angles represent the bending angle of the phalanges in the sagittal plane; abduction angles represent the spreading angle of the phalanges in the coronal plane. The local reference coordinate system is based on the reference plane of the palmar root vector.

[0114] Optionally, the terminal counts the numerical distribution of the flexion and extension angles and the abduction angles of each finger bone vector of each target three-dimensional hand posture, and extracts the maximum and minimum values ​​of the flexion and extension angles of each finger bone vector; as well as the maximum and minimum values ​​of the abduction angle.

[0115] The ranges of the abduction and flexion angles of all finger bone vectors are combined into a multidimensional space. Random sampling within this multidimensional space ensures that the generated gestures 100% meet the limits of human motion (for example, fingers will not bend backwards).

[0116] For each target three-dimensional hand posture of the target object, the terminal obtains the skeleton vectors of each finger corresponding to the target three-dimensional hand posture and the local reference coordinate system of each finger.

[0117] By minimizing the following objective function , as shown in formula (5), solve the normal vector of the reference plane , so that the normal vector and the basis vector 、 、 Maintain as much orthogonality as possible to construct the palm reference hyperplane :

[0118] Formula (5)

[0119] Where, Represents the standard orthogonal basis vectors of the world coordinate system; || ||2 represents the Euclidean distance; Represents minimizing the deviation between the normal vector and the basis vector; Represents the reference plane The normal vector of .

[0120] Therefore, the local reference coordinate system The three axes of are defined as shown in formula (6).

[0121] Formula (6)

[0122] Where, Represents bone vector The parent bone vector of the current bone. The anatomical counterpart is the direction vector of the previous bone of the current bone. represents the X-axis of the local coordinate system, which is geometrically defined as parallel to the palmar extension direction (thumb → little finger); represents the Y axis of the local coordinate system, which is geometrically defined as the perpendicular direction of the palmar surface normal; Represents the Z axis of the local coordinate system, which is geometrically defined as perpendicular to the palmar surface (direction of the normal vector).

[0123] The terminal sets the local reference coordinate system for each finger. Get the flexion and extension angle of the finger bone vector and the angle of the abduction angle .

[0124] Among them, α( ) represents the angle calculation function between two vectors, Represents bone vector A two-dimensional vector projected on the XZ plane to eliminate the Y-axis component. The flexion and extension angles are based on the bone vector The angle between the XZ plane projection and the Z axis is determined, and the angle of the abduction angle is based on the bone vector Projection on the XZ plane with bone vectors Self-determination.

[0125] In this embodiment, the movement (flexion / extension / abduction) of each finger is described in a local coordinate system that conforms to the anatomical structure, providing a physically reasonable parameterization basis for gesture recognition.

[0126] In an exemplary embodiment, Figure 6 As shown, the reference static biomechanical features include the reference root bone length, the reference angle between the root bones, and the reference palmar curvature; the dynamic biomechanical features include the flexion and extension angles and the abduction angles of the finger bone vectors; based on the reference static biomechanical features and the dynamic biomechanical features, the three-dimensional joint coordinates of the target three-dimensional hand posture and the two-dimensional joint coordinates corresponding to the three-dimensional joint coordinates are determined, including steps S602 to S608. Among them:

[0127] Step S602 determines the vector encoding based on the flexion and extension angles of each finger bone vector corresponding to the target three-dimensional hand posture, the abduction angles of each finger bone vector, and the reference root bone length, the angle between the reference root bones, the reference palm surface curvature and the surface direction constraint of the palm corresponding to the target three-dimensional hand posture.

[0128] The surface direction constraint is determined based on the reference palm curvature. When calculating the palm curvature using the inverse cosine arccos function, due to the symmetry of the arccos function, the calculation may generate multiple palm structure solutions. To ensure the uniqueness of the solution, the relative surface direction constraint is introduced. , as shown in formula (7).

[0129] Formula (7)

[0130] Optionally, the terminal determines the vector encoding by the angle of the flexion and extension angle of each finger bone vector corresponding to the target three-dimensional hand posture, the angle of the abduction angle of each finger bone vector, and the reference root bone length of the palm corresponding to the target three-dimensional hand posture, the angle between the reference root bones, the reference palm surface curvature and the surface direction constraint. , represents the normalized and encoded hand biomechanical parameter vector. The original target three-dimensional hand posture corresponding to the flexion and extension angles of each finger bone vector, the abduction angle of each finger bone vector, and the palm root bone length, the angle between the root bones, the palm curvature and the surface direction constraint of the target three-dimensional hand posture are encoded by another vector It should be noted that and are the same.

[0131] like Figure 5 As shown, the hand biomechanical parameter vector of the target three-dimensional hand posture data is , normalized, and obtained vector encoding .

[0132] Step S604: Determine the three-dimensional joint coordinates based on the vector encoding.

[0133] Among them, the three-dimensional joint coordinates represent the spatial position of the joint point.

[0134] Optionally, the terminal solves the problem based on the Sequential Least Squares Quadratic Programming (SLSQP) optimization algorithm. The vector encodes the corresponding 3D joint coordinates.

[0135] Step S606 , performing data enhancement on the three-dimensional joint coordinates through affine transformation to obtain different types of three-dimensional hand postures.

[0136] Optionally, the terminal performs data enhancement on the three-dimensional joint coordinates through affine transformation, such as translation and rotation, to enhance its geometric diversity and obtain different types of three-dimensional hand postures.

[0137] Step S608 : Projecting each type of three-dimensional hand posture through view projection to obtain the two-dimensional joint coordinates of each type of three-dimensional hand posture.

[0138] Among them, perspective projection is to simulate the three-dimensional hand posture of each disturbance type, that is, the 3D gesture, into the camera shooting effect to generate matching 2D key point data.

[0139] Optionally, the terminal obtains a corresponding two-dimensional joint coordinate representation by performing random perspective projection on the three-dimensional hand posture of each disturbance type.

[0140] In this embodiment, high-quality 3D pose data is expensive to acquire and difficult to annotate using traditional techniques, significantly limiting the diversity and scale of training data. Through perturbation and projection, a single target 3D hand pose can be derived into hundreds of visual representations, all of which conform to physical laws, thereby expanding diversity and scale. This approach eliminates the need for additional manual annotation and enables the automatic synthesis of a large number of physically feasible 2D-3D pose samples by modeling individual hand structures and motion constraints.

[0141] In an exemplary embodiment, a posture estimation model training method includes: obtaining human hand posture training data generated by the above method; and training a three-dimensional posture estimation model using the human hand posture training data.

[0142] Among them, the three-dimensional posture estimation model is used to process the two-dimensional joint coordinates in the human hand posture training data to obtain the three-dimensional joint coordinates.

[0143] Optionally, the terminal generates the hand posture training data by any of the above methods, such as Figure 7 As shown, the static biomechanical features corresponding to a target 3D hand posture of the target object are obtained, and the static biomechanical features corresponding to the target 3D hand posture are normalized. That is, first, an initial 3D hand posture data is normalized at the root node in the corresponding orthogonal reference coordinate system, such as position transformation and rotation, to obtain the target 3D hand posture, and the static biomechanical features corresponding to the target 3D hand posture are normalized. Replaced with the reference static biomechanical features of the target object to which the target 3D hand pose belongs Based on the reference static biomechanical features and dynamic biomechanical features of the target 3D hand posture, the vector encoding of the biomechanical features is obtained. , based on the sequential least squares quadratic programming optimization algorithm, solve the The vector encodes the consistent spatial positions of the 3D joints. The spatial positions of the 3D joints are expanded by performing spatial perturbations such as affine transformations. The expanded 3D hand gestures are then projected with random viewpoints to obtain corresponding 2D keypoint representations, thereby constructing a rich set of well-structured 2D-3D pose pairs. The 2D-3D pose pairs are then used to train a 3D pose estimation model. For example, when designing a 3D pose estimation model on the PyTorch platform, typical 2D-3D mapping models such as ST-GCN and VPose3D can be used. The parameters can be configured as follows: L2 loss (MSE) is used as the regression loss function; the input image size is 224×224, and the initial learning rate is set to 0.001. The batch size is 512, and the total number of training epochs is 500. The mean squared error (MSE) is used as the loss function during training. This configuration is applicable to both weakly supervised training and normalization strategies. In weakly supervised training, 200,000 samples are generated for each individual. The trained 3D pose estimation model is obtained.

[0144] Due to the significant differences in the biomechanical characteristics of the hands, the 2D-3D mapping relationship learned by the model is unstable, which limits the generalization performance of the model. The above method eliminates the differences in the biomechanical characteristics of the hands and expands the diversity of the human hand posture training data. When the posture estimation model is trained using the diverse expanded training data, on the one hand, the accuracy and generalization performance of the posture estimation model can be improved; on the other hand, the posture estimation model outputs three-dimensional joint coordinates, reducing the dependence on manually labeled data.

[0145] Using a trained 3D pose estimation model: 2D hand images are captured in real time using an external image acquisition device for subsequent keypoint extraction. The input images are processed using a high-precision 2D pose estimation algorithm (such as HRNet) to extract the 2D spatial coordinates of key joints, thereby obtaining a standard 2D hand pose representation. These 2D spatial coordinates are then fed into a trained 3D pose estimation model (such as ST-GCN or VPose3D) to infer the hand pose in 3D space. This regression process maintains real-time performance, good pose structure consistency, and spatial rationality, making it suitable for applications such as human-computer interaction and virtual reality.

[0146] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0147] Based on the same inventive concept, an embodiment of the present application also provides a device for generating biomechanical modeling and normalized training data for hand posture estimation, which is used to implement the above-mentioned method for generating biomechanical modeling and normalized training data for hand posture estimation. The implementation solution provided by the device is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations of one or more embodiments of the device for generating biomechanical modeling and normalized training data for hand posture estimation provided below can be found in the above-mentioned limitations of the method for generating biomechanical modeling and normalized training data for hand posture estimation, and will not be repeated here.

[0148] In an exemplary embodiment, Figure 8 As shown, a biomechanical modeling and normalized training data generation device for human hand posture estimation is provided, comprising: a first acquisition module 801, a second acquisition module 802, a first determination module 803 and a second determination module 804, wherein:

[0149] The first acquisition module 801 is used to acquire the three-dimensional hand gesture data of each target object.

[0150] The second acquisition module 802 is used to obtain, for each target object, the reference static biomechanical features and dynamic biomechanical features corresponding to each target three-dimensional hand posture of the target object; wherein, the reference static biomechanical features correspond to the target objects one-to-one, and the reference static biomechanical features are determined by normalizing the static biomechanical features of each target three-dimensional hand posture of each target object; the static biomechanical features are used to characterize the features of the palm corresponding to the target three-dimensional hand posture; and the dynamic biomechanical features are used to characterize the spatial angles of the fingers corresponding to the target three-dimensional hand posture.

[0151] The first determination module 803 is used to determine the three-dimensional joint coordinates of the target three-dimensional hand posture and the two-dimensional joint coordinates corresponding to the three-dimensional joint coordinates for each target three-dimensional hand posture based on reference static biomechanical characteristics and dynamic biomechanical characteristics; wherein the three-dimensional joint coordinates represent the spatial position of the hand joint points.

[0152] The second determining module 804 is configured to determine the hand posture training data based on the three-dimensional joint coordinates and the corresponding two-dimensional joint coordinates.

[0153] In an exemplary embodiment, the second acquisition module 802 is also used to calculate the static biomechanical characteristics of each target three-dimensional hand posture of the target object based on each target three-dimensional hand posture of the target object; the static biomechanical characteristics include the root bone length, the angle between the root bones and the palm surface curvature; based on the root bone length, the angle between the root bones and the palm surface curvature of each target three-dimensional hand posture of the target object, the reference static biomechanical characteristics corresponding to the target object are determined.

[0154] In an exemplary embodiment, the second acquisition module 802 is also used to obtain, for each target three-dimensional hand posture of the target object, each root bone vector of the palm corresponding to the target three-dimensional hand posture; calculate the Euclidean distance based on each root bone vector to obtain the root bone length corresponding to the target three-dimensional hand posture; calculate the angle between the root bone vectors of adjacent root bones through the inverse cosine function to obtain the angle between the root bones corresponding to the target three-dimensional hand posture; wherein, the number of angles between the root bones is less than the number of root bone vectors; determine the basis vectors according to the root bone vectors of adjacent root bones, and determine the palm curvature corresponding to the target three-dimensional hand posture according to the basis vectors; wherein, the basis vectors represent the standard orthogonal vectors of adjacent root bone vectors.

[0155] In an exemplary embodiment, the first acquisition module 801 is also used to obtain each initial three-dimensional hand posture of the target object; for each initial three-dimensional hand posture, an orthogonal reference coordinate system corresponding to the initial three-dimensional hand posture is obtained; the orthogonal reference coordinate system is a coordinate system with the palm center joint of each initial three-dimensional hand posture as the origin and the palm surface normal vector as the target direction; the root node position of each initial three-dimensional hand posture is transformed and rotated in the corresponding orthogonal reference coordinate system to obtain each target three-dimensional hand posture.

[0156] In an exemplary embodiment, the dynamic biomechanical characteristics include the angles of flexion and extension and the angles of abduction of the finger bone vectors; the second acquisition module 802 is also used to obtain, for each target three-dimensional hand posture of the target object, the bone vectors of each finger corresponding to the target three-dimensional hand posture and the local reference coordinate system of each finger; wherein the local reference coordinate system is determined based on the reference plane where the root bone vector of the palm is located; for each finger, the angles of flexion and extension and the angles of abduction of the finger bone vectors are obtained in the local reference coordinate system where each finger is located; wherein the angle of each flexion and extension angle is within the flexion and extension angle reference range; and the angle of each abduction angle is within the abduction angle reference range.

[0157] In an exemplary embodiment, the reference static biomechanical features include the reference root bone length, the reference root bone angle and the reference palm surface curvature; the dynamic biomechanical features include the angle of the flexion and extension angle and the angle of the abduction angle of the finger bone vector; the first determination module 803 is used to determine the vector encoding based on the angle of the flexion and extension angle of each finger bone vector corresponding to the target three-dimensional hand posture, the angle of the abduction angle of each finger bone vector, and the reference root bone length, the reference root bone angle, the reference palm surface curvature and the surface direction constraint of the palm corresponding to the target three-dimensional hand posture; based on the vector encoding, determine the three-dimensional joint coordinates; wherein the three-dimensional joint coordinates represent the spatial position of the joint point; perform data enhancement on the three-dimensional joint coordinates through affine transformation to obtain different three-dimensional hand postures; project the three-dimensional hand posture of each disturbance type through perspective projection to obtain the two-dimensional joint coordinates of the three-dimensional hand posture of each disturbance type.

[0158] In an exemplary embodiment, a posture estimation model training device is provided, comprising: a third acquisition module and a training module, wherein:

[0159] The third acquisition module is used to obtain the human hand posture training data generated by any of the above methods.

[0160] The training module is used to train the three-dimensional posture estimation model using the human hand posture training data; wherein the three-dimensional posture estimation model is used to process the two-dimensional joint coordinates in the human hand posture training data to obtain three-dimensional joint coordinates.

[0161] Each module in the aforementioned biomechanical modeling and normalized training data generation device for hand posture estimation and posture estimation model training device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device's memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0162] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store human hand posture data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for generating biomechanical modeling and normalized training data for human hand posture estimation is implemented.

[0163] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0164] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0165] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0166] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0167] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0168] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0169] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A biomechanical modeling and normalized training data generation method for hand posture estimation, characterized by: The method comprises: Obtaining three-dimensional hand posture data of each target object; For each of the target objects, reference static biomechanical features and dynamic biomechanical features corresponding to each of the target three-dimensional hand postures of the target object are obtained; wherein the reference static biomechanical features correspond to the target objects one-to-one, and the reference static biomechanical features are determined by normalizing the static biomechanical features of each of the target three-dimensional hand postures of each of the target objects; the static biomechanical features are used to characterize the features of the palm corresponding to the target three-dimensional hand posture; and the dynamic biomechanical features are used to characterize the spatial angles of the fingers corresponding to the target three-dimensional hand posture; For each target three-dimensional hand posture, determining, based on the reference static biomechanical features and the dynamic biomechanical features, the three-dimensional joint coordinates of the target three-dimensional hand posture and the two-dimensional joint coordinates corresponding to the three-dimensional joint coordinates; wherein the three-dimensional joint coordinates represent the spatial positions of the hand joint points; Based on each of the three-dimensional joint coordinates and the corresponding two-dimensional joint coordinates, human hand posture training data is determined.

2. The method according to claim 1, characterized in that The obtaining of reference static biomechanical features corresponding to each target three-dimensional hand posture of the target object includes: Calculating static biomechanical features of each target three-dimensional hand posture of the target object according to each target three-dimensional hand posture; the static biomechanical features including root bone length, angle between root bones, and palmar curvature; The reference static biomechanical characteristics corresponding to the target object are determined according to the root bone length, the angle between the root bones, and the palm surface curvature of each target three-dimensional hand posture of the target object.

3. The method according to claim 2, characterized in that The step of calculating the static biomechanical characteristics of each target three-dimensional hand posture of the target object according to each target three-dimensional hand posture includes: For each target three-dimensional hand posture of the target object, obtaining root bone vectors of the palm corresponding to the target three-dimensional hand posture; Calculating the Euclidean distance based on each of the root bone vectors to obtain the root bone length corresponding to the target three-dimensional hand posture; Calculating the angles between the root bone vectors of adjacent root bones using an arc cosine function to obtain the angles between the root bones corresponding to the target three-dimensional hand posture; wherein the number of the angles between the root bones is less than the number of the root bone vectors; Based on the root bone vectors of adjacent root bones, basis vectors are determined, and based on the basis vectors, the palm curvature corresponding to the target three-dimensional hand posture is determined; wherein the basis vectors represent standard orthogonal vectors of adjacent root bone vectors.

4. The method according to claim 1, wherein The obtaining of each target three-dimensional hand posture of the target object includes: Obtaining initial three-dimensional hand poses of the target object; For each of the initial three-dimensional hand postures, obtaining an orthogonal reference coordinate system corresponding to the initial three-dimensional hand posture; the orthogonal reference coordinate system is a coordinate system with the palm center joint of each initial three-dimensional hand posture as the origin and the palm surface normal vector as the target direction; Each of the initial three-dimensional hand postures is subjected to a root node position transformation and rotation in the corresponding orthogonal reference coordinate system to obtain each of the target three-dimensional hand postures.

5. The method according to claim 1, wherein The dynamic biomechanical features include the flexion and extension angles and the abduction angles of the finger bone vectors; obtaining the dynamic biomechanical features corresponding to each target three-dimensional hand posture of the target object includes: For each target three-dimensional hand posture of the target object, obtaining each bone vector of each finger corresponding to the target three-dimensional hand posture and a local reference coordinate system where each finger is located; wherein the local reference coordinate system is determined based on a reference plane where the root bone vector of the palm is located; For each finger, the flexion angle and the abduction angle of the finger bone vector are obtained in the local reference coordinate system where each finger is located; wherein the angle of each flexion angle is within the flexion angle reference range; and the angle of each abduction angle is within the abduction angle reference range.

6. The method according to claim 1, characterized in that The reference static biomechanical features include a reference root bone length, a reference angle between root bones, and a reference palmar curvature; the dynamic biomechanical features include a flexion angle and an abduction angle of a finger bone vector; and determining the three-dimensional joint coordinates of the target three-dimensional hand posture and the two-dimensional joint coordinates corresponding to the three-dimensional joint coordinates based on the reference static biomechanical features and the dynamic biomechanical features includes: Determining a vector code based on the flexion and extension angles of each finger bone vector and the abduction angles of each finger bone vector corresponding to the target three-dimensional hand posture, as well as the reference root bone length, the angle between the reference root bones, the reference palm surface curvature, and the surface direction constraint of the palm corresponding to the target three-dimensional hand posture; Determine three-dimensional joint coordinates based on the vector code; wherein the three-dimensional joint coordinates represent the spatial position of the joint point; Performing data enhancement on the three-dimensional joint coordinates through affine transformation to obtain different types of three-dimensional hand postures; Each type of 3D hand posture is projected through view projection to obtain the 2D joint coordinates of each type of 3D hand posture.

7. A posture estimation model training method, characterized in that: include: Obtaining human hand posture training data generated by the method according to any one of claims 1 to 6; Training a three-dimensional pose estimation model using the hand pose training data; in; The three-dimensional posture estimation model is used to process the two-dimensional joint coordinates in the human hand posture training data to obtain three-dimensional joint coordinates.

8. A biomechanical modeling and normalized training data generation device for hand posture estimation, characterized in that: The device comprises: A first acquisition module is used to acquire three-dimensional hand gesture data of each target object; A second acquisition module is configured to acquire, for each target object, reference static biomechanical features and dynamic biomechanical features corresponding to each target three-dimensional hand posture of the target object; wherein the reference static biomechanical features correspond one-to-one to the target object, and the reference static biomechanical features are determined by normalizing the static biomechanical features of each target three-dimensional hand posture of each target object; the static biomechanical features are used to characterize features of the palm corresponding to the target three-dimensional hand posture; and the dynamic biomechanical features are used to characterize spatial angles of fingers corresponding to the target three-dimensional hand posture; a first determination module configured to determine, for each target three-dimensional hand posture, three-dimensional joint coordinates of the target three-dimensional hand posture and two-dimensional joint coordinates corresponding to the three-dimensional joint coordinates based on the reference static biomechanical features and the dynamic biomechanical features; wherein the three-dimensional joint coordinates represent the spatial positions of the hand joint points; The second determination module is used to determine the human hand posture training data based on each of the three-dimensional joint coordinates and the corresponding two-dimensional joint coordinates.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images