Hand posture acquisition method and device, electronic equipment and readable storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING ZITIAO NETWORK TECH CO LTD
- Filing Date
- 2021-12-21
- Publication Date
- 2026-07-14
Smart Images
Figure CN116301303B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and readable storage medium for obtaining hand posture. Background Technology
[0002] Obtaining the 3D pose of the hand refers to accurately identifying the positions of various joints in a human hand from an image, thereby obtaining the hand's 3D pose. Hand pose recognition is commonly used in applications such as human-computer interaction, augmented reality, and virtual reality, and is one of the mainstream interaction methods currently available.
[0003] Currently, obtaining the 3D pose of the hand involves analyzing the positions of joints in an image to determine their location within a specific 3D coordinate system. The accuracy of the hand pose is crucial, significantly impacting the user experience. However, this method can be flawed due to factors such as occlusion of joints or image blurriness, leading to inaccurate joint positions and consequently affecting the overall accuracy of the 3D pose. Therefore, achieving more accurate 3D hand poses is a pressing issue that needs to be addressed. Summary of the Invention
[0004] To address the aforementioned technical problems, this disclosure provides a hand posture acquisition method, apparatus, electronic device, and readable storage medium.
[0005] Firstly, this disclosure provides a method for obtaining hand posture, including:
[0006] The first positions of the hand joints in multiple reference images are obtained; the multiple reference images include images obtained by mapping the hand joints based on multiple viewpoints when the hand is in a specific three-dimensional pose.
[0007] The second positions of the hand joints are obtained in multiple images to be processed; the multiple images to be processed include images of the hand obtained from multiple viewpoints, and the reference image and the image to be processed correspond one-to-one with the same viewpoint.
[0008] Least squares optimization (LM) is performed on the first position of the hand joint in the plurality of reference images and the second position of the hand joint in the plurality of images to be processed to obtain the adjusted second position corresponding to the hand joint.
[0009] The adjusted second position corresponding to the hand joint is mapped to a pre-established three-dimensional coordinate system to obtain the target three-dimensional pose of the hand.
[0010] As one possible implementation, the step of performing LM (Mean-Linguistic Modeling) on the hand joints at first positions in the plurality of reference images and second positions in the plurality of images to be processed to obtain the optimized positions corresponding to the hand joints includes:
[0011] The residual term is obtained based on the first position of the hand joint in the plurality of reference images and the second position of the hand joint in the plurality of images to be processed.
[0012] When the residual term is determined to be at its minimum value, the obtained position of the hand joint is the adjusted second position corresponding to the hand joint.
[0013] As one possible implementation, the hand joints include multiple joints; obtaining the first positions of the hand joints in multiple reference images includes:
[0014] When obtaining the specific three-dimensional pose, the position of each joint point in a first coordinate system; the first coordinate system is a three-dimensional coordinate system established based on the specific three-dimensional pose;
[0015] For each reference image, the position of the joint in the second coordinate system is obtained based on the position of the joint in the first coordinate system and the transformation relationship between the first coordinate system and the second coordinate system; wherein, the second coordinate system is the image coordinate system corresponding to the viewpoint of the reference image.
[0016] As one possible implementation, when obtaining the specific three-dimensional pose, the position of each joint point in the first coordinate system includes:
[0017] When acquiring the specific 3D pose, the position of each joint point in a first coordinate system; the first coordinate system is a world coordinate system established based on the multiple viewpoints.
[0018] For each reference image, the position of the joint in the second coordinate system is obtained based on the position of the joint in the first coordinate system and the transformation relationship between the first and second coordinate systems; wherein, the second coordinate system is the camera coordinate system of the viewpoint corresponding to the reference image;
[0019] For each viewpoint, based on the position of the joint point in the second coordinate system corresponding to the viewpoint and the corresponding camera parameters, the first position of the joint point in the reference image corresponding to the viewpoint is obtained.
[0020] As one possible implementation, when obtaining the specific three-dimensional pose, the position of each joint point in the first coordinate system includes:
[0021] Obtain the model data corresponding to the multiple joint points in the hand model of the specific three-dimensional pose;
[0022] For each joint, the position of the joint in the first coordinate system is obtained based on the model data corresponding to the joint and the model data corresponding to the associated joint; wherein, the associated joint is the parent node corresponding to the joint.
[0023] As one possible implementation, obtaining the position of the joint in the first coordinate system based on the model data corresponding to the joint and the model data corresponding to associated joints includes:
[0024] Based on the displacement data of the joint relative to the associated joint, the first rotation matrix, and the position of the associated joint in the third coordinate system, the position of the joint in the third coordinate system is obtained. The third coordinate system is a three-dimensional coordinate system established based on the hand model.
[0025] Based on the transformation relationship between the third coordinate system and the first coordinate system, the positions of each joint point in the first coordinate system are obtained;
[0026] The model data of the joint includes the rotation data of the joint and the displacement data of the joint relative to the associated joint; the first rotation matrix is the rotation matrix of the joint relative to the associated joint, and the first rotation matrix is obtained based on the rotation data of the joint and the corresponding first rotation matrix of the associated joint.
[0027] As one possible implementation, obtaining the position of the joint in the third coordinate system based on the displacement data of the joint relative to the associated joint, the first rotation matrix, and the position of the associated joint in the third coordinate system includes:
[0028] The displacement data of the joint point relative to the associated joint point is multiplied by the first rotation matrix, and the result of the multiplication is added to the position of the associated joint point in the third coordinate system to obtain the position of the joint point in the third coordinate system.
[0029] As one possible implementation, obtaining the position of the joint in the second coordinate system based on the position of the joint in the first coordinate system and the transformation relationship between the first and second coordinate systems includes:
[0030] The position of the joint in the camera coordinate system is obtained by multiplying the position of the joint in the first coordinate system with the second rotation matrix; the camera coordinate system is the camera coordinate system of the viewpoint corresponding to the second coordinate system; the second rotation matrix is the transformation matrix between the first coordinate system and the second coordinate system.
[0031] As one possible implementation, the specific three-dimensional pose is obtained by predicting based on the target three-dimensional pose of one or more previous hands.
[0032] Secondly, this disclosure provides a hand posture acquisition device, comprising:
[0033] An acquisition module is used to acquire the first positions of hand joints in multiple reference images; wherein, the multiple reference images include images obtained by mapping the hand joints based on multiple viewpoints when the hand is in a specific three-dimensional pose;
[0034] The acquisition module is further configured to acquire the second positions of the hand joints in multiple images to be processed; the multiple images to be processed include images of the hand obtained from multiple viewpoints, with a one-to-one correspondence between the reference image and the image to be processed at the same viewpoint;
[0035] The position adjustment module is used to perform least squares optimization (LM) on the first position of the hand joint in the plurality of reference images and the second position of the hand joint in the plurality of images to be processed, so as to obtain the adjusted second position corresponding to the hand joint.
[0036] The three-dimensional pose recognition module is used to map the adjusted second position corresponding to the hand joint points to a pre-established three-dimensional coordinate system to obtain the target three-dimensional pose of the hand.
[0037] Thirdly, this disclosure provides an electronic device, including: a memory and a processor;
[0038] The memory is configured to store computer program instructions;
[0039] The processor is configured to execute the computer program instructions, causing the electronic device to implement the hand gesture acquisition method as described in any of the first aspects.
[0040] Fourthly, this disclosure provides a readable storage medium, comprising: computer program instructions; at least one processor of an electronic device executes the computer program instructions, causing the electronic device to implement the hand gesture acquisition method as described in any of the first aspects.
[0041] Fifthly, this disclosure provides a computer program product that, when executed by a computer, enables the computer to implement the hand gesture acquisition method as described in any of the first aspects.
[0042] This disclosure provides a method, apparatus, electronic device, and readable storage medium for acquiring hand pose. The method acquires first positions of hand joints in multiple reference images and second positions of hand joints in multiple images to be processed. The multiple reference images include images obtained by mapping the hand joints to multiple viewpoints when the hand is in a specific 3D pose. The multiple images to be processed include images of the hand obtained from the multiple viewpoints, with a one-to-one correspondence between the images to be processed and the reference images. Least-squares optimization is performed based on the first and second positions corresponding to the hand joints to adjust the second positions of the hand joints in the multiple images to be processed. Based on the adjusted second positions of the hand joints in the multiple images to be processed, the target 3D pose of the hand is obtained. The method provided by this disclosure can improve the accuracy of the acquired 3D pose of the hand. Attached Figure Description
[0043] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0044] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 A schematic diagram illustrating an application scenario of the hand posture acquisition method provided in this embodiment of the disclosure;
[0046] Figure 2 A flowchart illustrating a hand posture acquisition method according to an embodiment of this disclosure;
[0047] Figure 3 A flowchart of a hand posture acquisition method provided in another embodiment of this disclosure;
[0048] Figure 4 A distribution diagram of various joints in the hand provided in an embodiment of this disclosure;
[0049] Figure 5 A schematic diagram of the structure of a hand posture acquisition device provided in an embodiment of this disclosure;
[0050] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation
[0051] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0052] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0053] For example, the hand gesture acquisition method provided in this disclosure can be executed by the hand gesture acquisition device provided in this disclosure, wherein the hand gesture acquisition device can be implemented by any software and / or hardware means. For example, the hand gesture acquisition device can be, but is not limited to, tablet computers, mobile phones (such as foldable screen phones, large screen phones, etc.), wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart TVs, smart screens, high-definition TVs, 4K TVs, smart speakers, smart projectors, and other Internet of Things (IoT) devices. This disclosure does not impose any restrictions on the specific type of electronic device.
[0054] Figure 1 This is a schematic diagram illustrating an application scenario of the hand posture acquisition method provided in an embodiment of this disclosure. (Refer to...) Figure 1 As shown, the scenario 100 provided in this embodiment includes: cameras 101 to 104 and a hand posture acquisition device 105.
[0055] Cameras 101 to 104 can be used to acquire images within their respective field of view and send the acquired images to the hand posture acquisition device 105.
[0056] This disclosure does not limit the resolution, model, or storage format of the acquired images in the cameras 101 to 104. It is understood that the better the camera parameters and performance, the higher the quality of the acquired images may be, and the more effective image information can be provided for the acquisition of the three-dimensional pose of the hand.
[0057] The hand pose acquisition device 105 is a device connected to cameras 101 to 104. The hand pose acquisition device 105 is able to receive images sent by cameras 101 to 104 respectively, and acquire the three-dimensional pose of the hand by executing the hand pose acquisition method provided in this disclosure.
[0058] Please continue reading. Figure 1 As shown, assuming Figure 1 The scene shown is a VR scene, and cameras 101 to 104 can be located in square areas (i.e., Figure 1 The positions of the four vertices of the area shown by the dashed line are shown in the diagram. Of course, the positions of the cameras can also be arranged in other ways, and this disclosure does not limit this. Figure 1 The illustration is merely an example. When a user is experiencing VR within the dotted-line area, cameras 101 to 104 can capture images of the user (including the user's hands) from four different perspectives. Then, cameras 101 to 104 can send the captured images to the hand pose acquisition device 105. The hand pose acquisition device 105 can obtain a highly accurate three-dimensional pose of the hand by executing the method provided in this disclosure. Afterwards, the control unit in the VR scene ( Figure 1 (Not shown in the image) It can also generate corresponding control commands based on the three-dimensional posture of the hand acquired by the hand posture acquisition device 105, and respond to the control commands to achieve interaction.
[0059] It should be noted that, Figure 1 The illustrated embodiment uses four cameras as an example. In actual applications, the number of cameras can be more or less. For example, the number of cameras can be three, five, six, etc. This disclosure does not limit this.
[0060] The hand posture acquisition method provided in this disclosure is described in detail below through several embodiments. The following embodiments use an electronic device executing the hand posture acquisition method as an example for illustration.
[0061] Figure 2 A flowchart illustrating a hand posture acquisition method according to an embodiment of this disclosure. (Refer to...) Figure 2 As shown, the method provided in this embodiment includes:
[0062] S201. Obtain the first positions of the hand joints in multiple reference images; the multiple reference images include images obtained by mapping the hand joints based on multiple viewpoints when the hand is in a specific three-dimensional pose.
[0063] Among them, the multiple reference images include images obtained by mapping the hand joints from multiple viewpoints when the hand is in a specific three-dimensional pose. For example, combined with... Figure 1In the illustrated embodiment, the multiple reference images may include: when the hand is in a specific three-dimensional pose, each based on... Figure 1 The four images shown are obtained by mapping each joint point of the hand from the four perspectives shown.
[0064] A specific 3D pose can be obtained by predicting the target 3D pose of one or more previous hands. It should be noted that user hand movements are typically continuous. Therefore, in practical applications, cameras at different viewpoints can acquire multiple consecutive images of the user's hand movements according to a preset method. For example, the preset method might involve periodically acquiring multiple consecutive images of the user's hand movements. Then, the target 3D pose of the first hand is jointly estimated based on the first frame captured by each camera. This first hand's target 3D pose can then be used as a specific 3D pose to acquire the target 3D pose of the second hand. Similarly, a specific 3D pose can be predicted based on the target 3D poses of the first and second hands, and used to acquire the target 3D pose of the third hand, and so on.
[0065] This disclosure does not limit the implementation method of an electronic device acquiring the first position of a hand joint in multiple reference images.
[0066] In one possible implementation, the electronic device can acquire a hand model in a specific three-dimensional pose and obtain model data of each joint of the hand at the required target scale. Assuming that the three-dimensional coordinate system used to unify the different viewpoints is the first coordinate system, the hand joints are mapped to the first coordinate system based on the model data of each hand joint, thus obtaining the position of each hand joint in the first coordinate system. Assuming that the image coordinate system corresponding to the reference image is the second coordinate system, it can be understood that the image coordinate system of the reference image corresponding to each viewpoint is different. For each reference image, the position of each hand joint in the first coordinate system can be mapped to the second coordinate system according to the transformation relationship between the first coordinate system and the corresponding second coordinate system, thereby obtaining the first position of the hand joint in the reference image.
[0067] It should be noted that the transformation relationship between the first coordinate system and the corresponding second coordinate system is a position transformation relationship, which can be represented by a transformation matrix.
[0068] The positions of each joint of the hand in the first coordinate system can be represented by the coordinate values of each joint in the first coordinate system. In the following text, this will be explained using... Figure 3 as well as Figure 4 The illustrated embodiment details how to calculate the position of the hand joints in the first coordinate system based on the model data corresponding to the hand joints.
[0069] S202. Obtain the second positions of the hand joints in multiple images to be processed, wherein the multiple images to be processed correspond one-to-one with the multiple reference images.
[0070] The electronic device can acquire multiple images to be processed and perform joint point detection on each image to obtain the second position of the hand joint point in each image. The second position of the hand joint point in the image can be represented by the coordinate value of the pixel corresponding to the hand joint point in the image coordinate system of the image to be processed.
[0071] For example, an electronic device can use a pre-trained machine learning model to analyze the image to be processed and obtain the second positions of the hand joints in each image to be processed. This disclosure does not limit the network structure, type, etc. of the machine learning model. For example, the machine learning model can be a deep neural network model, a convolutional neural network model, a machine learning model trained using the random forest algorithm, a decision tree model, etc.
[0072] S203. Based on the first position of the hand joint in multiple reference images and the second position of the hand joint in multiple images to be processed, perform least squares optimization to obtain the adjusted second position corresponding to the hand joint.
[0073] The electronic device can obtain the residual function corresponding to each joint point based on the one-to-one correspondence between multiple reference images and multiple images to be processed, as well as the first position of each hand joint point in the reference image and the second position of each hand joint point in the image to be processed. Based on the sum of the squares of the residual functions corresponding to each joint point, the residual term is obtained. When the residual term is minimized, the second position of each hand joint point in the multiple images to be processed is the adjusted second position corresponding to the hand joint point.
[0074] To make this step clearer, the following example uses a formula:
[0075] Suppose that the first position of the hand joint in the j-th reference image is (u j (i),v j (i)), the second position of the hand joint in the j-th image to be processed is (u j ′(i),v j ′(i)), where j is related to the viewpoint (or the camera corresponding to that viewpoint), and i is the index of the hand joint. Based on this, the residual term E can satisfy formula (1):
[0076]
[0077] Among them, the definition The θ that minimizes the residual term E is the adjusted second position corresponding to each joint of the hand. Where, u j (i)-u j ′(i) is the first residual function corresponding to joint point i, v j (i)-v j ′(i) is the second residual function corresponding to joint point i.
[0078] S204. Map the adjusted second position corresponding to the hand joints to the pre-established three-dimensional coordinate system to obtain the target three-dimensional pose of the hand.
[0079] Specifically, the adjusted second position corresponding to the hand joints is mapped to a pre-established three-dimensional coordinate system. Based on the connection relationship between multiple joints of the hand, the target three-dimensional pose of the hand can be identified and obtained.
[0080] In this embodiment, the method obtains the first positions of hand joints in multiple reference images and the second positions of hand joints in multiple images to be processed. The multiple reference images are obtained by mapping the hand joints to multiple viewpoints when the hand is in a specific 3D pose. The multiple images to be processed are images of the hand obtained from the multiple viewpoints, with a one-to-one correspondence between the images to be processed and the reference images. Least-squares optimization is performed based on the first and second positions of the hand joints to adjust the second positions of the hand joints in the multiple images to be processed. Based on the adjusted second positions of the hand joints in the multiple images to be processed, the 3D pose of the hand is obtained. The method provided in this embodiment optimizes pose information through multi-view observation, maximizing the matching of the state of multiple views, thereby improving the accuracy of the obtained 3D pose of the hand.
[0081] In the introduction Figure 3 Before introducing the embodiments shown, the coordinate systems involved will be described first:
[0082] 1. The first coordinate system is used to unify the three-dimensional coordinate system established from multiple perspectives.
[0083] 2. The second coordinate system, namely the camera coordinate system, is a three-dimensional coordinate system related to the viewpoint. Therefore, it can also be understood as the camera coordinate system corresponding to the viewpoint of the reference image. The second coordinate system can be established as a three-dimensional coordinate system with the center of focus of the corresponding viewpoint as the origin and the optical axis as the Z-axis.
[0084] 2. The third coordinate system, namely the model coordinate system, is a three-dimensional coordinate system established based on the hand model. The third coordinate system can be a left-hand three-dimensional coordinate system or a right-hand three-dimensional coordinate system, depending on whether the human hand targeted in this solution is a left-hand or a right-hand hand.
[0085] Figure 3 A flowchart illustrating a hand posture acquisition method according to an embodiment of this disclosure. (Refer to...) Figure 3 As shown, the method provided in this embodiment includes:
[0086] It should be noted that, in combination Figure 2 The embodiment shown, Figure 2 In the illustrated embodiment, step S201 can be implemented by steps S301 to S303 in this embodiment.
[0087] S301. Obtain the model data corresponding to the multiple joints in the hand model of the specific three-dimensional posture.
[0088] The hand includes multiple joints, among which... Figure 4 This is a schematic diagram illustrating the distribution of joint points in the human hand according to an embodiment of this disclosure. (Refer to...) Figure 4 As shown, taking the human right hand as an example, Figure 4 The solid black circles in the diagram represent joints on the hand. Therefore, the 20 joints on the human hand are: 0. Wrist joint; 1. Joint at the base of the thumb; 2. Joint at the middle of the thumb; 3. Joint at the tip of the thumb; 4. Joint at the base of the index finger; 5. Joint at the middle of the index finger; 6. Joint at the tip of the index finger; 7. Joint at the base of the middle finger; 8. Joint at the middle of the middle finger; 9. Joint at the middle of the middle finger; 10. Joint at the tip of the middle finger; 11. Joint at the base of the ring finger; 12. Joint at the middle of the ring finger; 13. Joint at the middle of the ring finger; 14. Joint at the tip of the ring finger; 15. Joint at the base of the little finger; 16. Joint at the middle of the little finger; 17. Joint at the middle of the little finger; 18. Joint at the tip of the little finger; 19.
[0089] It should be noted that, Figure 4 The diagram shows the distribution of joints in the human right hand. The division of joints in the human left hand is similar to that of the right hand, and can be referred to in the same way. For the sake of simplicity, it will not be repeated here.
[0090] Please continue to refer to Figure 4 In the embodiment shown, the degrees of freedom of the model data corresponding to the joints 0 to 19 are different.
[0091] The model data corresponding to wrist joint point 0 is 6-DOF data. The model data can include the position data (x, y, z) and rotation data (x, y, z, w) of wrist joint point 0. The rotation data is represented by quaternions.
[0092] It should be noted that the 6DOF data corresponding to wrist joint point 0 represents the state of the entire hand. Therefore, the position data of wrist joint point 0, position(x, y, z), can represent the position of wrist joint point 0 in the model coordinate system. In addition, as an important reference joint point of the hand, the position of wrist joint point 0 determines the position of the hand in the world coordinate system. Therefore, the position data of hand joint point 0 can also represent the position of wrist joint point 0 in the world coordinate system. Rotation(x, y, z, w) represents the rotation data of wrist joint point 0, where x, y, and z represent the rotation axes and w represents the rotation angle. Similarly, the rotation data of wrist joint point 0 can represent the rotation state of the hand in the model coordinate system, and also the rotation state of wrist joint point 0 in the world coordinate system.
[0093] For the joints at the base of the fingers, such as joint 1 at the base of the thumb, joint 4 at the base of the index finger, joint 8 at the base of the middle finger, joint 12 at the base of the ring finger, and joint 16 at the base of the little finger, the model data for the joints at the base of the fingers are all 2-DOF data, including rotational data. For the joints in the middle of the fingers, such as joint 2 at the middle of the thumb, joint 5 at the middle of the index finger, joint 6 at the middle of the index finger, joint 9 at the middle of the middle of the middle finger, joint 10 at the middle of the middle of the middle of the middle finger, joint 13 at the middle of the ring finger, joint 14 at the middle of the ring finger, joint 17 at the middle of the little finger, and joint 18 at the middle of the little finger, the model data for the joints in the middle of the fingers are 1-DOF data, including rotational information. The rotation data corresponding to the joints at the base of the finger and the joints in the middle of the finger can be represented by quaternions, that is, in the form of rotation(x, y, z, w).
[0094] It should be noted that, based on the analysis of the joints of the hand, the joints at the base of the fingers can swing left and right and bend inward. Therefore, the model data of the joints at the base of the fingers is 2DOF data. The joints in the middle of the fingers can only bend inward and cannot swing left and right. Therefore, the model data of the joints in the middle of the fingers is 1DOF data.
[0095] Furthermore, the rotation data for the finger root joint and the finger middle joint respectively represent the rotation data of the child joint relative to the parent joint, which is local rotation data. Therefore, the rotation data corresponding to these joints express the rotation of the joints in the first coordinate system.
[0096] exist Figure 4Based on the embodiment shown, joints 3, a7, a11, a15, and a19 are all fingertip joints. The pose of these fingertip joints is controlled by the associated parent node, and the fingertip joints have no child nodes. Therefore, the model data corresponding to the fingertip joints can include displacement data but not rotation data. That is, these joints do not have DOF data. However, these fingertip joints also need to have their positions calculated in the first coordinate system to participate in the subsequent least squares optimization.
[0097] right Figure 4 The distribution statistics of model data for each joint of the hand shown are as follows: a total of 26 DOF data points are distributed across 15 joint points.
[0098] S302. For each joint, based on the model data corresponding to the joint and the model data corresponding to the associated joints, obtain the position of the joint in the first coordinate system when it is in a specific three-dimensional pose.
[0099] Based on their connections, the parent-child relationships between multiple joints in the hand can be defined. For example, wrist joint 0 is the parent node of joint 4 at the base of the index finger, joint 4 at the base of the index finger is the parent node of joint 5 in the middle of the index finger, joint 5 in the middle of the index finger is the parent node of joint 6 in the middle of the index finger, and joint 6 in the middle of the index finger is the parent node of joint 7 at the tip of the index finger. The parent-child relationships between wrist joint 0 and the joints of the other fingers are similar to those between wrist joint 0 and the joints of the index finger in the previous example, and will not be repeated here for the sake of simplicity.
[0100] If multiple joints of the hand have a parent-child relationship, the position of each joint in the first coordinate system can be calculated based on the parent-child relationship.
[0101] As one possible implementation, the positions of each hand joint in the third coordinate system can be obtained first based on the model data corresponding to each joint and the parent-child relationship between the joints. The third coordinate system is a three-dimensional coordinate system established based on the hand model, and therefore, the third coordinate system can also be called the model coordinate system. Then, based on the transformation relationship between the third coordinate system and the first coordinate system, the positions of each hand joint in the third coordinate system are mapped to the first coordinate system to obtain the positions of each hand joint in the first coordinate system.
[0102] The positions of each joint of the hand in the third coordinate system can be obtained using the following formula (2):
[0103] p 子节点 =R 父节点 *T 子节点 +p 父节点 Formula (2)
[0104] Where, p 子节点 This indicates the position of the joint corresponding to the child node in the third coordinate system; p 父节点 Indicates the position of the key point corresponding to the parent node in the third coordinate system; T 子节点 This represents the displacement of the joint corresponding to the child node relative to the joint corresponding to the parent node, and can be obtained from the model data, including the displacement data of the joints. R 父节点 Let R be the first rotation matrix, representing the rotation matrix of the joint corresponding to the child node relative to the joint corresponding to the parent node, where R... 父节点 It can be obtained from the rotation data of the joints corresponding to the child nodes and the first rotation matrix corresponding to the parent node.
[0105] Next, taking the index finger as an example, we will explain how to obtain the positions of each joint of the index finger in the first coordinate system based on the model data of the joints and the model data of the associated joints.
[0106] 1. Joint point 4 at the base of the index finger
[0107] The position of key point 4 in the third index system can be obtained by formula (3):
[0108] p(4)=R(0)*T(4)+p(0) Formula (3)
[0109] Where p(4) represents the position of the joint point 4 at the base of the index finger in the third coordinate system.
[0110] p(0) represents the position information of wrist joint point 0, that is, position(x, y, z) in the 6DOF data corresponding to the wrist joint point.
[0111] R(0) represents the rotation matrix of the joint point 4 at the base of the index finger relative to the joint point 0 of the wrist, which corresponds to the first rotation matrix mentioned above; T(4) represents the displacement of the joint point at the base of the index finger relative to the joint point of the wrist. When obtaining a hand model in a specific three-dimensional pose, T(4) can be obtained from the model data, and T(4) can be a constant.
[0112] 2. Joint point 5 in the middle of the index finger
[0113] The position of key point 5 in the third coordinate system can be obtained by formula (4):
[0114]
[0115] Where p(5) represents the position of the joint point 5 in the middle of the index finger in the third coordinate system.
[0116] p(4) represents the position information of the joint point 4 at the base of the index finger in the third coordinate system.
[0117] R(4) represents the rotation matrix of the joint point 5 in the middle of the index finger relative to the joint point 4 in the middle of the index finger, which corresponds to the first rotation matrix mentioned above; r(4) represents the rotation matrix corresponding to the rotation data (1DOF data) of the joint point 4 in the middle of the index finger.
[0118] T(5) represents the displacement of joint 5 in the middle of the index finger relative to joint 4 at the base of the index finger. T(5) can be obtained from the model data when acquiring a hand model in a specific three-dimensional pose, and T(5) can be a constant.
[0119] 3. The joint point in the middle of the index finger (6)
[0120] The position of key point 6 in the third coordinate system can be obtained by formula (5):
[0121]
[0122] Where p(6) represents the position of the joint point 6 in the middle of the index finger in the third coordinate system.
[0123] p(5) represents the position information of the joint point 5 at the base of the index finger in the third coordinate system.
[0124] R(5) represents the rotation matrix of the joint point 6 in the middle of the index finger relative to the joint point 5 in the middle of the index finger, which corresponds to the first rotation matrix mentioned above; r(5) represents the rotation matrix corresponding to the rotation data (1DOF data) of the joint point 5 in the middle of the index finger.
[0125] T(6) represents the displacement of joint 6 in the middle of the index finger relative to joint 5 in the middle of the index finger. When obtaining a hand model in a specific three-dimensional pose, T(6) can be obtained from the model data and can be a constant.
[0126] 4. The joint point at the tip of the index finger (7)
[0127] The position of key point 7 in the third coordinate system can be obtained by formula (6):
[0128]
[0129] Where p(7) represents the position of the joint point 7 at the tip of the index finger in the third coordinate system.
[0130] p(6) represents the position information of the joint point 6 in the middle of the index finger in the third coordinate system.
[0131] R(6) represents the rotation matrix of the joint point 7 at the tip of the index finger relative to the joint point 6 in the middle of the index finger, which corresponds to the aforementioned second rotation matrix; r(6) represents the rotation matrix corresponding to the rotation data (1DOF data) of the joint point 6 in the middle of the index finger.
[0132] T(7) represents the displacement of the joint 7 at the tip of the index finger relative to the joint 6 in the middle of the index finger. T(7) can be obtained from the model data when acquiring a hand model in a specific three-dimensional pose, and T(7) can be a constant.
[0133] The positions of the joints of the index finger in the third coordinate system can be obtained using the above method. The method for obtaining the positions of the joints of the other fingers in the third coordinate system is similar to that for the index finger, and will not be repeated here for simplicity.
[0134] It should be noted that although the thumb has one less joint point than the index finger, the calculation process is similar. The parent-child relationship between the joint points can be substituted into formula (1) for calculation.
[0135] By performing the above calculations for each joint, the position of each hand joint in the third coordinate system can be obtained when the hand is in a specific three-dimensional pose.
[0136] Next, based on the transformation relationship between the third coordinate system and the first coordinate system, for example, when this transformation relationship is represented by a transformation matrix, the position of each hand joint in the first coordinate system can be obtained by multiplying the position of the hand joint in the third coordinate system with the transformation matrix.
[0137] Assuming the transformation matrix between the third coordinate system and the first coordinate system is M1, and the position of the hand joint in the third coordinate system is p(i), then the position of the hand joint in the first coordinate system is p′(i) = M1*p(i).
[0138] In some cases, the first coordinate system and the third coordinate system are aligned, which can also be understood as no transformation is needed between the first coordinate system and the third coordinate system. In this case, the position of the hand joint in the third coordinate system is the same as the position of the hand joint in the first coordinate system.
[0139] S303. For each reference image, based on the positions of multiple joints of the hand in the first coordinate system and the transformation relationship between the first coordinate system and the corresponding second coordinate system, obtain the positions of the multiple joints of the hand in the second coordinate system, where the second coordinate system is the camera coordinate system corresponding to the viewpoint of the reference image.
[0140] The purpose of this step is to complete the multi-view projection, that is, to map the positions of each joint of the hand in the first coordinate system to the images corresponding to each view. This can be achieved by using a second rotation matrix to represent the transformation relationship between the first coordinate system and the second coordinate system.
[0141] For example, suppose the transformation matrix between the world coordinate system and the camera coordinate system corresponding to the j-th viewpoint is K.j If the position of the hand joint in the world coordinate system is p′(i), then the position of the hand joint in the camera coordinate system corresponding to the j-th viewpoint is p. j (i)=K j *p′(i).
[0142] It should be noted that, for each reference image from different viewpoints, the corresponding second rotation matrix is the transformation matrix K. j .
[0143] S304. For each reference image, based on the camera parameters corresponding to the viewpoint and the positions of each hand joint in the second coordinate system corresponding to the viewpoint, obtain the first position of each hand joint in the reference image.
[0144] In this step, the camera parameters corresponding to the viewpoint include camera intrinsic parameters, such as the camera's focal length and the camera's center offset.
[0145] Assume the camera intrinsic parameter corresponding to a viewpoint is (f xj ,f yj ,cu j ,cv j ), where f xj f yj cu represents the focal length parameter of the camera at the j-th viewpoint. j ,cv j This represents the center offset of the camera corresponding to the j-th viewpoint. Therefore, the first position of each joint of the hand in the reference image corresponding to the j-th viewpoint can be calculated using the following formula (7).
[0146] Formula (7) is as follows:
[0147]
[0148] For each viewpoint (i.e. each reference image), the first position of each joint of the hand in each reference image can be obtained by calculating according to the above formula (7).
[0149] S305. Obtain the second positions of the hand joints in multiple images to be processed, wherein the multiple images to be processed correspond one-to-one with the multiple reference images.
[0150] S306. Based on the first position of the hand joint in multiple reference images and the second position of the hand joint in multiple images to be processed, perform least squares optimization to obtain the adjusted second position corresponding to the hand joint.
[0151] S307. Map the adjusted second position corresponding to the hand joints to the pre-established three-dimensional coordinate system to obtain the target three-dimensional pose of the hand.
[0152] In this embodiment, steps S305 to S307 are respectively connected with... Figure 2 In the illustrated embodiment, steps S202 to S204 are similar and can be referred to the foregoing. Figure 2 The detailed descriptions of the illustrated embodiments are omitted here for the sake of brevity.
[0153] It should be noted that, in conjunction with the foregoing Figure 4 In the illustrated embodiment, the DOF data for each joint of the hand are distributed across 15 joints, totaling 26 DOF data points. Therefore, combined with... Figure 2 In the illustrated embodiment, as shown in formula (1) in step S203, θ can be 26 DOF data points when the optimized residual term E is minimized, corresponding to... Figure 4 The distribution of DOF data for each joint in the illustrated embodiment.
[0154] Based on the adjusted 26 DOF data points, combined with the aforementioned parent-child relationships between hand joints, the positions of the hand joints in the preset three-dimensional coordinate system can be obtained, thereby obtaining the target three-dimensional pose of the hand.
[0155] Furthermore, as mentioned above, the first coordinate system is a three-dimensional coordinate system established to unify views from multiple perspectives. Therefore, the preset three-dimensional coordinate system in S307 can be the aforementioned first coordinate system.
[0156] The method provided in this embodiment obtains first positions of hand joints in multiple reference images and second positions of hand joints in multiple images to be processed. The multiple reference images include images obtained by mapping the hand joints based on multiple viewpoints when the hand is in a specific 3D pose. The multiple images to be processed include images of the hand obtained from the multiple viewpoints, with a one-to-one correspondence between the images to be processed and the reference images. Least-squares optimization is performed based on the first and second positions corresponding to the hand joints to adjust the second positions of the hand joints in the multiple images to be processed. Based on the adjusted second positions of the hand joints in the multiple images to be processed, the target 3D pose of the hand is obtained. The method provided in this disclosure optimizes DOF data through multi-view observation, which can match the state of multiple views to the greatest extent, thereby improving the accuracy of the obtained 3D pose of the hand. Furthermore, driving the hand pose with DOF data not only avoids generating 3D hand poses that do not conform to the rules but also reduces rendering complexity.
[0157] For example, this disclosure provides a hand posture acquisition device.
[0158] Figure 5 This is a schematic diagram of the structure of a hand posture acquisition device provided in an embodiment of the present disclosure.
[0159] Reference Figure 5 As shown, the hand posture acquisition device 500 provided in this embodiment includes:
[0160] The acquisition module 501 is used to acquire the first positions of the hand joints in multiple reference images; wherein, the multiple reference images include images obtained by mapping the hand joints based on multiple viewpoints when the hand is in a specific three-dimensional pose.
[0161] The acquisition module 501 is further configured to acquire the second positions of the hand joints in multiple images to be processed; the multiple images to be processed include images of the hand obtained from multiple viewpoints, with a one-to-one correspondence between the reference image and the image to be processed at the same viewpoint.
[0162] The position adjustment module 502 is used to perform LM (Learning Model) on the hand joints in the plurality of reference images and the hand joints in the plurality of images to be processed to obtain the adjusted second position corresponding to the hand joints.
[0163] The three-dimensional pose recognition module 503 is used to map the adjusted second position corresponding to the hand joint point to a pre-established three-dimensional coordinate system to obtain the target three-dimensional pose of the hand.
[0164] As one possible implementation, the position adjustment module 502 is specifically used to obtain a residual term based on the first position of the hand joint in the plurality of reference images and the second position of the hand joint in the plurality of images to be processed; when the residual term is determined to be the minimum value, the obtained position of the hand joint is the adjusted second position corresponding to the hand joint.
[0165] In one possible implementation, the hand joints include multiple joints; the acquisition module 501 is specifically used to acquire the position of each joint in a first coordinate system when the specific three-dimensional pose is assumed; the first coordinate system is a world coordinate system established based on the multiple viewpoints; for each reference image, the position of the joint in the second coordinate system is acquired based on the position of the joint in the first coordinate system and the transformation relationship between the first coordinate system and the second coordinate system; wherein, the second coordinate system is the camera coordinate system of the viewpoint corresponding to the reference image; for each viewpoint, the first position of the joint in the reference image corresponding to the viewpoint is acquired based on the position of the joint in the second coordinate system corresponding to the viewpoint and the corresponding camera parameters.
[0166] As one possible implementation, the acquisition module 501 is specifically used to acquire model data corresponding to the plurality of joints in the hand model of the specific three-dimensional pose; for each joint, the position of the joint in the first coordinate system is acquired based on the model data corresponding to the joint and the model data corresponding to the associated joint; wherein, the associated joint is the parent node corresponding to the joint.
[0167] As one possible implementation, the acquisition module 501 is specifically used to acquire the position of the joint point in the third coordinate system based on the displacement data of the joint point relative to the associated joint point, the first rotation matrix, and the position of the associated joint point in the third coordinate system, wherein the third coordinate system is a three-dimensional coordinate system established based on the hand model; and to acquire the position of each joint point in the first coordinate system based on the transformation relationship between the third coordinate system and the first coordinate system.
[0168] The model data of the joint includes the rotation data of the joint and the displacement data of the joint relative to the associated joint; the first rotation matrix is the rotation matrix of the joint relative to the associated joint, and the first rotation matrix is obtained based on the rotation data of the joint and the corresponding first rotation matrix of the associated joint.
[0169] As one possible implementation, the acquisition module 501 is specifically used to multiply the displacement data of the joint point relative to the associated joint point with the first rotation matrix, and add the result of the multiplication to the position of the associated joint point in the third coordinate system to obtain the position of the joint point in the third coordinate system.
[0170] As one possible implementation, the acquisition module 501 is specifically used to acquire the position of the joint point in the camera coordinate system based on the product of the position of the joint point in the first coordinate system and the second rotation matrix; the camera coordinate system is the camera coordinate system of the viewpoint corresponding to the second coordinate system; the second rotation matrix is the transformation matrix between the first coordinate system and the second coordinate system.
[0171] As one possible implementation, the specific three-dimensional pose is obtained by predicting based on the target three-dimensional pose of one or more previous hands.
[0172] The hand posture acquisition device provided in this embodiment can be used to execute the technical solution of any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and can be referred to the detailed description of the foregoing method embodiments. For the sake of brevity, it will not be repeated here.
[0173] Figure 6This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. (Refer to...) Figure 6 As shown, the electronic device 600 provided in this embodiment includes a memory 601 and a processor 602.
[0174] The memory 601 can be a separate physical unit, connected to the processor 602 via a bus 603. Alternatively, the memory 601 and processor 602 can be integrated together, implemented in hardware, etc.
[0175] The memory 601 is used to store program instructions, and the processor 602 calls the program instructions to execute the technical solutions of any of the above method embodiments.
[0176] Optionally, when some or all of the methods in the above embodiments are implemented by software, the electronic device 600 may also include only the processor 602. The memory 601 for storing programs is located outside the electronic device 600, and the processor 602 is connected to the memory via circuits / wires for reading and executing the programs stored in the memory.
[0177] The processor 602 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP.
[0178] The processor 602 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0179] The memory 601 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); the memory may also include combinations of the above types of memory.
[0180] This disclosure also provides a readable storage medium, including: computer program instructions; when executed by at least one processor of an electronic device, the computer program instructions implement the hand posture acquisition method shown in any of the above method embodiments.
[0181] This disclosure also provides a computer program product, which, when executed by a computer, enables the computer to implement the hand posture acquisition method shown in any of the above method embodiments.
[0182] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0183] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for obtaining hand posture, characterized in that, include: The first positions of the hand joints in multiple reference images are obtained; the multiple reference images include images obtained by mapping the hand joints based on multiple viewpoints when the hand is in a specific three-dimensional pose predicted based on the target three-dimensional pose of the previous one or more hands. The second positions of the hand joints are obtained in multiple images to be processed; the multiple images to be processed include images of the hand obtained from multiple viewpoints, and the reference image and the image to be processed correspond one-to-one with the same viewpoint. Least squares optimization (LM) is performed on the first position of the hand joint in the plurality of reference images and the second position of the hand joint in the plurality of images to be processed to obtain the adjusted second position corresponding to the hand joint. The adjusted second position corresponding to the hand joint is mapped to a pre-established three-dimensional coordinate system to obtain the target three-dimensional pose of the hand.
2. The method according to claim 1, characterized in that, The step of performing LM (Learning by Mean Square) on the hand joints at their first positions in the plurality of reference images and their second positions in the plurality of images to be processed, to obtain the optimized positions corresponding to the hand joints, includes: The residual term is obtained based on the first position of the hand joint in the plurality of reference images and the second position of the hand joint in the plurality of images to be processed. When the residual term is determined to be at its minimum value, the obtained position of the hand joint is the adjusted second position corresponding to the hand joint.
3. The method according to claim 1 or 2, characterized in that, The hand joints include multiple joints; obtaining the first positions of the hand joints in multiple reference images includes: When acquiring the specific 3D pose, the position of each joint point in a first coordinate system; the first coordinate system is a world coordinate system established based on the multiple viewpoints. For each reference image, the position of the joint in the second coordinate system is obtained based on the position of the joint in the first coordinate system and the transformation relationship between the first and second coordinate systems; wherein, the second coordinate system is the camera coordinate system of the viewpoint corresponding to the reference image; For each viewpoint, based on the position of the joint point in the second coordinate system corresponding to the viewpoint and the corresponding camera parameters, the first position of the joint point in the reference image corresponding to the viewpoint is obtained.
4. The method according to claim 3, characterized in that, When obtaining the specific three-dimensional pose, the position of each joint point in the first coordinate system includes: Obtain the model data corresponding to the multiple joints in the hand model of the specific three-dimensional pose; For each joint, the position of the joint in the first coordinate system is obtained based on the model data corresponding to the joint and the model data corresponding to the associated joint; wherein, the associated joint is the parent node corresponding to the joint.
5. The method according to claim 4, characterized in that, The step of obtaining the position of the joint in the first coordinate system based on the model data corresponding to the joint and the model data corresponding to the associated joint includes: Based on the displacement data of the joint relative to the associated joint, the first rotation matrix, and the position of the associated joint in the third coordinate system, the position of the joint in the third coordinate system is obtained. The third coordinate system is a three-dimensional coordinate system established based on the hand model. Based on the transformation relationship between the third coordinate system and the first coordinate system, the positions of each joint point in the first coordinate system are obtained; The model data of the joint includes the rotation data of the joint and the displacement data of the joint relative to the associated joint; the first rotation matrix is the rotation matrix of the joint relative to the associated joint, and the first rotation matrix is obtained based on the rotation data of the joint and the corresponding first rotation matrix of the associated joint.
6. The method according to claim 5, characterized in that, The step of obtaining the position of the joint in the third coordinate system based on the displacement data of the joint relative to the associated joint, the first rotation matrix, and the position of the associated joint in the third coordinate system includes: The displacement data of the joint point relative to the associated joint point is multiplied by the first rotation matrix, and the result of the multiplication is added to the position of the associated joint point in the third coordinate system to obtain the position of the joint point in the third coordinate system.
7. The method according to claim 3, characterized in that, The step of obtaining the position of the joint in the second coordinate system based on the position of the joint in the first coordinate system and the transformation relationship between the first and second coordinate systems includes: The position of the joint in the camera coordinate system is obtained by multiplying the position of the joint in the first coordinate system with the second rotation matrix; the camera coordinate system is the camera coordinate system of the viewpoint corresponding to the second coordinate system; the second rotation matrix is the transformation matrix between the first coordinate system and the second coordinate system.
8. A hand posture acquisition device, characterized in that, include: An acquisition module is used to acquire the first positions of hand joints in multiple reference images; wherein, the multiple reference images include images obtained by mapping the hand joints based on multiple viewpoints when the hand is a specific three-dimensional pose predicted based on the target three-dimensional pose of one or more previous hands. The acquisition module is further configured to acquire the second positions of the hand joints in multiple images to be processed; the multiple images to be processed include images of the hand obtained from multiple viewpoints, with a one-to-one correspondence between the reference image and the image to be processed at the same viewpoint; The position adjustment module is used to perform least squares optimization (LM) on the first position of the hand joint in the plurality of reference images and the second position of the hand joint in the plurality of images to be processed, so as to obtain the adjusted second position corresponding to the hand joint. The three-dimensional pose recognition module is used to map the adjusted second position corresponding to the hand joint points to a pre-established three-dimensional coordinate system to obtain the target three-dimensional pose of the hand.
9. An electronic device, characterized in that, include: Memory and processor; The memory is configured to store computer program instructions; The processor is configured to execute the computer program instructions, causing the electronic device to implement the hand posture acquisition method as described in any one of claims 1 to 7.
10. A readable storage medium, characterized in that, include: Computer program instructions; At least one processor of the electronic device executes the computer program instructions, causing the electronic device to implement the hand gesture acquisition method as described in any one of claims 1 to 7.