A method and electronic device for predicting the pose of a 3D model

Abstract

This application provides a method and electronic device for predicting the pose of a 3D model, used to improve the accuracy of pose prediction. The method includes: segmenting an image containing a target object into a hand region using a preset algorithm, obtaining a hand region image and a human body region image excluding the hand region; extracting features from the hand region image and the human body region image respectively, obtaining multiple hand feature maps and human body feature maps of different resolutions; predicting the 3D pose of the hand and human body respectively based on the initial 3D pose data of the region image and each feature map of the region image for any one of the hand region image and the human body region image, obtaining 3D pose data of the hand and 3D pose data of the human body; and stitching the 3D keypoints of the hand and the human body together by the rotation amount between each 3D keypoint in each 3D pose data to obtain the pose of the 3D model of the target object.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and more particularly to a method and electronic device for predicting the pose of a 3D model. Background Technology

[0002] In recent years, with the maturity of internet technology, computer vision technology has also made significant progress. Using human animation data captured by cameras, understanding human behavior and actions from images, and performing technical analysis of image data are key issues in computer vision technology. Human pose estimation is also one of the application directions of computer vision. The computer's computing platform needs to calculate the positions of the human body's joints in image data in real time to achieve human pose estimation.

[0003] Existing methods for estimating human pose often fail to accurately predict hand movements due to the small size of the hand portion in an image. This results in low accuracy in predicting human pose. Summary of the Invention

[0004] This application provides a method and electronic device for predicting the pose of a 3D model, which can improve the accuracy of predicting hand movements and further improve the accuracy of predicting human poses.

[0005] In a first aspect, embodiments of this application provide a method for predicting the pose of a 3D model, comprising:

[0006] A preset algorithm is used to segment the hand region of an image containing the target object, resulting in a hand region image and a human body region image excluding the hand region.

[0007] Feature extraction is performed on the hand region image and the human body region image respectively to obtain a multi-level pyramid feature map of the hand region image and a multi-level pyramid feature map of the human body region image, wherein the multi-level pyramid feature map includes multiple feature maps with different resolutions.

[0008] For any one of the hand region image and the human body region image, the three-dimensional pose of the hand or human body is predicted based on the initial three-dimensional pose data of the region image and the multi-level pyramid feature map of the region image, so as to obtain the three-dimensional pose data of the hand or the three-dimensional pose data of the human body.

[0009] By using the rotation amounts between the three-dimensional key points in each three-dimensional pose data, the three-dimensional key points of the hand are stitched together with the three-dimensional key points of the human body to obtain the pose of the three-dimensional model of the target object.

[0010] A second aspect of this application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0011] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0012] A preset algorithm is used to segment the hand region of an image containing the target object, resulting in a hand region image and a human body region image excluding the hand region.

[0013] Feature extraction is performed on the hand region image and the human body region image respectively to obtain a multi-level pyramid feature map of the hand region image and a multi-level pyramid feature map of the human body region image, wherein the multi-level pyramid feature map includes multiple feature maps with different resolutions.

[0014] For any one of the hand region image and the human body region image, the three-dimensional pose of the hand or human body is predicted based on the initial three-dimensional pose data of the region image and the multi-level pyramid feature map of the region image, so as to obtain the three-dimensional pose data of the hand or the three-dimensional pose data of the human body.

[0015] By using the rotation amounts between the three-dimensional key points in each three-dimensional pose data, the three-dimensional key points of the hand are stitched together with the three-dimensional key points of the human body to obtain the pose of the three-dimensional model of the target object.

[0016] According to a third aspect of the present invention, a computer storage medium is provided, the computer storage medium storing a computer program for performing the method as described in the first aspect.

[0017] In the above embodiments of this application, the hand region of an image containing the target object is segmented, and then features are extracted from the segmented hand region image and the human body region image to obtain multiple feature maps of different resolutions. Then, based on the obtained feature maps of different resolutions, the 3D pose of the human body is estimated, resulting in 3D pose data for the hand and the human body. Finally, by using the rotation amounts between the 3D keypoints in each 3D pose data, the 3D keypoints of the hand and the human body are stitched together to obtain the pose of the 3D model of the target object. Therefore, in this embodiment, by cropping the hand and extracting features at different scales, richer features of the hand region are obtained. Then, based on the extracted multi-scale features, the hand pose is determined, making the determined hand pose more accurate. Finally, the hand and the human body are stitched together to obtain the pose of the 3D model, thereby improving the accuracy of the predicted pose. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 An exemplary illustration shows one of the application scenarios provided in the embodiments of this application;

[0020] Figure 2 The second example illustration shows a schematic diagram of an application scenario provided in an embodiment of this application;

[0021] Figure 3 One of the flowcharts illustrating a method for predicting the pose of a 3D model provided in an embodiment of this application is shown as an example;

[0022] Figure 4 An exemplary schematic diagram illustrates the process of determining three-dimensional pose data of body parts provided in an embodiment of this application;

[0023] Figure 5A An exemplary illustration shows a flowchart for determining three-dimensional pose data of a hand, provided in an embodiment of this application.

[0024] Figure 5B An exemplary illustration shows a flowchart for determining three-dimensional pose data of a human body according to an embodiment of this application;

[0025] Figure 6 An exemplary illustration is provided in an embodiment of this application, which is a schematic diagram of the SMPL skeleton;

[0026] Figure 7 An exemplary illustration shows a schematic diagram of the skeleton of the smplx human body parameterization model provided in an embodiment of this application;

[0027] Figure 8 An exemplary illustration shows a schematic diagram of the skeleton of the left and right hands provided in an embodiment of this application;

[0028] Figure 9 An exemplary illustration is provided in an embodiment of this application, showing a spliced ​​hand and human body.

[0029] Figure 10 The illustration shows a specific flowchart of the process for optimizing the pose of a 3D model according to an embodiment of this application;

[0030] Figure 11 A second flowchart of a method for predicting the pose of a 3D model provided in an embodiment of this application is illustrated.

[0031] Figure 12 An exemplary schematic diagram of the structure of the apparatus for predicting the pose of a 3D model provided in an embodiment of this application is shown;

[0032] Figure 13 An exemplary hardware structure diagram of the calibration device provided in an embodiment of this application is shown. Detailed Implementation

[0033] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0034] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0035] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0036] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to be omnipresent but not exclusive; for example, a product or device comprising a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device.

[0037] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0038] The following is an overview of the ideas behind the embodiments of this application.

[0039] Current human pose estimation techniques cannot accurately predict hand movements because the hand portion is too small in a single image. This results in a low accuracy rate for predicted human poses.

[0040] To address the issue of low accuracy in predicting human pose in existing technologies, this application provides a method for predicting the pose of a 3D model. The method involves segmenting an image containing the target object into a hand region, extracting features from the segmented hand and human body regions to obtain multiple feature maps of different resolutions, estimating the 3D pose of the human body based on these feature maps, and obtaining 3D pose data for both the hand and the human body. Finally, by adjusting the rotation between 3D keypoints in each 3D pose data set, the 3D keypoints of the hand and the human body are stitched together to obtain the pose of the 3D model of the target object. Therefore, this embodiment improves the accuracy of the predicted pose by cropping the hand and extracting features at different scales, thus obtaining richer features in the hand region. The hand pose is then determined based on these extracted multi-scale features, resulting in a more accurate prediction. Finally, the hand and human body are stitched together to obtain the pose of the 3D model.

[0041] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0042] Figure 1 An exemplary illustration shows a schematic diagram of an application scenario provided in an embodiment of this application; such as Figure 1 As shown, this application scenario uses an electronic device as a server as an example. This application scenario includes a camera 110 and a server 120. Server 120 can be implemented using a single server or multiple servers. Server 120 can be implemented using a physical server or a virtual server.

[0043] In one possible application scenario, after camera 110 acquires an image containing the target object, it sends the image containing the target object to server 120. Server 120 uses a preset algorithm to segment the hand region of the image containing the target object, obtaining a hand region image and a human body region image excluding the hand region. It then performs feature extraction on the hand region image and the human body region image respectively, obtaining multi-level pyramid feature maps for the hand region image and the human body region image. The multi-level pyramid feature maps include multiple feature maps with different resolutions. Then, server 120... For any one of the hand region image and the human body region image, the three-dimensional pose of the hand or the human body is predicted based on the initial three-dimensional pose data of the region image and the multi-level pyramid feature map of the region image, respectively, to obtain the three-dimensional pose data of the hand or the three-dimensional pose data of the human body. The three-dimensional pose data includes each three-dimensional keypoint and the rotation between each three-dimensional keypoint. Finally, the server 120 stitches the three-dimensional keypoints of the hand and the three-dimensional keypoints of the human body together using the rotation between each three-dimensional keypoint in the three-dimensional pose data to obtain the pose of the three-dimensional model of the target object.

[0044] like Figure 2 The diagram illustrates another application scenario, which includes a camera 110, a server 120, and a memory 130. After acquiring an image containing a target object, the camera 110 sends the image to the server 120. The server 120 uses a preset algorithm to segment the hand region of the image containing the target object, obtaining a hand region image and a human body region image excluding the hand region. Feature extraction is then performed on both the hand region image and the human body region image to obtain multi-level pyramid feature maps for the hand region image and the human body region image, respectively. The multi-level pyramid feature maps include multiple feature maps with different resolutions. Then, the server 120 analyzes the hand region image and the human body region image... For any region image in the image, the three-dimensional pose of the hand or the human body is predicted based on the initial three-dimensional pose data of the region image and the multi-level pyramid feature map of the region image, respectively, to obtain the three-dimensional pose data of the hand or the three-dimensional pose data of the human body. The three-dimensional pose data includes each three-dimensional key point and the rotation amount between each three-dimensional key point. Finally, the server 120 stitches the three-dimensional key points of the hand and the three-dimensional key points of the human body together using the rotation amount between each three-dimensional key point in the three-dimensional pose data to obtain the pose of the three-dimensional model of the target object, and stores the pose of the three-dimensional model of the target object in the memory 130.

[0045] The description in this application focuses only on a single camera 110, a single server 120, and a single memory 130. However, those skilled in the art should understand that the illustrated camera 110, single server 120, and single memory 130 are intended to illustrate the operation of the camera 110, single server 120, and single memory 130 involved in the technical solutions of this application, and are not intended to imply any limitation on the number, type, or location of the camera 110, single server 120, and single memory 130. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application.

[0046] It should be noted that the method for predicting the pose of a 3D model proposed in this application is not only applicable to... Figure 1 and Figure 2 The application scenarios shown can also be applied to any device that predicts the pose of a 3D model.

[0047] The following describes a method for predicting the pose of a 3D model using exemplary embodiments of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the application scenarios described above are only shown to facilitate understanding of the methods and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0048] like Figure 3 The diagram shown illustrates a method for predicting the pose of a 3D model, including the following steps:

[0049] Step 301: Use a preset algorithm to segment the hand region of the image containing the target object to obtain a hand region image and a human body region image other than the hand region;

[0050] In one implementation, step 301 can be implemented as follows: using a region detection algorithm to detect the hand region in an image containing the target object, to obtain the position of the hand region in the image, wherein the hand region includes a left hand region and a right hand region; cropping the target image according to the position of the hand region to obtain the hand region image and the human body region image.

[0051] In this embodiment, the position of the hand region is represented using a bounding box. Therefore, when cropping the hand region, the cropping can be performed according to the bounding box of the hand region to obtain the hand region image and the human body region image. Furthermore, the image containing the target object in this embodiment is taken with a standard RGB (RGB color mode) camera, with a resolution of 720p or 1080p. Since a 1080p camera has lower resolution but higher operating efficiency, it is recommended to use a 1080p camera for taking pictures. However, the specific camera selection can be set according to the actual situation, and this embodiment does not limit the camera or resolution used.

[0052] It should be noted that the region detection algorithm in this embodiment can be set according to the actual situation, and this embodiment does not limit the specific method of the region detection algorithm.

[0053] To further improve the accuracy of hand pose prediction, in one embodiment, the hand region image is upsampled using a bilinear interpolation algorithm before performing step 302.

[0054] This results in richer images of the hand region, enabling the extraction of more features and improving the accuracy of hand pose prediction.

[0055] Step 302: Perform feature extraction on the hand region image and the human body region image respectively to obtain a multi-level pyramid feature map of the hand region image and a multi-level pyramid feature map of the human body region image, wherein the multi-level pyramid feature map includes multiple feature maps with different resolutions;

[0056] It should be noted that the feature extraction network used in this embodiment is ResNet50 to extract features from the hand region image and the human body region image respectively. This embodiment uses a high-resolution feature map extraction network because using a high-resolution feature extraction network will yield multiple layers of spatial feature maps at different scales, i.e., a multi-level pyramid feature map in this embodiment. However, the specific feature extraction network can be set according to the actual situation, and this embodiment does not limit the feature extraction network.

[0057] Step 303: For any one of the hand region image and the human body region image, predict the three-dimensional pose of the hand or the human body based on the initial three-dimensional pose data of the region image and the multi-level pyramid feature map of the region image, respectively, to obtain the three-dimensional pose data of the hand or the three-dimensional pose data of the human body, wherein the three-dimensional pose data includes each three-dimensional key point and the rotation amount between each three-dimensional key point.

[0058] Specifically, if the image region is a hand region image, the obtained 3D pose data is the 3D pose data of the hand. If the image region is a human body region image, the obtained 3D pose data is the 3D pose data of the human body. The body parts included in the obtained 3D pose data are the same as those included in the region image.

[0059] The following section details the methods for determining the three-dimensional pose data of the hand and the human body, such as... Figure 4 The diagram illustrates the process for determining the 3D pose data of a body part (human body or hand), including the following steps:

[0060] Step 401: Traverse the multi-level pyramid feature maps of the region image in ascending order of image resolution;

[0061] Step 402: For any feature map that has been traversed, input the feature map and the target 3D pose data corresponding to the feature map into the 3D pose data prediction network to obtain the predicted 3D pose data, wherein the target 3D pose data includes the initial 3D pose data or the predicted 3D pose data obtained from the previous traversal.

[0062] In one embodiment, the target 3D pose data corresponding to the feature map is determined in the following manner:

[0063] Method 1: If the feature map is the initial feature map to be traversed, then the target 3D pose data corresponding to the feature map is the initial 3D pose data. The initial 3D pose data is pre-set.

[0064] like Figure 5A As shown in the figure, the hand features Figure 1 This refers to the initial feature map that is traversed, and then it is related to the hand features. Figure 1 The corresponding target 3D pose data is the initial 3D pose data of the hand.

[0065] Method 2: If the feature map is not the initial feature map to be traversed, then the target 3D pose data corresponding to the feature map is the predicted 3D pose data obtained from the previous traversal.

[0066] like Figure 5A As shown, Figure 5A Hand features Figure 2 and hand features Figure 3 If the feature map is not initially traversed, then it is related to the hand features. Figure 2 The corresponding target 3D pose data is the hand 3D pose data 1 obtained from the previous traversal, along with hand features. Figure 3The corresponding target 3D pose data is the hand 3D pose data 2 obtained from the previous traversal.

[0067] Step 403: Determine if there are still feature maps that have not been traversed. If yes, return to step 401; otherwise, proceed to step 404.

[0068] Step 404: Determine the predicted 3D pose data corresponding to the feature map with the highest resolution as the 3D pose data of the body part, wherein the body part includes the human body or the hand, and the body part is the same as the body part contained in the region image.

[0069] Specifically, if the input feature maps are multi-level pyramid feature maps of the hand, then the corresponding body part is the hand. If the input feature maps are multi-level pyramid feature maps of the human body, then the corresponding body part is the human body.

[0070] like Figure 5A The image shown is a flowchart illustrating the process of determining the three-dimensional pose data of the hand. Figure 5A The example used here is a multi-level pyramid feature map of the hand with three feature maps. However, there is no limit to the number of feature maps in a multi-level pyramid feature map of the hand; the number of feature maps can be set according to the actual situation.

[0071] from Figure 5A As can be seen, when the hand region image is input into the feature extraction network for feature extraction, three feature maps are obtained. These represent the hand features. Figure 1 Hand features Figure 2 and hand features Figure 3 First, hand features Figure 1 The initial 3D hand pose data is input into a 3D pose prediction network for hand pose estimation, resulting in 3D hand pose data 1. Then, hand features are analyzed. Figure 2 The 3D hand pose data 1 is input into the 3D pose prediction network for hand pose estimation again, resulting in 3D hand pose data 2. Then, the hand features are... Figure 3 The three-dimensional hand pose data 2 is input into the three-dimensional pose data prediction network to estimate the hand pose, thereby obtaining the three-dimensional hand pose data 3, and the three-dimensional hand pose data 3 is determined as the three-dimensional pose data of the hand.

[0072] It should be noted that in this embodiment, the hands include the left hand and the right hand, that is, the left hand and the right hand perform the functions respectively. Figure 5A The process yields the 3D pose data of the left hand and the right hand.

[0073] like Figure 5B The image shown is a flowchart illustrating the process of determining the three-dimensional pose data of the human body. Figure 5B This example uses three feature maps from a multi-level pyramid feature map of the human body. However, there is no limit to the total number of feature maps in a multi-level pyramid feature map of the human body; the number of feature maps can be set according to the actual situation.

[0074] from Figure 5B As can be seen, when a human body region image is input into a feature extraction network for human body feature extraction, three feature maps are obtained. These represent human body features. Figure 1 Human characteristics Figure 2 and human characteristics Figure 3 First, human body characteristics Figure 1 The initial 3D human pose data is input into a 3D pose prediction network for human pose estimation, resulting in 3D human pose data 1. Then, human features are... Figure 2 Human 3D pose data 1 is input into a 3D pose prediction network for further human pose estimation, resulting in human 3D pose data 2. Then, human features are... Figure 3 Human three-dimensional pose data 2 is input into a three-dimensional pose data prediction network to estimate human pose, resulting in human three-dimensional pose data 3, which is then identified as the human three-dimensional pose data.

[0075] Step 304: By using the rotation amounts between the three-dimensional key points in each three-dimensional pose data, the three-dimensional key points of the hand are stitched together with the three-dimensional key points of the human body to obtain the pose of the three-dimensional model of the target object.

[0076] The 3D pose data includes each 3D keypoint and the rotation amounts between them. Each 3D keypoint has a rotation amount with respect to its parent 3D keypoint and a rotation amount with respect to its child 3D keypoints.

[0077] like Figure 6 The diagram shows the skeleton of the SMPL (Skinned Multi-Person Linear) human body parametric model. Keypoint 0 is the root keypoint, and keypoints 1, 2, and 3 are child keypoints of keypoint 0. Keypoint 1 is the parent keypoint of keypoint 4. Keypoint 2 is the parent keypoint of keypoint 5. Keypoint 3 is the parent keypoint of keypoint 6, and so on. This creates a hinged relationship in the estimation of the human body parametric model.

[0078] like Figure 7The image shows a schematic diagram of the skeleton of the SMPLX human parametric model, which includes the skeletal information of the hands. Specifically, 3D keypoints 22 and 23 in the SMPLX skeleton have been replaced with the left and right hands. The skeletal diagrams of the left and right hands are shown below. Figure 8 As shown, the 3D keypoints of the hand also have a hinge relationship, that is, the child 3D keypoint has a rotation amount relative to the parent 3D keypoint. The root 3D keypoints of the left and right hands are 3D keypoints 22 and 23, so the local rotation amount of the hand can be multiplied onto the body to stitch the hand and the human body together.

[0079] After stitching the hand and body together, the hand has fewer pixels, which may cause inaccurate rotation of the body and wrist during pose estimation, resulting in unnatural wrist movements after stitching. For example... Figure 9 As shown in the image, the stitched hand and the human body do not appear natural to each other.

[0080] Therefore, in order to further improve the accuracy of human pose, in one embodiment, after performing step 304, it is necessary to optimize the pose of the 3D model, such as... Figure 10 The diagram shows the specific process for optimizing the pose of a 3D model, including the following steps:

[0081] Step 1001: For any arm model of the three-dimensional model, based on the position of the three-dimensional key point of the elbow in the arm model and the position of the three-dimensional key point of the wrist in the arm model, determine the rotation angle of the bone between the wrist and the elbow relative to the preset plane, wherein the three-dimensional key point of the elbow is in the preset plane.

[0082] Step 1002: Determine whether the rotation angle is greater than the specified angle. If yes, proceed to step 1003; otherwise, end.

[0083] In this embodiment, the specified angle is 90 degrees, but the specified angle can be set according to the actual situation. This embodiment does not limit the specific value of the specified angle. Furthermore, the direction of the rotation angle in this embodiment can be either a positive or negative rotation angle.

[0084] Step 1003: Obtain the optimized angle based on the specified angle and the rotation angle;

[0085] In one embodiment, the absolute value of the difference between the specified angle and the rotation angle is determined as the optimized angle.

[0086] Step 1004: Adjust the rotation angle of the bones between the wrist and the elbow using the optimized angle to obtain the optimized three-dimensional model.

[0087] In one embodiment, the rotation angle of the bones between the wrist and the elbow is adjusted in a specified direction by the optimized angle to obtain an optimized 3D model. For example, if the direction of the rotation angle of the bones between the wrist and the elbow is positive, the bones are rotated in a negative direction by the optimized angle, so that the rotation angle of the bones between the wrist and the elbow is equal to the specified angle. If the direction of the rotation angle of the bones between the wrist and the elbow is negative, the bones are rotated in a positive direction by the optimized angle, so that the rotation angle of the bones between the wrist and the elbow is equal to the specified angle.

[0088] To further connect the technical solutions in this application, the following is combined with... Figure 11 A detailed explanation may include the following steps:

[0089] Step 1101: Use a region detection algorithm to detect the hand region in the image containing the target object, and obtain the position of the hand region in the image, wherein the hand region includes the left hand region and the right hand region;

[0090] Step 1102: Crop the target image according to the position of the hand region to obtain the hand region image and the human body region image;

[0091] Step 1103: Upsample the hand region image using a bilinear interpolation algorithm;

[0092] Step 1104: Perform feature extraction on the hand region image and the human body region image respectively to obtain a multi-level pyramid feature map of the hand region image and a multi-level pyramid feature map of the human body region image, wherein the multi-level pyramid feature map includes multiple feature maps with different resolutions.

[0093] Step 1105: For any one of the hand region image and the human body region image, traverse the multi-level pyramid feature map of the region image in ascending order of image resolution;

[0094] Step 1106: For any feature map that has been traversed, input the feature map and the target 3D pose data corresponding to the feature map into the 3D pose data prediction network to obtain the predicted 3D pose data, wherein the target 3D pose data includes the initial 3D pose data or the predicted 3D pose data obtained from the previous traversal.

[0095] Step 1107: Determine if there are still feature maps that have not been traversed. If yes, return to step 1105; otherwise, proceed to step 1108.

[0096] Step 1108: Determine the predicted 3D pose data corresponding to the feature map with the highest resolution as the 3D pose data of the body part, wherein the body part includes the human body or the hand, and the body part is the same as the body part contained in the region image.

[0097] Step 1109: By using the rotation amount between each three-dimensional key point in each three-dimensional pose data, the three-dimensional key points of the hand are stitched together with the three-dimensional key points of the human body to obtain the pose of the three-dimensional model of the target object.

[0098] Step 1110: For any arm model of the three-dimensional model, based on the position of the three-dimensional key point of the elbow in the arm model and the position of the three-dimensional key point of the wrist in the arm model, determine the rotation angle of the bone between the wrist and the elbow relative to the preset plane, wherein the three-dimensional key point of the elbow is in the preset plane.

[0099] Step 1111: Determine whether the rotation angle is greater than the specified angle. If yes, proceed to step 1112; otherwise, end.

[0100] Step 1112: Obtain the optimized angle based on the specified angle and the rotation angle;

[0101] Step 1113: Adjust the rotation angle of the bones between the wrist and the elbow using the optimized angle to obtain the optimized three-dimensional model.

[0102] Based on the same inventive concept, the method for predicting the pose of a 3D model as described above can also be implemented by a device for predicting the pose of a 3D model. The effect of predicting the pose of the 3D model is similar to that of the aforementioned method, and will not be described again here.

[0103] Figure 12 This is a schematic diagram of a device for predicting the pose of a three-dimensional model according to an embodiment of the present disclosure.

[0104] like Figure 12 As shown, the apparatus 1200 for predicting the pose of a 3D model disclosed herein may include a segmentation module 1210, a feature extraction module 1220, a 3D pose prediction module 1230, and a stitching module 1240.

[0105] The segmentation module 1210 is used to segment the hand region of an image containing a target object using a preset algorithm to obtain a hand region image and a human body region image other than the hand region.

[0106] The feature extraction module 1220 is used to extract features from the hand region image and the human body region image respectively, to obtain a multi-level pyramid feature map of the hand region image and a multi-level pyramid feature map of the human body region image, wherein the multi-level pyramid feature map includes multiple feature maps with different resolutions.

[0107] The three-dimensional pose prediction module 1230 is used to predict the three-dimensional pose of the hand or the human body based on the initial three-dimensional pose data of the region image and the multi-level pyramid feature map of the region image, respectively, to obtain the three-dimensional pose data of the hand or the three-dimensional pose data of the human body. The three-dimensional pose data includes each three-dimensional key point and the rotation amount between each three-dimensional key point.

[0108] The stitching module 1240 is used to stitch the three-dimensional key points of the hand with the three-dimensional key points of the human body by using the rotation amount between each three-dimensional key point in each three-dimensional pose data, so as to obtain the pose of the three-dimensional model of the target object.

[0109] In one embodiment, the three-dimensional pose prediction module 1230 is specifically used for:

[0110] The multi-level pyramid feature maps of the region image are traversed in order of increasing image resolution;

[0111] For any feature map that has been traversed, the feature map and the target 3D pose data corresponding to the feature map are input into the 3D pose data prediction network to obtain the predicted 3D pose data. The target 3D pose data includes the initial 3D pose data or the predicted 3D pose data obtained from the previous traversal.

[0112] Determine if there are still feature maps that have not been traversed;

[0113] If it exists, then return to the step of traversing the multi-level pyramid feature maps of the region image in ascending order of image resolution, until there are no feature maps that have not been traversed. Then, the predicted 3D pose data corresponding to the feature map with the highest resolution is determined as the 3D pose data of the body part, wherein the body part includes the human body or the hand, and the body part is the same as the body part contained in the region image.

[0114] In one embodiment, the apparatus further includes:

[0115] The target 3D pose determination module 1250 is used to determine the target 3D pose data corresponding to the feature map in the following manner:

[0116] If the feature map is the initial feature map to be traversed, then the target 3D pose data corresponding to the feature map is the initial 3D pose data; or,

[0117] If the feature map is not the initial feature map to be traversed, then the target 3D pose data corresponding to the feature map is the predicted 3D pose data obtained from the previous traversal.

[0118] In one embodiment, the segmentation module 1210 is specifically used for:

[0119] A region detection algorithm is used to detect the hand region in an image containing a target object, thereby obtaining the location of the hand region in the image, wherein the hand region includes the left hand region and the right hand region;

[0120] The target image is cropped according to the position of the hand region to obtain the hand region image and the human body region image.

[0121] In one embodiment, the apparatus further includes:

[0122] The optimization module 1260 is used to stitch together the three-dimensional key points of the hand and the three-dimensional key points of the human body to obtain the pose of the three-dimensional model of the target object. Then, for any arm model of the three-dimensional model, based on the position of the three-dimensional key points of the elbow and the position of the three-dimensional key points of the wrist in the arm model, the rotation angle of the bones between the wrist and the elbow relative to a preset plane is determined, wherein the three-dimensional key points of the elbow are in the preset plane.

[0123] If the rotation angle is greater than the specified angle, then the optimized angle is obtained based on the specified angle and the rotation angle;

[0124] The rotation angle of the bones between the wrist and the elbow is adjusted using the optimized angle to obtain the optimized three-dimensional model.

[0125] After introducing a method and apparatus for predicting the pose of a three-dimensional model according to an exemplary embodiment of the present invention, an electronic device according to another exemplary embodiment of the present invention will be introduced next.

[0126] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as "circuit", "module", or "system".

[0127] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code that, when executed by the processor, causes the processor to perform the steps in the method for predicting the pose of a 3D model according to various exemplary embodiments of the present invention described above. For example, the processor may perform actions such as... Figure 3 Steps 301-304 are shown in the diagram.

[0128] The following reference Figure 13 To describe an electronic device 1300 according to this embodiment of the present invention. Figure 13 The electronic device 1,300 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0129] like Figure 13 As shown, the electronic device 1300 is presented in the form of a general-purpose electronic device. The components of the electronic device 1300 may include, but are not limited to: at least one processor 1301, at least one computer storage medium 1302, and a bus 1303 connecting different system components (including the computer storage medium 1302 and the processor 1301).

[0130] Bus 1303 represents one or more of several bus structures, including a computer storage media bus or computer storage media controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0131] Computer storage medium 1302 may include readable media in the form of volatile computer storage media, such as random access computer storage medium (RAM) 1321 and / or cache storage medium 1322, and may further include read-only computer storage medium (ROM) 1323.

[0132] The computer storage medium 1302 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0133] Electronic device 1300 can also communicate with one or more external devices 1304 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with electronic device 1300, and / or any device that enables electronic device 1300 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1305. Furthermore, electronic device 1300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1306. As shown, network adapter 1306 communicates with other modules used in electronic device 1300 via bus 1303. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] In some possible implementations, various aspects of the method for predicting the pose of a 3D model provided by the present invention can also be implemented in the form of a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps in the method for predicting the pose of a 3D model according to various exemplary embodiments of the present invention as described above.

[0135] The program product may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access computer storage media (RAM), read-only computer storage media (ROM), erasable programmable read-only computer storage media (EPROM or flash memory), optical fibers, portable compact disk read-only computer storage media (CD-ROM), optical computer storage media, magnetic computer storage media, or any suitable combination thereof.

[0136] The program product for predicting the pose of a 3D model according to embodiments of the present invention can be a portable compact disc read-only computer storage medium (CD-ROM) and include program code, and can run on an electronic device. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0137] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0138] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0139] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0140] It should be noted that although several modules of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0141] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk computer storage media, CD-ROMs, optical computer storage media, etc.) containing computer-usable program code.

[0143] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable computer storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for predicting the pose of a 3D model, characterized in that, The method includes: A preset algorithm is used to segment the hand region of an image containing the target object, resulting in a hand region image and a human body region image excluding the hand region. Feature extraction is performed on the hand region image and the human body region image respectively to obtain a multi-level pyramid feature map of the hand region image and a multi-level pyramid feature map of the human body region image, wherein the multi-level pyramid feature map includes multiple feature maps with different resolutions. For any one of the hand region image and the human body region image, the multi-level pyramid feature maps of the region image are traversed in ascending order of image resolution. For any traversed feature map, the feature map and the target 3D pose data corresponding to the feature map are input into the 3D pose data prediction network to obtain predicted 3D pose data. The target 3D pose data includes initial 3D pose data or predicted 3D pose data obtained from the previous traversal. It is determined whether there are any feature maps that have not been traversed. If so, the step of traversing the multi-level pyramid feature maps of the region image in ascending order of image resolution is returned until there are no feature maps that have not been traversed. Then, the predicted 3D pose data corresponding to the feature map with the highest resolution is determined as the 3D pose data of the body part. The body part includes the human body or the hand, and the body part is the same as the body part contained in the region image. By using the rotation amounts between the three-dimensional key points in each three-dimensional pose data, the three-dimensional key points of the hand are stitched together with the three-dimensional key points of the human body to obtain the pose of the three-dimensional model of the target object.

2. The method according to claim 1, characterized in that, The target 3D pose data corresponding to the feature map is determined in the following manner: If the feature map is the initial feature map to be traversed, then the target 3D pose data corresponding to the feature map is the initial 3D pose data; or, If the feature map is not the initial feature map to be traversed, then the target 3D pose data corresponding to the feature map is the predicted 3D pose data obtained from the previous traversal.

3. The method according to claim 1, characterized in that, The step of segmenting the hand region of an image containing the target object using a preset algorithm to obtain a hand region image and a human body region image excluding the hand region includes: The hand region is detected using a region detection algorithm to obtain the location of the hand region in the image, wherein the hand region includes the left hand region and the right hand region; The image containing the target object is cropped according to the position of the hand region to obtain the hand region image and the human body region image.

4. The method according to claim 1, characterized in that, After stitching together the 3D key points of the hand with the 3D key points of the human body to obtain the pose of the 3D model of the target object, the method further includes: For any arm model in the three-dimensional model, based on the positions of the three-dimensional key points of the elbow and the three-dimensional key points of the wrist in the arm model, the rotation angle of the bones between the wrist and the elbow relative to a preset plane is determined, wherein the three-dimensional key points of the elbow are in the preset plane. If the rotation angle is greater than the specified angle, then the optimized angle is obtained based on the specified angle and the rotation angle; The rotation angle of the bones between the wrist and the elbow is adjusted using the optimized angle to obtain the optimized three-dimensional model.

5. The method according to claim 1, characterized in that, Before performing feature extraction on the hand region image and the human body region image respectively, the method further includes: The hand region image is upsampled using a bilinear interpolation algorithm.

6. An electronic device, characterized in that, It includes a processor and a memory, which are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: A preset algorithm is used to segment the hand region of an image containing the target object, resulting in a hand region image and a human body region image excluding the hand region. Feature extraction is performed on the hand region image and the human body region image respectively to obtain a multi-level pyramid feature map of the hand region image and a multi-level pyramid feature map of the human body region image, wherein the multi-level pyramid feature map includes multiple feature maps with different resolutions. For any one of the hand region image and the human body region image, the multi-level pyramid feature maps of the region image are traversed in ascending order of image resolution. For any traversed feature map, the feature map and the target 3D pose data corresponding to the feature map are input into the 3D pose data prediction network to obtain predicted 3D pose data. The target 3D pose data includes initial 3D pose data or predicted 3D pose data obtained from the previous traversal. It is determined whether there are any feature maps that have not been traversed. If so, the step of traversing the multi-level pyramid feature maps of the region image in ascending order of image resolution is returned until there are no feature maps that have not been traversed. Then, the predicted 3D pose data corresponding to the feature map with the highest resolution is determined as the 3D pose data of the body part. The body part includes the human body or the hand, and the body part is the same as the body part contained in the region image. By using the rotation amounts between the three-dimensional key points in each three-dimensional pose data, the three-dimensional key points of the hand are stitched together with the three-dimensional key points of the human body to obtain the pose of the three-dimensional model of the target object.

7. The electronic device according to claim 6, characterized in that, The processor executes the segmentation of the hand region in the image containing the target object using a preset algorithm to obtain a hand region image and a human body region image excluding the hand region. Specifically, it is configured as follows: A region detection algorithm is used to detect the hand region in an image containing a target object, thereby obtaining the location of the hand region in the image, wherein the hand region includes the left hand region and the right hand region; The image containing the target object is cropped according to the position of the hand region to obtain the hand region image and the human body region image.

8. The electronic device according to claim 6, characterized in that, The processor is also configured to: After stitching the three-dimensional key points of the hand with the three-dimensional key points of the human body to obtain the pose of the three-dimensional model of the target object, for any arm model of the three-dimensional model, based on the position of the three-dimensional key points of the elbow and the position of the three-dimensional key points of the wrist in the arm model, the rotation angle of the bones between the wrist and the elbow relative to the preset plane is determined, wherein the three-dimensional key points of the elbow are in the preset plane. If the rotation angle is greater than the specified angle, then the optimized angle is obtained based on the specified angle and the rotation angle; The rotation angle of the bones between the wrist and the elbow is adjusted using the optimized angle to obtain the optimized three-dimensional model.

9. The electronic device according to claim 6, characterized in that, The processor is also configured to: Before performing feature extraction on the hand region image and the human body region image respectively, the hand region image is upsampled using a bilinear interpolation algorithm.

Images