Method, device, computer equipment and storage medium for object three-dimensional reconstruction

Abstract

The present application relates to a method, apparatus, computer device, storage medium, and computer program product for three-dimensional reconstruction of an object. The method comprises: extracting image features from a target object image to obtain multiple target feature maps; detecting key points based on the multiple target feature maps to obtain a two-dimensional position heat map corresponding to each key point of the target object; predicting parent vectors based on the two-dimensional position heat map of each key point and each target feature map to obtain a parent vector heat map corresponding to each key point under each three-dimensional coordinate axis; predicting three-dimensional positions based on the parent vector heat map corresponding to each key point and each target feature map to obtain three-dimensional position information corresponding to each key point; and performing three-dimensional reconstruction based on the three-dimensional position information of each key point to obtain a three-dimensional object model of the target object. The present method can improve the accuracy of three-dimensional reconstruction.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, computer equipment, storage medium, and computer program product for three-dimensional reconstruction of an object. Background Art

[0002] With the development of computer vision technology, the application of various human-computer interaction devices is becoming increasingly widespread, which has promoted the development of 3D object reconstruction technology. The objects of 3D reconstruction can be human bodies, hands, or faces. For example, 3D reconstruction of human hands in images or videos can be performed to obtain a reconstructed 3D human hand. Then, based on the 3D human hand, human gestures can be accurately recognized, realizing human-computer interaction.

[0003] In conventional technology, a three-dimensional reconstruction model for three-dimensional reconstruction is usually obtained through training, and the three-dimensional reconstruction is performed using the three-dimensional reconstruction model.

[0004] However, since the samples required for training the 3D reconstruction model are usually scarce, the number of samples used to train the 3D reconstruction model is small, and the 3D reconstruction model cannot be sufficiently trained. As a result, when the 3D reconstruction model is used for 3D reconstruction, the accuracy of the 3D reconstruction is low. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium and computer program product for three-dimensional reconstruction of an object that can improve the accuracy of three-dimensional reconstruction in order to address the above technical problems.

[0006] In a first aspect, the present application provides a method for three-dimensional reconstruction of an object. The method includes: extracting image features of a target object image to obtain multiple target feature maps; the target object image is an image including a target object; detecting key points based on the multiple target feature maps to obtain two-dimensional position heat maps corresponding to each key point of the target object; predicting parent vectors based on the two-dimensional position heat maps of each key point and each target feature map to obtain parent vector heat maps under each three-dimensional coordinate axis corresponding to each key point; the pixel position in the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, the pixel value at the pixel position in the parent vector heat map is the confidence that the vector component represented by the pixel position belongs to the parent key point vector, the parent key point vector refers to a vector pointing from the position of the parent key point corresponding to the key point to the position of the key point; predicting three-dimensional positions based on the parent vector heat map corresponding to each key point and each target feature map to obtain three-dimensional position information corresponding to each key point; performing three-dimensional reconstruction based on the three-dimensional position information of each key point to obtain a three-dimensional object model of the target object.

[0007] In the second aspect, the present application also provides a three-dimensional reconstruction device for an object. The device includes: a feature extraction module for extracting image features of a target object image to obtain a plurality of target feature maps; the target object image is an image including a target object; a key point detection module for detecting key points based on the plurality of target feature maps to obtain a two-dimensional position heat map corresponding to each key point of the target object; a vector prediction module for predicting a parent vector based on the two-dimensional position heat map of each key point and each target feature map to obtain a parent vector heat map under each three-dimensional coordinate axis corresponding to each key point; the pixel position in the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, and the pixel value at the pixel position in the parent vector heat map is the confidence that the vector component represented by the pixel position belongs to the parent key point vector, and the parent key point vector refers to a vector pointing from the position of the parent key point corresponding to the key point to the position of the key point; a three-dimensional position prediction module for predicting a three-dimensional position based on the parent vector heat map corresponding to each key point and each target feature map to obtain three-dimensional position information corresponding to each key point;

[0008] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction based on the three-dimensional position information of each key point to obtain a three-dimensional object model of the target object.

[0009] In some embodiments, the vector prediction module is also used to: perform fusion processing based on the two-dimensional position heat map of each key point and each target feature map to obtain a fused feature map corresponding to each key point; use the fused feature map corresponding to each key point to perform parent vector prediction to obtain a parent vector heat map under each three-dimensional coordinate axis corresponding to each key point.

[0010] In some embodiments, each key point includes a root key point; the three-dimensional position prediction module is also used to: perform root vector prediction based on the parent vector heat map corresponding to each key point and each target feature map, to obtain the root vector heat map under each three-dimensional coordinate axis corresponding to each key point; the pixel position in the root vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, and the pixel value at the pixel position in the root vector heat map is the confidence that the vector component represented by the pixel position in the root vector heat map belongs to the root key point vector, and the root key point vector refers to the vector pointing from the position of the root key point to the position of the key point; based on the root vector heat map under each three-dimensional coordinate axis corresponding to each key point, the three-dimensional position information corresponding to each key point is determined.

[0011] In some embodiments, in terms of determining the three-dimensional position information corresponding to each of the key points based on the root vector heat map under each three-dimensional coordinate axis corresponding to each of the key points, the three-dimensional position prediction module is specifically used to: determine the position of the root key point to obtain the root key point position; for each of the key points, determine the root key point vector of the key point according to the root vector heat map under each three-dimensional coordinate axis corresponding to the key point; offset the root key point vector based on the root key point position to obtain the three-dimensional position information corresponding to the key point.

[0012] In some embodiments, the key points include a root key point; the three-dimensional reconstruction module is also used to: for each of the key points, determine the relative position information between the key point and the root key point based on the three-dimensional position information of the key point and the three-dimensional position information of the root key point; determine the direction vector of the parent key point corresponding to the key point pointing to the key point based on the three-dimensional position information of the key point and the three-dimensional position information of the corresponding parent key point; use the relative position information and direction vector corresponding to each of the key points to determine the posture information of the target object; perform three-dimensional reconstruction based on the posture information to obtain a three-dimensional object model of the target object.

[0013] In some embodiments, in performing three-dimensional reconstruction based on the posture information to obtain a three-dimensional object model of the target object, the three-dimensional reconstruction module is specifically used to: determine the bone length between adjacent key points based on the three-dimensional position information of each key point; determine the shape information of the target object using the bone length between the adjacent key points; and perform three-dimensional reconstruction based on the posture information and the shape information to obtain a three-dimensional object model of the target object.

[0014] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned object three-dimensional reconstruction method when executing the computer program.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned method for three-dimensional reconstruction of an object.

[0016] In a fifth aspect, the present application further provides a computer program product, which includes a computer program that implements the steps in the above-mentioned object three-dimensional reconstruction method when executed by a processor.

[0017] The above-mentioned object three-dimensional reconstruction method, device, computer equipment, storage medium and computer program product predict parent vectors based on the two-dimensional position heat map of key points and the target feature map, and obtain the parent vector heat map under each three-dimensional coordinate axis corresponding to each key point, thereby obtaining three-dimensional features based on two-dimensional features, and since the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point contains the parent key point vector, the parent key point vector refers to the vector pointing from the position of the parent key point corresponding to the key point to the position of the key point, that is, the parent key point vector provides the relative position relationship between the key points, so that the three-dimensional position information corresponding to each key point obtained based on the parent vector heat map and each target feature map corresponding to each key point is more accurate, so that the three-dimensional object model of the target object obtained by three-dimensional reconstruction based on the three-dimensional position information of each key point is more accurate, thereby improving the accuracy of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A diagram of an application environment of a method for three-dimensional reconstruction of an object in one embodiment;

[0019] Figure 2 1 is a flow chart of a method for three-dimensional reconstruction of an object in one embodiment;

[0020] Figure 3 1 is a flow chart of various task stages of a method for 3D reconstruction of an object in one embodiment;

[0021] Figure 4 Schematic diagram of a parent vector heat map in one embodiment;

[0022] Figure 5 A schematic diagram of determining posture information in one embodiment;

[0023] Figure 6 is a schematic flow chart of a method for three-dimensional reconstruction of an object in another embodiment;

[0024] Figure 7 is a structural block diagram of an apparatus for three-dimensional reconstruction of an object in one embodiment;

[0025] Figure 8 is a diagram of the internal structure of a computer device in one embodiment;

[0026] Figure 9 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0028] The object 3D reconstruction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104 or placed on the cloud or other network servers.

[0029] Specifically, server 104 can store the target object image or obtain the target object image sent by terminal 102. Server 104 extracts image features from the target object image to obtain multiple target feature maps. Based on the multiple target feature maps, server 104 detects key points to obtain a two-dimensional position heat map corresponding to each key point of the target object. Based on the two-dimensional position heat map of each key point and each target feature map, server 104 predicts the three-dimensional position of each key point based on the parent vector heat map and each target feature map, obtaining three-dimensional position information corresponding to each key point. Server 104 performs three-dimensional reconstruction based on the three-dimensional position information of each key point to obtain a three-dimensional object model of the target object. The target object image is an image including the target object. The pixel position in the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis. The pixel value at the pixel position in the parent vector heat map is the confidence level that the vector component represented by the pixel position belongs to the parent key point vector. The parent key point vector refers to the vector pointing from the position of the parent key point corresponding to the key point to the position of the key point. The server 104 can store the 3D object model and can also send the 3D object model to the terminal 102. The terminal 102 can receive the 3D object model and use the 3D object model to implement human-computer interaction. For example, if the target object is a human hand and the 3D object model is a 3D human hand model, the terminal 102 can use the 3D human hand model to perform gesture recognition and implement human-computer interaction.

[0030] The terminal 102 may be, but is not limited to, a desktop terminal or a mobile terminal, and the mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a laptop computer, and a smartwatch, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0031] In some embodiments, as Figure 2 As shown, a method for three-dimensional reconstruction of an object is provided. The method can be executed by a terminal or a server, or by a terminal and a server together. Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0032] Step 202 : extract image features from the target object image to obtain a plurality of target feature maps; the target object image is an image including the target object.

[0033] The target object image is an image including a target object. The target object is the object to be 3D reconstructed. The target object can be any of a human hand, a human foot, a human face, or a human body. For example, the target object can be a human hand, and the target object image is an image of a hand. The target feature map is obtained by extracting image features from the target object image. There can be multiple target feature maps, and multiple means at least two.

[0034] Specifically, the server obtains the target object image, and then performs image feature extraction on the target object image to obtain multiple target feature maps. The server can use a feature extraction network to extract features from the target object image, for example, Figure 3 As shown, the target object image is a human hand image. In the feature extraction stage, the server can input the human hand image into the feature extraction network and output the target feature map of the human hand image, that is, the feature map F. The feature extraction network can be a CNN (Convolutional Neural Network)

[0035] In some embodiments, in order to improve the efficiency of image feature extraction, the server can use a convolution layer with a step size of 3 and a convolution kernel size of 7×7 to perform convolution processing on the target object image to obtain a convolution result, and then use the CRelu (Concatenated Rectified Linear Unit) activation function to perform feature activation on the convolution result to obtain a convolution activation feature. The server can then input the convolution activation feature into the feature extraction network to extract image features and output multiple target feature maps. By using a convolution layer with a step size of 3 and a convolution kernel size of 7×7 for convolution processing, the target object image can be quickly downsampled and the speed of feature extraction can be improved; and using the CRelu activation function can eliminate feature redundancy in convolution and improve the generalization ability of the network. Among them, the convolution result is obtained by convolution processing on the target object image, and the convolution activation feature is obtained by feature activation on the convolution result. The feature extraction network can be a depth-separable convolution network, for example, it can be mobileV2.

[0036] In step 204 , key points are detected based on the multiple target feature maps to obtain a two-dimensional position heat map corresponding to each key point of the target object.

[0037] Among them, the target object has multiple key points, and multiple means at least two. The key points correspond to key point numbers, and the key point numbers are used to identify the positions of the key points on the target object. For example, the target object is a human hand, and the human hand has 21 key points, among which the key point with key point number 0, i.e., key point 0, is located at the wrist. Multiple key points can constitute a key point tree. The key points in the key point tree have a hierarchical relationship. There is a root key point in the key point tree, which is located at the top level of the key point tree. The farther the key point is from the root key point, the lower its level. The root key point has no parent key point. The key points other than the root key point each correspond to a parent key point. The parent key point has a higher level than the child key point, and the parent key point is adjacent to the corresponding child key point. In the key point tree, the parent key point can drive the movement of the child key point.

[0038] One key point corresponds to one two-dimensional position heat map. For example, if the target object is a human hand and the human hand has 21 key points, then the 21 key points correspond to 21 two-dimensional position heat maps.

[0039] A heatmap is a confidence map, also known as a heatmap. The pixel value at each pixel position in the heatmap is the confidence level, and the pixel value range is [0, 1]. A two-dimensional position heatmap is used to identify the two-dimensional coordinates of a key point. The pixel value at each pixel position in the two-dimensional position heatmap represents the confidence level that the pixel position is the two-dimensional position of the key point. That is, the pixel position with the largest pixel value is the two-dimensional position of the key point. For example, in the two-dimensional position heatmap corresponding to key point i, if the pixel value at the pixel position (x, y) is the largest, then the pixel position (x, y) is the two-dimensional position of key point i.

[0040] Specifically, the server detects key points based on multiple target feature maps and obtains two-dimensional position heat maps corresponding to each key point of the target object. The server can use the trained key point detection network to detect key points on the target feature map and obtain two-dimensional position heat maps corresponding to each key point of the target object. For example, Figure 3 As shown in the figure, the target object image is a human hand image. In the 2D (two dimensional) detection stage, the target feature map, i.e., feature map F, is input into the trained key point detection network, and the output is a two-dimensional position heat map corresponding to each key point of the human hand, i.e., heat map H. The key point detection network is used to detect the positions of the key points of the target object in the target object image.

[0041] In some embodiments, the server can use the sample object image to train the key point detection network to be trained to obtain a trained key point detection network. The server can obtain a sample feature map of the sample object image, and then input the sample feature map into the key point detection network to be trained for key point detection, and output the predicted two-dimensional position heat map corresponding to each key point of the target object, and then obtain a first loss value based on the predicted two-dimensional position heat map and the two-dimensional position heat map label corresponding to each key point, and adjust the key point detection network to be trained based on the first loss value until the network converges to obtain a trained key point detection network. Wherein, the sample object image is an image including the target object, the sample feature map is obtained by performing image feature extraction on the sample object image, the sample object image corresponds to a two-dimensional position heat map label, the two-dimensional position heat map label is used to characterize the true two-dimensional position of each key point of the target object in the sample object image, the first loss value characterizes the difference between the predicted two-dimensional position heat map and the two-dimensional position heat map label, and the loss function corresponding to the first loss value can be an L2 norm loss function, and the L2 norm is the minimum square error, as shown below:

[0042] L H =||H GT -H|| 2 ;

[0043] Among them, L H is the first loss value, H GT is the 2D position heat map label of the sample object image, and H is the predicted 2D position heat map corresponding to the key point.

[0044] Step 206, based on the two-dimensional position heat map of each key point and each target feature map, the parent vector is predicted to obtain the parent vector heat map under each three-dimensional coordinate axis corresponding to each key point; the pixel position in the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, and the pixel value at the pixel position in the parent vector heat map is the confidence that the vector component represented by the pixel position belongs to the parent key point vector. The parent key point vector refers to the vector from the position of the parent key point corresponding to the key point to the position of the key point.

[0045] Parent vector prediction involves predicting the parent keypoint vector corresponding to a keypoint. A parent keypoint vector is a vector that points from the position of the keypoint's parent keypoint to the keypoint's position. The parent vector heatmap corresponding to each keypoint in the three-dimensional coordinate axes represents the parent keypoint vector. The three-dimensional coordinate axes are the x-axis, y-axis, and z-axis. Therefore, each keypoint corresponds to a parent vector heatmap in the x-axis, y-axis, and z-axis, respectively. Thus, each keypoint corresponds to three parent vector heatmaps, and the two-dimensional position heatmap corresponding to each keypoint corresponds to three parent vector heatmaps. The pixel position in the parent vector heatmap in the three-dimensional coordinate axis corresponding to the keypoint represents the vector component in that three-dimensional coordinate axis. The pixel value at that pixel position in the parent vector heatmap represents the confidence that the vector component represented by that pixel position belongs to the parent keypoint vector. That is, the pixel position with the largest pixel value represents the vector represented by the parent keypoint vector. The parent vector heatmap is also called a delta map. In the parent vector heatmap in the x-axis, the horizontal axis of the pixel position represents the coordinate of the parent keypoint corresponding to the keypoint in the x-axis, and the vertical axis of the pixel position represents the coordinate of the keypoint in the x-axis. In the parent vector heatmap on the y-axis, the horizontal coordinate of the pixel position represents the coordinate of the parent keypoint corresponding to the keypoint on the y-axis, and the vertical coordinate of the pixel position represents the coordinate of the keypoint on the y-axis. In the parent vector heatmap on the z-axis, the horizontal coordinate of the pixel position represents the coordinate of the parent keypoint corresponding to the keypoint on the z-axis, and the vertical coordinate of the pixel position represents the coordinate of the keypoint on the z-axis.

[0046] For example, Figure 4 As shown, the target object is a human hand. Figure 4 (a) in the figure is a two-dimensional position heat map corresponding to a key point. Figure 4 (b) in the figure is the parent vector heat map under the x-axis corresponding to the key point. Figure 4 (c) in the figure is the parent vector heat map under the y-axis corresponding to the key point. Figure 4 (d) is the parent vector heat map under the z-axis corresponding to the key point. Assume that in the parent vector heat map under the x-axis corresponding to the key point k, the pixel position is (x k-1 , x k ) has the largest pixel value; in the parent vector heat map under the y-axis, the pixel position is (y k-1 ,y k ) has the largest pixel value; in the parent vector heat map under the z coordinate axis, the pixel position is (z k-1 , z k ) is the largest, then the vector components of the parent key point k-1 pointing to key point k corresponding to key point k in the x-axis, y-axis, and z-axis are x k -x k-1 、y k -y k-1 and zk -z k-1 , then (x k-1 ,y k-1 , z k-1 ) can be the three-dimensional position of the parent key point corresponding to the key point k, (x k ,y k , z k ) can be the three-dimensional position of key point k, that is, the parent key point vector corresponding to key point k is position (x k-1 ,y k-1 , z k-1 ) points to the position (x k ,y k , z k ) vector, which can be expressed as (x k -x k-1 ,y k -y k-1 , z k -z k-1 ).

[0047] Specifically, the server can predict parent vectors based on the 2D position heatmaps of each keypoint and the target feature maps, obtaining parent vector heatmaps corresponding to each keypoint in each 3D coordinate axis. The server can use a trained parent vector prediction network to predict parent vectors based on the 2D position heatmaps of each keypoint and the target feature maps. The parent vector prediction network is used to predict the parent keypoint vector corresponding to each keypoint.

[0048] In some embodiments, the server can perform a fusion process based on the two-dimensional position heat map of each key point and each target feature map to obtain a first fused feature map corresponding to each key point, and then perform a parent vector prediction based on each first fused feature map to obtain a parent vector heat map under each three-dimensional coordinate axis corresponding to each key point. The server can input the first fused feature map into a trained parent vector prediction network to perform parent vector prediction to obtain a parent vector heat map under each three-dimensional coordinate axis corresponding to each key point. For example, Figure 3 As shown in the figure, in the 2D detection stage, the server can fuse the feature map F with the heat map H to obtain a first fused feature map, and then input the first fused feature map into the trained parent vector prediction network to output the parent vector heat map corresponding to each key point under each three-dimensional coordinate axis, that is, the delta map D.

[0049] In some embodiments, the server can use the sample object image to train the parent vector prediction network to be trained to obtain a trained parent vector prediction network. The server obtains the sample fusion feature map corresponding to the sample object image, inputs the sample fusion feature map into the parent vector prediction network to be trained for parent vector prediction, and outputs the predicted parent vector heat map under each three-dimensional coordinate axis corresponding to each key point, and then obtains a second loss value based on the predicted parent vector heat map and the parent vector heat map label. The second loss value is used to adjust the parameters of the parent vector prediction network to be trained until the network converges to obtain a trained parent vector prediction network. Among them, the sample fusion feature map is obtained by fusing the target feature map and the two-dimensional position heat map corresponding to the sample object image. The sample object image corresponds to a parent vector heat map label. The parent vector heat map label is used to characterize the parent key point vector corresponding to each key point of the target object in the sample object image. The second loss value characterizes the difference between the predicted parent vector heat map and the parent vector heat map label. The loss function corresponding to the second loss value can be an L2 loss function, as shown below:

[0050] L delta =||H GT ×(D GT -D)|| 2 ;

[0051] Among them, L delta is the second loss value, D GT is the parent vector heat map label corresponding to the sample object image, D is the predicted parent vector heat map under each three-dimensional coordinate axis corresponding to each key point, H is the predicted two-dimensional position heat map corresponding to the key point, H GT Weighted for training delta as a mask image.

[0052] Step 208 : Predict the three-dimensional position based on the parent vector heat map and each target feature map corresponding to each key point, and obtain the three-dimensional position information corresponding to each key point.

[0053] The three-dimensional position information refers to the three-dimensional coordinates of the key point under the three-dimensional coordinate axis.

[0054] Specifically, the server can obtain a second fused feature map corresponding to each key point based on the parent vector heat map and each target feature map corresponding to each key point, and then predict the three-dimensional position based on the second fused feature map corresponding to each key point to obtain the three-dimensional position information corresponding to each key point. The second fused feature map can be obtained based on the parent vector heat map and each target feature map corresponding to each key point.

[0055] In some embodiments, after the server performs a fusion process based on the two-dimensional position heat map corresponding to each key point and each target feature map to obtain the first fused feature map corresponding to each key point, it can also perform a fusion process based on the first fused feature map corresponding to each key point and the parent vector heat map to obtain the second fused feature map corresponding to each key point, and then predict the three-dimensional position of the second fused feature map corresponding to each key point. The second fused feature map can also be obtained by fusion processing based on the first fused feature map corresponding to each key point and the parent vector heat map. For example, Figure 3 As shown, in the 3D (three dimensional) detection stage, the first fused feature map and the delta map corresponding to each key point can be fused to obtain a second fused feature map corresponding to each key point.

[0056] In some embodiments, the server can perform root vector prediction based on the second fused feature map corresponding to each key point, obtain the root vector heat map corresponding to each key point under each three-dimensional coordinate axis, and then determine the three-dimensional position information corresponding to each key point based on the root vector heat map corresponding to each key point under each three-dimensional coordinate axis. Among them, the root vector prediction is to predict the root key point vector corresponding to the key point. The root key point vector refers to the vector pointing from the position of the root key point to the position of the key point. The root vector heat map under the three-dimensional coordinate axis corresponding to the key point is used to characterize the root key point vector. The three-dimensional coordinate axis is the x-axis, the y-axis and the z-axis. A key point corresponds to the root vector heat map under the x-axis, the y-axis and the z-axis, that is, one key point corresponds to three root vector heat maps. The pixel position in the root vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis. The pixel value at the pixel position in the root vector heat map is the confidence that the vector component represented by the pixel position in the root vector heat map belongs to the root key point vector, that is, the vector represented by the pixel position with the largest pixel value belongs to the root key point vector.

[0057] For example, a key point corresponds to the root vector heat map under the x-axis, the root vector heat map under the y-axis, and the root vector heat map under the z-axis. Assume that the pixel position in the root vector heat map under the x-axis corresponding to the key point k is (x root , x k ) has the largest pixel value, then x k -x root That is, the vector component of the vector from the root key point to the key point in the direction of the x-axis, that is, the vector component of the root key point vector corresponding to the key point k in the direction of the x-axis.

[0058] Step 210 : Perform three-dimensional reconstruction based on the three-dimensional position information of each key point to obtain a three-dimensional object model of the target object.

[0059] The three-dimensional reconstruction is a process of reconstructing a target object in a target object image into a three-dimensional object model. The three-dimensional object model is, for example, a three-dimensional human hand model, which is a three-dimensional mesh model.

[0060] Specifically, the server can determine the pose and shape information of the target object based on the three-dimensional position information of the key points, and then obtain a three-dimensional mesh model having the geometric shape of the target object based on the pose and shape information of the target object, i.e., a three-dimensional object model of the target object. The pose information is a vector sequence representing the pose of the target object, also known as pose parameters; the shape information is a vector sequence representing the shape of the target object, also known as shape parameters. For example, if the target object is a human hand, which has 21 key points, the pose information is a 21×3-dimensional vector sequence, i.e., one key point corresponds to a value of three dimensions, representing the axis-angle expression of the rotation angle of the key point relative to its corresponding parent key point, which can represent the degree of bending of each joint of the human hand. The shape information is a 10-dimensional vector sequence, with each dimension value corresponding to a shape indicator of the human hand, such as the fatness or thinness of the human hand.

[0061] In some embodiments, generally, the server constructs a three-dimensional skeleton using the three-dimensional position information corresponding to each key point, and then performs three-dimensional reconstruction based on the three-dimensional skeleton to obtain a three-dimensional object model of the target object. However, the accuracy of three-dimensional reconstruction based on the three-dimensional skeleton is low, and a high-precision three-dimensional object model of the target object cannot be obtained. In the object three-dimensional reconstruction method of the present application, the server can determine the posture information and shape information of the target object based on the three-dimensional position information of the key points, and then use the posture information and shape information of the target object to obtain a three-dimensional mesh model with the geometric shape of the target object, that is, a three-dimensional object model of the target object. Figure 3 As shown, the target object is a human hand. In the three-dimensional reconstruction stage, the posture information and shape information of the human hand can be used to obtain a three-dimensional mesh model with the geometric shape of the human hand, that is, a three-dimensional human hand model.

[0062] In the above-mentioned object three-dimensional reconstruction method, parent vector prediction is performed based on the two-dimensional position heat map of the key points and the target feature map, and the parent vector heat map under each three-dimensional coordinate axis corresponding to each key point is obtained, thereby obtaining three-dimensional features based on two-dimensional features, and since the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point contains the parent key point vector, the parent key point vector refers to the vector pointing from the position of the parent key point corresponding to the key point to the position of the key point, that is, the parent key point vector provides the relative position relationship between the key points, so that the three-dimensional position information corresponding to each key point obtained based on the parent vector heat map and each target feature map corresponding to each key point is more accurate, so that the three-dimensional object model of the target object obtained by three-dimensional reconstruction based on the three-dimensional position information of each key point is more accurate, thereby improving the accuracy of three-dimensional reconstruction.

[0063] In some embodiments, as Figure 3 As shown in the figure, the object 3D reconstruction method can be divided into four stages, namely feature extraction, 2D detection, 3D detection and 3D reconstruction, and the network models corresponding to each stage can be trained independently. For example, the feature extraction network in the feature extraction stage, the key point detection network in the 2D detection stage, the parent vector detection network in the 3D detection stage, and the root vector detection network. Independent training of each network model is conducive to modular research and development and deployment.

[0064] In some embodiments, step 204 includes: performing fusion processing based on the two-dimensional position heat map of each key point and each target feature map to obtain a fused feature map corresponding to each key point; using the fused feature map corresponding to each key point to predict the parent vector, and obtain a parent vector heat map under each three-dimensional coordinate axis corresponding to each key point.

[0065] The fused feature map refers to a first fused feature map, which is obtained by fusing the two-dimensional position heat map of each key point and each target feature map.

[0066] Specifically, when the number of target feature maps is different from the number of key points, the server can perform convolution processing on each target feature map to obtain multiple convolution feature maps. The number of convolution feature maps is the same as the number of key points of the target object. The server can establish a one-to-one correspondence between the convolution feature map and the two-dimensional position heat map. For each key point, the two-dimensional position heat map corresponding to the key point can be fused with the corresponding convolution feature map to obtain the first fused feature map corresponding to the key point. The server can then input the first fused feature map of each key point into the trained parent vector prediction network for parent vector prediction to obtain the parent vector heat map corresponding to each key point under each three-dimensional coordinate axis. Among them, the convolution feature map is obtained by performing convolution processing on each target feature map.

[0067] In this embodiment, since the fused feature map is obtained by fusing the two-dimensional position heat map of each key point and each target feature map, and the target feature map has abstract semantic features, and the two-dimensional position heat map has the features of the two-dimensional position of the key point, that is, the low-level features of the key point representation, the fused feature map has both abstract semantic features and low-level features of the key point representation, so the parent vector heat map obtained by using the fused feature map to predict the parent vector is more accurate, thereby improving the accuracy of the three-dimensional reconstruction of the object.

[0068] In some embodiments, each key point includes a root key point; step 206 includes: performing root vector prediction based on the parent vector heat map corresponding to each key point and each target feature map, to obtain the root vector heat map under each three-dimensional coordinate axis corresponding to each key point; the pixel position in the root vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, and the pixel value at the pixel position in the root vector heat map is the confidence that the vector component represented by the pixel position in the root vector heat map belongs to the root key point vector, and the root key point vector refers to the vector pointing from the position of the root key point to the position of the key point; based on the root vector heat map under each three-dimensional coordinate axis corresponding to each key point, the three-dimensional position information corresponding to each key point is determined.

[0069] Among them, the root vector prediction is to predict the root key point vector corresponding to the key point. The root key point vector refers to the vector pointing from the position of the root key point to the position of the key point. The root vector heat map under the three-dimensional coordinate axis corresponding to the key point is used to represent the root key point vector. The three-dimensional coordinate axis is the x-axis, y-axis and z-axis. A key point corresponds to the root vector heat map under the x-axis, y-axis and z-axis respectively, that is, a key point corresponds to three root vector heat maps. The pixel position in the root vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis. The pixel value at the pixel position in the root vector heat map is the confidence that the vector component represented by the pixel position in the root vector heat map belongs to the root key point vector, that is, the vector represented by the pixel position with the largest pixel value belongs to the root key point vector. The root vector heat map can also be called a position map.

[0070] Specifically, the server fuses the parent vector heatmaps corresponding to each keypoint with the target feature maps to obtain a second fused feature map corresponding to each keypoint. The server then predicts the root vector based on the second fused feature map to obtain a root vector heatmap corresponding to each keypoint in each 3D coordinate axis. The server then determines the 3D position information corresponding to each keypoint based on the root vector heatmaps in each 3D coordinate axis.

[0071] In some embodiments, the server may further perform a fusion process based on the first fused feature map and the parent vector heat map corresponding to each key point to obtain a second fused feature map corresponding to each key point, and then use the trained root vector prediction network to perform root vector prediction based on the second fused feature map to obtain a root vector heat map corresponding to each key point under each three-dimensional coordinate axis. Figure 3 As shown in FIG, in the 3D detection stage, the second fused feature map corresponding to each key point can be input into the trained root vector prediction network, and the root vector heat map corresponding to each key point under each three-dimensional coordinate axis, that is, the position map L, can be output.

[0072] In some embodiments, the server can use the sample object image to train the root vector prediction network to obtain a trained root vector prediction network. The specific training process is the same as the training process of the parent vector prediction network to be trained, and will not be repeated here.

[0073] In this embodiment, a root vector prediction is performed based on the parent vector heat map corresponding to each key point and each target feature map to obtain a root vector heat map containing the root key vector. Since the parent vector heat map contains the parent key point vector, the parent vector heat map can provide the relative position relationship between each key point for the root vector prediction, making the obtained root vector heat map more accurate, and thus the three-dimensional position information obtained based on the root vector heat map corresponding to each key point is more accurate.

[0074] In some embodiments, based on the root vector heat map under each three-dimensional coordinate axis corresponding to each key point, determining the three-dimensional position information corresponding to each key point includes: determining the position of the root key point to obtain the root key point position; for each key point, determining the root key point vector of the key point according to the root vector heat map under each three-dimensional coordinate axis corresponding to the key point; offsetting the root key point vector based on the root key point position to obtain the three-dimensional position information corresponding to the key point.

[0075] Among them, the root key point position refers to the three-dimensional coordinates of the root key point under the three-dimensional coordinate axis. For example, the root key point position can be (0, 0, 0). In the root vector heat map under the x-axis, the horizontal coordinate of the pixel position represents the coordinate of the root key point under the x-axis, and the vertical coordinate of the pixel position represents the coordinate of the key point under the x-axis. In the root vector heat map under the y-axis, the horizontal coordinate of the pixel position represents the coordinate of the root key point under the y-axis, and the vertical coordinate of the pixel position represents the coordinate of the key point under the y-axis. In the root vector heat map under the z-axis, the horizontal coordinate of the pixel position represents the coordinate of the root key point under the z-axis, and the vertical coordinate of the pixel position represents the coordinate of the key point under the z-axis.

[0076] Specifically, the server can determine the root key point vector of the key point based on the pixel position with the largest pixel value in the root vector heat map under the x-axis, y-axis, and z-axis corresponding to the key point. For example, assuming that in the root vector heat map under the x-axis corresponding to the key point k, the pixel position is (x root , x k ) has the largest pixel value; in the root vector heat map under the y-axis, the pixel position is (y root ,y k ) has the largest pixel value; in the root vector heat map under the z coordinate axis, the pixel position is (z root , z k ) is the largest pixel value, then the vector components of the root key point vector of key point k in the x-axis, y-axis and z-axis are the components x in the x-axis direction respectively. k -x root , the component y in the y-axis direction k -y root and the component z in the z-axis direction k -z root , then (x root ,y root , z root ) can be the three-dimensional position of the root key point, (x k ,y k , z k ) can be the three-dimensional position of the key point k, that is, the root key point vector corresponding to the key point k is the position (x root ,y root , z root ) points to the position (x k ,y k , z k ) vector, which can be expressed as (x k -x root ,y k -y root , z k -z root ).

[0077] In some embodiments, the server can determine the position of the root key point, obtain the root key point position, and then offset the root key point vector based on the root key point position to obtain the three-dimensional position information corresponding to the key point. For example, when performing three-dimensional reconstruction, the three-dimensional position information of the root key point can be initialized to (0, 0, 0). Assuming that the root key point vector corresponding to the key point k is (x k -x root ,y k -y root , z k -z root), offset the root key point vector (x k -x root ,y k -y root , z k -z root ), then the three-dimensional position information of the key point k is (x k -x root ,y k -y root , z k -z root ).

[0078] In this embodiment, for each key point, the root key point vector of the key point is determined according to the root vector heat map under each three-dimensional coordinate axis corresponding to the key point, and the root key point vector is offset based on the root key point position, so as to quickly obtain the three-dimensional position information corresponding to the key point, so that three-dimensional reconstruction can be performed based on the three-dimensional position information corresponding to each key point, thereby improving the efficiency of three-dimensional reconstruction.

[0079] In some embodiments, each key point includes a root key point; step 210 also includes: for each key point, determining the relative position information between the key point and the root key point based on the three-dimensional position information of the key point and the three-dimensional position information of the root key point; determining the direction vector of the parent key point corresponding to the key point pointing to the key point based on the three-dimensional position information of the key point and the three-dimensional position information of the corresponding parent key point; using the relative position information and direction vector corresponding to each key point, determining the posture information of the target object; performing three-dimensional reconstruction based on the posture information to obtain a three-dimensional object model of the target object.

[0080] Among them, the three-dimensional position information can be expressed by three-dimensional coordinates, and the relative position information is used to characterize the relative position of the key point relative to the root key point, that is, the difference between the three-dimensional coordinates. For example, the three-dimensional position information of the root key point is (x root ,y root , z root ), the three-dimensional position information of the key point is (x k ,y k , z k ), then the relative position information between the key point and the root key point is (x k -x root ,y k -y root , z k -z root ), that is, the position of the root key point (x root ,y root , z root ) points to the position of the key point (x k ,y k, z k ). The direction vector from the parent key point corresponding to the key point to the key point is obtained based on the three-dimensional position information of the key point and the three-dimensional position information of the corresponding parent key point. The direction vector from the parent key point corresponding to the key point to the key point is a vector with the three-dimensional position of the parent key point as the starting point and the three-dimensional position of the key point as the end point, that is, the starting point coordinates of the direction vector from the parent key point corresponding to the key point to the key point are the three-dimensional coordinates of the parent key point, and the coordinates of the end point are the three-dimensional coordinates of the key point. Posture information is a vector sequence that characterizes the posture of the target object.

[0081] Specifically, for each key point, the server can determine the relative position information between the key point and the root key point based on the three-dimensional position information of the key point and the three-dimensional position information of the root key point, and determine the direction vector of the key point's corresponding parent key point pointing to the key point based on the three-dimensional position information of the key point and the three-dimensional position information of the corresponding parent key point. Then, the relative position information and direction vector corresponding to the key points are input into the trained posture prediction network for posture information prediction, and the posture information of the target object is output. For example, Figure 5 As shown in the figure, the target object is a human hand. The relative position information between each keypoint and the root keypoint, and the direction vector pointing from the parent keypoint to each keypoint are input into the trained pose prediction network poseNet to obtain the pose information of the human hand, i.e., the pose parameters. The pose prediction network is used to predict pose information and can be composed of 7 fully connected layers.

[0082] In this embodiment, the relative position information and direction vector corresponding to each key point are obtained based on the three-dimensional position information corresponding to the root key point, each key point and the parent key point corresponding to each key point, and the posture information of the target formation is obtained based on the relative position information and direction vector corresponding to each key point. Since the posture information is determined based on the relative position information and direction vector corresponding to each key point, three-dimensional reconstruction is performed based on the posture information, and the obtained three-dimensional object model of the target object is more accurate, thereby improving the accuracy of three-dimensional reconstruction.

[0083] In some embodiments, performing three-dimensional reconstruction based on posture information to obtain a three-dimensional object model of the target object includes: determining the bone length between adjacent key points based on the three-dimensional position information of each key point; determining the shape information of the target object using the bone length between adjacent key points; and performing three-dimensional reconstruction based on the posture information and shape information to obtain a three-dimensional object model of the target object.

[0084] The bone length between adjacent keypoints is determined based on the 3D position information of each keypoint, which is the distance between each keypoint and its corresponding parent keypoint. Shape information is a vector sequence that represents the shape of the target object.

[0085] Specifically, the server can calculate the distance between each keypoint and its corresponding parent keypoint based on the 3D position information of each keypoint, obtain the bone length between adjacent keypoints, and then input the bone length between adjacent keypoints into the trained shape information prediction network for shape information prediction, outputting the shape information of the target formation. The shape prediction network is used for shape information prediction and can be composed of 7 fully connected layers.

[0086] In some embodiments, the server may input the posture information and shape information of the target object into the object 3D reconstruction network for 3D reconstruction, and obtain a 3D mesh model having the geometric shape of the target object, i.e., a 3D object model, for example, Figure 3 As shown, in the MANO parameter estimation stage, when the target object is a human hand, the hand's pose and shape information is input into the 3D hand reconstruction network, and a 3D hand model is output. The object 3D reconstruction network uses the pose and shape information of the target object for 3D reconstruction, and can be, for example, a MANO parameter model.

[0087] In some embodiments, the pose and shape information of the target object can be obtained by processing each target feature map of the target object image, and then the pose and shape information of the target object can be input into the three-dimensional object reconstruction model to obtain the three-dimensional object model of the target object. The three-dimensional object reconstruction method of the present application determines the pose and shape information of the target object based on the three-dimensional position information of each key point and the relative position information between the key points. It takes into account both the three-dimensional position information of each key point and the relative position between the key point and the corresponding parent key point, which can effectively improve the accuracy of the three-dimensional reconstruction of the object.

[0088] In this embodiment, the bone length between adjacent key points is determined based on the three-dimensional position information of each key point, and the shape information of the target object is determined using the bone length between adjacent key points. Thus, three-dimensional reconstruction can be performed based on the posture information and shape information of the target object to obtain a three-dimensional object model. Since the posture information and shape information are determined based on the three-dimensional position information of each key point and the relative position information between the key points, the accuracy of the three-dimensional reconstruction is improved.

[0089] In some embodiments, as Figure 6 As shown, a method for 3D reconstruction of an object is provided, which is described by taking the application of the method to a server as an example, and includes the following steps:

[0090] Step 602: extract image features from the target object image to obtain multiple target feature maps.

[0091] Wherein, the target object image is an image including the target object;

[0092] Step 604 : Detect key points based on multiple target feature maps to obtain a two-dimensional position heat map corresponding to each key point of the target object.

[0093] Step 606: Perform fusion processing based on the two-dimensional position heat map of each key point and each target feature map to obtain a first fused feature map corresponding to each key point.

[0094] In step 608, the parent vector is predicted using the first fused feature map corresponding to each key point, and a parent vector heat map corresponding to each key point under each three-dimensional coordinate axis is obtained.

[0095] Among them, the pixel position in the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, the pixel value at the pixel position in the parent vector heat map is the confidence that the vector component represented by the pixel position belongs to the parent key point vector, and the parent key point vector refers to the vector from the position of the parent key point corresponding to the key point to the position of the key point.

[0096] Step 610 , performing fusion processing based on the first fused feature map corresponding to each key point and the parent vector heat map under each three-dimensional coordinate axis, to obtain a second fused feature map corresponding to each key point.

[0097] In step 612, root vector prediction is performed based on the second fused feature map corresponding to each key point, and a root vector heat map corresponding to each key point under each three-dimensional coordinate axis is obtained.

[0098] Among them, the pixel position in the root vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, the pixel value at the pixel position in the root vector heat map is the confidence that the vector component represented by the pixel position in the root vector heat map belongs to the root key point vector, and the root key point vector refers to the vector pointing from the position of the root key point to the position of the key point.

[0099] Step 614 : Determine the three-dimensional position information corresponding to each key point based on the root vector heat map under each three-dimensional coordinate axis corresponding to each key point.

[0100] Step 616 : For each key point, determine the posture information of the target object based on the three-dimensional position information of the key point and the three-dimensional position information of the root key point.

[0101] Among them, each key point includes a root key point.

[0102] Step 618: Determine the shape information of the target object based on the three-dimensional position information of each key point.

[0103] Step 620 : Perform three-dimensional reconstruction based on the posture information and shape information to obtain a three-dimensional object model of the target object.

[0104] In this embodiment, parent vector prediction is performed based on the first fusion feature of the key point to obtain a parent vector heat map under each three-dimensional coordinate axis corresponding to each key point. Since the first fusion feature is obtained by fusion processing based on each target feature map and the two-dimensional position heat map, the accuracy of the parent vector heat map is improved; and since the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point contains the parent key point vector, the parent key point vector refers to a vector pointing from the position of the parent key point corresponding to the key point to the position of the key point, that is, the parent key point vector provides the relative position relationship between the key points, so that the three-dimensional position information corresponding to each key point obtained based on the parent vector heat map corresponding to each key point and each target feature map is more accurate; since the posture information and shape information of the target object are determined based on the three-dimensional position information of each key point and the relative position information between the key points, three-dimensional reconstruction based on the posture information and shape information of the target object is performed, and the accuracy of the three-dimensional object model of the target object obtained is higher, thereby improving the accuracy of the three-dimensional reconstruction.

[0105] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0106] Based on the same inventive concept, embodiments of the present application also provide a 3D object reconstruction device for implementing the aforementioned 3D object reconstruction method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the 3D object reconstruction device provided below can be found in the above-mentioned limitations of the 3D object reconstruction method and will not be further elaborated here.

[0107] In some embodiments, as Figure 7As shown, a 3D object reconstruction device is provided, comprising: a feature extraction module 702, a key point detection module 704, a vector prediction module 706, a 3D position prediction module 708 and a 3D reconstruction module 710, wherein:

[0108] The feature extraction module 702 is used to extract image features from the target object image to obtain multiple target feature maps; the target object image is an image including the target object.

[0109] The key point detection module 704 is used to detect key points based on multiple target feature maps to obtain a two-dimensional position heat map corresponding to each key point of the target object.

[0110] The vector prediction module 706 is used to predict the parent vector based on the two-dimensional position heat map of each key point and each target feature map, and obtain the parent vector heat map under each three-dimensional coordinate axis corresponding to each key point; the pixel position in the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, and the pixel value at the pixel position in the parent vector heat map is the confidence that the vector component represented by the pixel position belongs to the parent key point vector. The parent key point vector refers to the vector from the position of the parent key point corresponding to the key point to the position of the key point.

[0111] The three-dimensional position prediction module 708 is used to predict the three-dimensional position based on the parent vector heat map and each target feature map corresponding to each key point, and obtain the three-dimensional position information corresponding to each key point.

[0112] The 3D reconstruction module 710 is configured to perform 3D reconstruction based on the 3D position information of each key point to obtain a 3D object model of the target object.

[0113] In some embodiments, the vector prediction module is also used to: perform fusion processing based on the two-dimensional position heat map of each key point and each target feature map to obtain a fused feature map corresponding to each key point; use the fused feature map corresponding to each key point to perform parent vector prediction to obtain a parent vector heat map under each three-dimensional coordinate axis corresponding to each key point.

[0114] In some embodiments, each key point includes a root key point; the three-dimensional position prediction module is also used to: perform root vector prediction based on the parent vector heat map corresponding to each key point and each target feature map, to obtain the root vector heat map under each three-dimensional coordinate axis corresponding to each key point; the pixel position in the root vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, and the pixel value at the pixel position in the root vector heat map is the confidence that the vector component represented by the pixel position in the root vector heat map belongs to the root key point vector, and the root key point vector refers to the vector pointing from the position of the root key point to the position of the key point; based on the root vector heat map under each three-dimensional coordinate axis corresponding to each key point, the three-dimensional position information corresponding to each key point is determined.

[0115] In some embodiments, in terms of determining the three-dimensional position information corresponding to each key point based on the root vector heat map under each three-dimensional coordinate axis corresponding to each key point, the three-dimensional position prediction module is specifically used to: determine the position of the root key point to obtain the root key point position; for each key point, determine the root key point vector of the key point according to the root vector heat map under each three-dimensional coordinate axis corresponding to the key point; offset the root key point vector based on the root key point position to obtain the three-dimensional position information corresponding to the key point.

[0116] In some embodiments, each key point includes a root key point; the three-dimensional reconstruction module is also used to: for each key point, determine the relative position information between the key point and the root key point based on the three-dimensional position information of the key point and the three-dimensional position information of the root key point; determine the direction vector of the parent key point corresponding to the key point pointing to the key point based on the three-dimensional position information of the key point and the three-dimensional position information of the corresponding parent key point; use the relative position information and direction vector corresponding to each key point to determine the posture information of the target object; perform three-dimensional reconstruction based on the posture information to obtain a three-dimensional object model of the target object.

[0117] In some embodiments, in performing three-dimensional reconstruction based on posture information to obtain a three-dimensional object model of the target object, the three-dimensional reconstruction module is specifically used to: determine the bone length between adjacent key points based on the three-dimensional position information of each key point; determine the shape information of the target object using the bone length between adjacent key points; and perform three-dimensional reconstruction based on posture information and shape information to obtain a three-dimensional object model of the target object.

[0118] Each module in the above-mentioned object 3D reconstruction apparatus may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0119] In some embodiments, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store relevant data involved in the object three-dimensional reconstruction method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for three-dimensional reconstruction of an object is implemented.

[0120] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for three-dimensional reconstruction of an object is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0121] Those skilled in the art will understand that Figure 8 and Figure 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0122] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned object three-dimensional reconstruction method when executing the computer program.

[0123] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned object three-dimensional reconstruction method are implemented.

[0124] In some embodiments, a computer program product is provided, comprising a computer program, which implements the steps in the above-mentioned object three-dimensional reconstruction method when executed by a processor.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0126] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0127] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for three-dimensional reconstruction of an object, characterized in that: The method comprises: Extracting image features from a target object image to obtain a plurality of target feature maps; the target object image is an image including a target object; Detect key points based on the multiple target feature maps to obtain two-dimensional position heat maps corresponding to the key points of the target object; A parent vector is predicted based on the two-dimensional position heat map of each key point and each target feature map, and a parent vector heat map under each three-dimensional coordinate axis corresponding to each key point is obtained; the pixel position in the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, and the pixel value at the pixel position in the parent vector heat map is the confidence that the vector component represented by the pixel position belongs to the parent key point vector, and the parent key point vector refers to the vector from the position of the parent key point corresponding to the key point to the position of the key point; Predicting the three-dimensional position based on the parent vector heat map corresponding to each key point and each target feature map, to obtain the three-dimensional position information corresponding to each key point; Three-dimensional reconstruction is performed based on the three-dimensional position information of each key point to obtain a three-dimensional object model of the target object.

2. The method according to claim 1, characterized in that The parent vector prediction is performed based on the two-dimensional position heat map of each key point and each target feature map to obtain the parent vector heat map under each three-dimensional coordinate axis corresponding to each key point, including: Performing fusion processing based on the two-dimensional position heat map of each key point and each target feature map to obtain a fusion feature map corresponding to each key point; The parent vector is predicted using the fusion feature maps corresponding to each of the key points to obtain a parent vector heat map under each three-dimensional coordinate axis corresponding to each of the key points.

3. The method according to claim 1, characterized in that Each key point includes a root key point; the three-dimensional position prediction based on the parent vector heat map corresponding to each key point and each target feature map is performed to obtain the three-dimensional position information corresponding to each key point, including: Root vector prediction is performed based on the parent vector heat map corresponding to each of the key points and each of the target feature maps, and a root vector heat map under each three-dimensional coordinate axis corresponding to each of the key points is obtained; the pixel position in the root vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, and the pixel value at the pixel position in the root vector heat map is the confidence that the vector component represented by the pixel position in the root vector heat map belongs to the root key point vector, and the root key point vector refers to the vector from the position of the root key point to the position of the key point; Based on the root vector heat maps under the three-dimensional coordinate axes corresponding to each of the key points, the three-dimensional position information corresponding to each of the key points is determined.

4. The method according to claim 3, characterized in that The determining of the three-dimensional position information corresponding to each key point based on the root vector heat map under each three-dimensional coordinate axis corresponding to each key point includes: Determine the position of the root key point to obtain the root key point position; For each key point, determining the root key point vector of the key point according to the root vector heat map under each three-dimensional coordinate axis corresponding to the key point; The root key point vector is offset based on the root key point position to obtain the three-dimensional position information corresponding to the key point.

5. The method according to claim 1, characterized in that The key points include a root key point; and performing three-dimensional reconstruction based on the three-dimensional position information of each key point to obtain a three-dimensional object model of the target object includes: For each of the key points, determining relative position information between the key point and the root key point based on the three-dimensional position information of the key point and the three-dimensional position information of the root key point; Determine a direction vector pointing from the parent key point corresponding to the key point to the key point based on the three-dimensional position information of the key point and the three-dimensional position information of the corresponding parent key point; Determine the posture information of the target object using the relative position information and direction vectors corresponding to each of the key points; Perform three-dimensional reconstruction based on the posture information to obtain a three-dimensional object model of the target object.

6. The method according to claim 5, characterized in that The performing three-dimensional reconstruction based on the posture information to obtain a three-dimensional object model of the target object includes: Determine the bone length between adjacent key points based on the three-dimensional position information of each key point; Determining shape information of the target object using the bone lengths between the adjacent key points; Three-dimensional reconstruction is performed based on the posture information and the shape information to obtain a three-dimensional object model of the target object.

7. A device for three-dimensional reconstruction of an object, characterized in that: The device comprises: A feature extraction module is used to extract image features of a target object image to obtain a plurality of target feature maps; the target object image is an image including a target object; A key point detection module is used to detect key points based on the multiple target feature maps to obtain a two-dimensional position heat map corresponding to each key point of the target object; A vector prediction module is configured to predict a parent vector based on the two-dimensional position heat map of each key point and each target feature map, thereby obtaining a parent vector heat map under each three-dimensional coordinate axis corresponding to each key point; the pixel position in the parent vector heat map under the three-dimensional coordinate axis corresponding to the key point represents the vector component under the three-dimensional coordinate axis, the pixel value at the pixel position in the parent vector heat map is the confidence that the vector component represented by the pixel position belongs to the parent key point vector, and the parent key point vector refers to a vector pointing from the position of the parent key point corresponding to the key point to the position of the key point; A three-dimensional position prediction module is used to predict the three-dimensional position based on the parent vector heat map corresponding to each key point and each target feature map, to obtain the three-dimensional position information corresponding to each key point; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction based on the three-dimensional position information of each key point to obtain a three-dimensional object model of the target object.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images