A three-dimensional reconstruction method, device, electronic device and readable storage medium

Abstract

The present application relates to the field of image processing technology, and specifically discloses a three-dimensional reconstruction method, a three-dimensional reconstruction device, an electronic device, and a computer-readable storage medium. The three-dimensional reconstruction method comprises: extracting features from an image of an object to be reconstructed to obtain a feature vector, wherein the feature vector is used to characterize the shape feature information of the object to be reconstructed; generating a feature map based on the feature vector and a preset template for the object to be reconstructed, wherein the preset template is used to characterize the three-dimensional structure information of the object to be reconstructed; inputting the feature map into a trained graph convolutional neural network to obtain a three-dimensional reconstruction result of the object to be reconstructed. Through the solution of the present application, the stability of three-dimensional reconstruction can be improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a three-dimensional reconstruction method, a three-dimensional reconstruction device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Three-dimensional reconstruction of human body parts has always been a hot topic in computer vision and has been widely used in the fields of virtual reality (VR) and augmented reality (AR). Traditional three-dimensional reconstruction technology relies on relatively complex and expensive equipment, such as three-dimensional scanners, multi-view cameras or inertial sensors. Although three-dimensional reconstruction technologies based on single images have been developed, these three-dimensional reconstruction technologies still have problems such as unstable reconstruction effects. Summary of the invention

[0003] The present application provides a three-dimensional reconstruction method, a three-dimensional reconstruction device, an electronic device and a computer-readable storage medium, which can solve the problem of unstable reconstruction effect existing in the existing three-dimensional reconstruction technology.

[0004] In a first aspect, the present application provides a three-dimensional reconstruction method, comprising:

[0005] Extracting features from the image of the object to be reconstructed to obtain a feature vector, wherein the feature vector is used to represent shape feature information of the object to be reconstructed;

[0006] Generate a feature map according to the feature vector and a preset template for the object to be reconstructed, wherein the preset template is used to characterize the three-dimensional structure information of the object to be reconstructed;

[0007] The feature map is input into a trained graph convolutional neural network (GCN) to obtain a three-dimensional reconstruction result of the object to be reconstructed.

[0008] In a second aspect, the present application provides a three-dimensional reconstruction device, comprising:

[0009] An extraction module is used to extract features from the image of the object to be reconstructed to obtain a feature vector, wherein the feature vector is used to represent the shape feature information of the object to be reconstructed;

[0010] A generating module, used for generating a feature map according to the feature vector and a preset template for the object to be reconstructed, wherein the preset template is used for representing the three-dimensional structure information of the object to be reconstructed;

[0011] The reconstruction module is used to input the above feature map into the trained graph convolutional neural network to obtain the three-dimensional reconstruction result of the above object to be reconstructed.

[0012] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0014] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0015] Compared with the prior art, the present application has the following beneficial effects: for an image of an object to be reconstructed, the present application first extracts features from the image to obtain a feature vector for characterizing the shape feature information of the object to be reconstructed, then combines the feature vector with a preset template for the object to be reconstructed to generate a feature map, and finally inputs the feature map into a trained graph convolutional neural network to obtain a three-dimensional reconstruction result of the object to be reconstructed. In the above process, by combining the feature vector with the preset template, the feature map finally generated not only includes the shape features of the object to be reconstructed, but also obtains the three-dimensional structural information of the object to be reconstructed displayed by the preset template. As a result, the trained graph convolutional neural network can better process the feature map, and the obtained three-dimensional reconstruction result will be more accurate, ensuring the stability of the three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a schematic diagram of the implementation process of the three-dimensional reconstruction method provided in the embodiment of the present application;

[0018] Figure 2 This is an example diagram of a preset template when the object to be reconstructed is a human body provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of the structure of the first functional module of the graph convolutional neural network provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of the structure of the i-th functional module of the graph convolutional neural network provided in an embodiment of the present application;

[0021] Figure 5 It is a schematic diagram of the structure of the Nth functional module of the graph convolutional neural network provided in an embodiment of the present application;

[0022] Figure 6 This is an example diagram of the overall structure of a graph convolutional neural network provided in an embodiment of the present application;

[0023] Figure 7 is an example diagram of the working framework of the three-dimensional reconstruction method provided in an embodiment of the present application;

[0024] Figure 8 is a structural block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0025] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0027] At present, the existing three-dimensional reconstruction technology still has the problem of unstable reconstruction effect. To solve this problem, the embodiment of the present application proposes a three-dimensional reconstruction method, a three-dimensional reconstruction device, an electronic device and a computer-readable storage medium. After extracting the feature vector of the object to be reconstructed in the image, the feature vector can be combined with a preset template representing the three-dimensional structural information of the object to be reconstructed to obtain a feature map, so that the feature map not only contains the shape features of the object to be reconstructed, but also contains the three-dimensional structural information of the object to be reconstructed displayed by the preset template, which can be better processed by the trained graph convolutional neural network, and a more accurate three-dimensional reconstruction result can be obtained, ensuring the stability of the three-dimensional reconstruction. In order to illustrate the technical solution proposed in the present application, it is described below through specific embodiments.

[0028] The following is an explanation of the three-dimensional reconstruction method proposed in the embodiment of the present application. Figure 1 , the implementation process of the 3D reconstruction method is described in detail as follows:

[0029] Step 101 , extracting features from the image of the object to be reconstructed to obtain a feature vector.

[0030] In an embodiment of the present application, the electronic device may photograph the object to be reconstructed by means of its own camera, so that the electronic device obtains an image of the object to be reconstructed; alternatively, a third-party device equipped with a camera may photograph the object to be reconstructed, and after obtaining the image of the object to be reconstructed, the image may be transmitted to the electronic device by wireless or wired means, so that the electronic device obtains the image of the object to be reconstructed. The method for obtaining the image of the object to be reconstructed is not limited herein.

[0031] After obtaining the image of the object to be reconstructed, the electronic device can perform feature extraction on the image to obtain a feature vector. Considering that the three-dimensional reconstruction result obtained by the three-dimensional reconstruction operation is mainly to restore the posture of the object to be reconstructed, and the posture is most related to the shape characteristics of the object to be reconstructed, the feature extraction operation here is mainly aimed at the shape feature information of the object to be reconstructed; that is, the obtained feature vector is actually used to characterize the shape feature information of the object to be reconstructed. It can be understood that the shape feature information includes contour features describing the boundary shape of the object to be reconstructed and / or regional features describing the internal shape of the object to be reconstructed, etc., which are not limited here.

[0032] In some embodiments, in addition to the image information of the object to be reconstructed, the image of the object to be reconstructed may also contain some redundant information and noise information. To prevent the redundant information and noise information from affecting the accuracy of subsequent feature extraction, the electronic device may first pre-process the image, such as segmentation and resizing operations; then step 101 may include:

[0033] A1. Segment the image based on the object to be reconstructed to obtain a local image.

[0034] The electronic device may first perform border detection on the image based on the object to be reconstructed, that is, identify the border of the object to be reconstructed from the image. The border is usually a rectangular frame; of course, it can also be a border of other preset shapes, which is not limited here. The image area within the border can be regarded as the region of interest. The image to be processed is segmented based on the border, and the region of interest can be segmented from the image, thereby eliminating a large amount of redundant information and noise information contained in the background of the image to a certain extent, and obtaining a local image mainly containing the object to be reconstructed, and the object to be reconstructed can be located as close to the center of the local image as possible.

[0035] Just as an example, when the object to be reconstructed is a human body, the electronic device may use OpenPose, a two-dimensional key point detection technology for the human body, to perform border detection on the image.

[0036] A2. Adjust the size of the partial image to the preset size.

[0037] To facilitate the generation of subsequent feature maps, the electronic device can unify the size of the local image: if the size of the local image is inconsistent with the preset size, the local image is scaled until the size of the local image is consistent with the preset size. As an example only, the preset size can be: 224×224, in pixels. The number of channels of the local image is usually 3, which is used to represent the three channels of red, green, and blue (RGB). That is, the final size of the local image is: 224×224×3.

[0038] A3. Extract features of the resized local image by using a Convolutional Neural Network (CNN) encoder to obtain a feature vector.

[0039] The electronic device can pre-train the convolutional neural network for classification tasks on a given data set. The pre-training process can refer to the general training process of the neural network at present, which will not be described here. After the pre-training is completed, the classification layer in the convolutional neural network is removed, and the feature extraction layer before the classification layer is retained. The retained result constitutes the encoder.

[0040] Just as an example, the convolutional neural network may be ResNet50, the given data set may be the ImageNet data set, and the output of the encoder of the convolutional neural network is a 2048-dimensional feature vector.

[0041] Step 102: Generate a feature map based on the feature vector and a preset template for the object to be reconstructed.

[0042] In the embodiment of the present application, the preset template is used to characterize the three-dimensional structural information of the object to be reconstructed, specifically the three-dimensional structural information of the object to be reconstructed in a specified posture. Different preset templates are set according to different types of objects to be reconstructed. As an example only, when the object to be reconstructed is a human body, the preset template is a human body grid map; when the object to be reconstructed is a hand, the preset template is a hand grid map.

[0043] In some embodiments, considering that the graph convolutional neural network is a neural network structure proposed to better process the graph data structure, in order to improve the efficiency of the graph convolutional neural network model for three-dimensional reconstruction, the electronic device can express the preset template in a graph structure and combine it with the feature vector to generate a feature map. Since the preset template is usually a grid graph, it can be regarded as an undirected graph consisting of a vertex set and an edge set, which can be expressed as G = (V, E); wherein G represents the grid graph; V represents the vertex set; and E represents the edge set. Since the grid in the grid graph is composed of triangular faces, the grid graph can also be regarded as an undirected graph consisting of a vertex set and a triangular face set, which can be expressed as M = (V, F); wherein M represents the vertex information matrix of the grid graph; V represents the vertex set; and F represents the triangular face set, and each triangular face in the triangular face set F is composed of a triangle composed of three vertices in the vertex set V. It can be understood that the information of the edge set of the grid graph is actually contained in the triangular face set.

[0044] In an application scenario, the object to be reconstructed is a human body, and the preset template is a human body grid map, then step 101 may include:

[0045] B1. Construct a graph structure of a preset format based on the human body grid graph.

[0046] As an example only, the human body grid diagram used by the electronic device for the human body may be a standard template defined by the SMPL (Skinned Multi-Person Linear) model. Figure 2 As shown, the human body mesh diagram represents the three-dimensional mesh of the human body in T-Pose.

[0047] Through the graph structure of the general format shown above, the electronic device can first represent the human body grid graph as M smpl =(V,F), where V represents the vertex set of the human body mesh, which has a total of 6890 vertices; F represents the triangular face set of the human body mesh, and three vertices constitute a triangular face. smpl =(V,F) to convert, and then get the graph structure expression M in the preset format smpl =(V,A); where V still represents the vertex set; A represents the adjacency matrix of the human body mesh graph, A∈{0,1} 6890×6890 , which is used to indicate that the value of the element in the adjacency matrix is ​​0 or 1, specifically: if the i-th vertex is connected to the j-th vertex, then (A)ij=1, otherwise (A)ij=0. The electronic device expresses the preset template of the object to be reconstructed through the graph structure in the preset format, which provides a basis for the subsequent use of the graph convolutional neural network to predict the three-dimensional coordinates of the human body mesh.

[0048] In some embodiments, in order to reduce the computational complexity of subsequent graph convolution operations, the electronic device can perform 4-fold downsampling of the standard template defined by the SMPL model, and use the downsampling result as the preset template (i.e., the human body mesh map). It can be understood that the number of vertices of the human body mesh map obtained after 4-fold downsampling is reduced to 1723. Subsequently, the 3D reconstruction result output by the graph convolutional neural network only needs to be upsampled 4 times to obtain the final 3D reconstruction result and complete the reconstruction task.

[0049] For the downsampled human body mesh map, the final preset format graph structure is represented as follows:

[0050] M h =(V h ,A h ),V h ∈R 1723×3 ,A h ∈{0,1} 1723×1723

[0051] Among them, M h Represents the vertex information matrix of the downsampled human body mesh; V h V represents the vertex set of the downsampled human body mesh graph; h ∈R 1723×3 Indicates that the three-dimensional coordinate values ​​of the 1723 vertices contained in the vertex set are all real numbers; A h A represents the adjacency matrix of the downsampled human body grid graph. The meaning of the adjacency matrix has been explained above and will not be repeated here. h ∈{0,1} 1723×1723 Indicates that the value of the element in the adjacency matrix is ​​0 or 1.

[0052] B2. Fusion and splicing of feature vectors and graph structures to obtain feature graphs.

[0053] The 2048-dimensional feature vector obtained in step 101 is f∈R 2048 The vertex information matrix of the human body mesh image obtained after 4 times downsampling in step B1 is M h ∈R 1723×3 After the two are fused and spliced, a feature map can be obtained, which is the input of the subsequent graph convolutional neural network and can be expressed as F in ∈R 1723×2051 The above operation can be understood as: splicing the 2048-dimensional feature vector to each vertex.

[0054] In another application scenario, the object to be reconstructed is a hand, and the preset template is a hand mesh map. The process of generating its feature map is similar to steps B1 and B2, except that the preset template has changed. As an example only, the hand mesh map used by the electronic device for the hand can be a standard template defined by the MANO (hand Model with Articulated and Non-rigiddefOrmations) model. It should be noted that since the hand mesh map contains a small number of vertices (usually more than 700), there is no need to downsample the hand mesh map.

[0055] In summary, the feature map actually combines the vertex position information of the preset template and the shape feature information of the part to be reconstructed shown in the image.

[0056] Step 103: input the feature map into the trained graph convolutional neural network to obtain a three-dimensional reconstruction result of the object to be reconstructed.

[0057] In the embodiment of the present application, the graph convolutional neural network takes the feature map as input and finally outputs the transformed mesh vertex position information of the object to be reconstructed as the 3D reconstruction result. The overall structure of the graph convolutional neural network is introduced below:

[0058] The graph convolutional neural network includes N functional modules connected in series; wherein the input of the first functional module is the input of the graph convolutional neural network, the output of the Nth functional module is the output of the graph convolutional neural network, and N is an integer greater than 2. It can be understood that the first functional module is mainly used to receive the input of the graph convolutional neural network, the i-th functional module is mainly used to perform data calculation and transmission operations, and the N-th functional module is mainly used to output the final predicted mesh vertex position information of the object to be reconstructed, wherein i is an integer greater than 1 and less than N.

[0059] Specifically, each functional module includes the following three basic units: convolution unit, normalization unit and activation function unit. The specific structure of each functional module is introduced below:

[0060] See also Figure 3 , Figure 3 The first functional module has at least three designated structures ( Figure 3Only 3 are shown in the figure), and the at least three specified structures are connected in series in sequence, and the specified structure includes a convolution unit, a normalization unit and an activation function unit connected in series in sequence; in the at least three specified structures: the input of the first specified structure is the input of the first functional module (that is, the input of the graph convolutional neural network); the residual between the output of the last specified structure and the input of the first functional module (that is, the input of the graph convolutional neural network) is the output of the first functional module.

[0061] See also Figure 4 , Figure 4 The structure of the i-th functional module is shown in FIG. 1 . For the i-th functional module, it includes at least two specified structures ( Figure 4 Only two are shown in the figure), and the at least two designated structures are connected in series in sequence. The designated structure is the same as the designated structure of the first functional module and will not be repeated here. In the at least two designated structures: the input of the first designated structure is the output of the i-1th functional module, and the residual between the output of the last designated structure and the output of the i-1th functional module is the output of the i-th functional module.

[0062] See also Figure 5 , Figure 5 The structural diagram of the Nth functional module is given. For the Nth functional module, it includes a convolution unit, a normalization unit, an activation function unit and a convolution unit connected in series in sequence; wherein, the input of the first convolution unit is the output of the N-1th functional module, and the residual between the output of the second convolution unit and the output of the N-1th functional module is the output of the Nth functional module. It can be understood that since the role of the Nth functional module is to output the final predicted mesh vertex position information of the object to be reconstructed, the data output by the second convolution unit of the Nth functional module no longer needs to be processed by data normalization and activation function.

[0063] See also Figure 6 , Figure 6 The overall structure of a graph convolutional neural network consisting of four functional modules is given. For ease of understanding, the parameter f in and parameter f out To represent the change in the feature dimension size of each functional module during the graph convolution operation. Figure 6 For example, the first functional module includes three convolutional units, and the change in the size of its feature dimension can be expressed as (f in ,f out1 ,f out2 ,f out3 ) in the form of in is the feature dimension size of the initial input, f out1 、f out2 and fout3 are the feature dimensions of the outputs of the three convolutional units. Since the normalization unit and the activation function unit do not change the feature dimension, the feature dimension of the second convolutional unit is the same as the feature dimension of the first convolutional unit, and so on. Figure 6 For example, the second, third and fourth functional modules each include two convolutional units, and the change in the size of its feature dimension can be expressed as (f in ,f out1 ,f out2 ) format, the meaning of its parameters can be found in the previous text and will not be repeated here.

[0064] As an example only, when the object to be reconstructed is a human body, the obtained feature map F in ∈R 1723×2051 The feature map passes through the graph convolutional neural network, and the final output F of the graph convolutional neural network can be obtained. out ∈R 1723×3 After upsampling, we get M out ∈R 6890×3 , that is, the final three-dimensional reconstruction result of the human body is obtained.

[0065] In one example, the convolution unit may be a Chebyshev convolution unit, which specifically uses Chebyshev polynomials to construct a Chebyshev convolution algorithm. Through the Chebyshev convolution unit, the processing speed of the graph convolutional neural network can be accelerated to a certain extent, and the efficiency of three-dimensional reconstruction can be improved.

[0066] The Chebyshev polynomial is expressed as follows:

[0067] T0(x)=1;T1(x)=x;T n+1 (x) = 2xT n (x)-T n-1 (x)

[0068] Based on the Chebyshev polynomial, the Chebyshev convolution algorithm can be obtained, which is expressed as follows:

[0069]

[0070] Among them, F in ∈R N×fin Represents the input features.

[0071] F out ∈R N×fout Represents the output features.

[0072] K represents the use of K-order Chebyshev polynomials. In the embodiment of the present application, each Chebyshev convolution unit of the graph convolutional neural network takes K=3.

[0073] θ k ∈R fin×fout Represents the feature change matrix, and the parameters in it are the values ​​that the graph convolutional neural network needs to learn.

[0074] Represents the scaled Laplacian matrix of the preset template. When the object to be reconstructed is a human body and the preset template used is the downsampling result of the standard template defined by the SMPL model, N is the number of vertices after downsampling, 1723. The scaled Laplacian matrix is ​​specifically:

[0075]

[0076]

[0077] Where I is a unit matrix, (D h ) ij =∑ j (A h ) ij is a diagonal matrix, λ max YesL p The largest eigenvalue of a matrix.

[0078] For ease of understanding, taking K=3 as an example, the Chebyshev convolution algorithm given above is expanded as follows:

[0079]

[0080]

[0081] W=[θ0,θ1,θ2] T ∈R 3fin×fout

[0082] Among them, L is an intermediate parameter and has no actual physical meaning. W is the parameter that the graph convolutional neural network needs to learn.

[0083] In some embodiments, when the object to be reconstructed is a human body, the graph convolutional neural network can be trained using the two datasets Human3.6M and MSCOCO. Specifically, since these two datasets do not store the real human body meshes in each training sample, but only store the position information of the real human body's three-dimensional joints, it is necessary to pre-fit the real human body mesh based on the position information of the real human body's three-dimensional joints of each training sample to obtain a high-precision real human body mesh, which can be used as a strong label and put into the training process of the graph convolutional neural network for use. That is, the real human body mesh mentioned here is actually a high-precision result fitted based on the real human body's three-dimensional joints. It can be understood that the training process of the graph convolutional neural network is basically the same as the training process of a general neural network, except that a new loss function is used to make the three-dimensional reconstruction results output by the trained graph convolutional neural network model smoother and more complete, and its practicality is also higher. The loss function is:

[0084] loss = λ a L v +λ b L j +λ c L n +λ d L e

[0085] Among them, λ a , b , c and d are all hyperparameters.

[0086] Among them, L v Represents the mesh loss, which is used to describe the position difference between the real human mesh and the predicted human mesh. M* represents the position of each vertex of the real human mesh, M represents the position of each vertex of the predicted human mesh, and L1 loss is used. Then the mesh loss L v It is expressed as follows:

[0087] L v =||MM*||1

[0088] Among them, L j It represents the 3D joint loss, which is used to describe the position difference between the real 3D joint of the human body and the predicted 3D joint of the human body. 3D* represents the position of the real human 3D joint, JM represents the predicted human joint position, J∈R v ×N Represents the matrix of joints extracted from the human body mesh, M represents the position of each vertex of the predicted human body mesh, and using L1 loss, the three-dimensional joint loss is expressed as follows:

[0089] Lj =||JM-J 3D* ||1

[0090] Among them, L n Represents the surface normal loss, which is used to describe the angle difference between the normal vector of the triangular face of the real human mesh and the normal vector of the triangular face of the predicted human mesh. f represents the triangular face of the predicted human mesh, n f * represents the unit normal vector of the triangular face corresponding to f in the real human body mesh, m i and m j Represent the coordinates of the two vertices in f, then the surface normal loss L n It is expressed as follows:

[0091]

[0092] Among them, L e It represents the surface edge loss, which is used to describe the difference between the side length of the triangular face of the real human body mesh and the side length of the triangular face of the predicted human body mesh. f represents the triangular face of the predicted human body mesh, m i and m j Respectively represent the coordinates of two vertices in f, m * i and Respectively represent the real human body mesh and m i and m j The corresponding vertex coordinates, then the surface edge loss L e It is expressed as follows:

[0093]

[0094] For easier understanding, see Figure 7 , Figure 7 Taking the human body as an example, a working framework example of the three-dimensional reconstruction method in the embodiment of the present application is given. The working framework consists of two parts, namely, an encoder based on a convolutional neural network and a human body three-dimensional vertex regressor based on a graph convolutional neural network. After photographing a certain person, the original image of the human body is obtained. The original image is used as the initial input. After preprocessing, a local image is obtained. The local image is encoded into a set of feature vectors through an encoder based on a convolutional neural network. The set of feature vectors is fused and spliced ​​with the grid vertex position information in the preset human body grid map to form a feature map as the input of the graph convolutional neural network. Finally, the graph convolutional neural network will regress a new set of grid vertex position information to make it consistent with the two-dimensional observation of the human body in the original image, and complete the three-dimensional reconstruction task of the human body.

[0095] As can be seen from the above, through the embodiment of the present application, the image of the object to be reconstructed is first feature extracted to obtain a feature vector for characterizing the shape feature information of the object to be reconstructed, and then the feature vector is combined with a preset template for the object to be reconstructed to generate a feature map, and finally the feature map is input into the trained graph convolutional neural network to obtain the three-dimensional reconstruction result of the object to be reconstructed. In the above process, through the combination of the feature vector and the preset template, the feature map finally generated not only contains the shape features of the object to be reconstructed, but also obtains the three-dimensional structure information of the object to be reconstructed displayed by the preset template. As a result, the trained graph convolutional neural network can better process the feature map, and the obtained three-dimensional reconstruction result will be more accurate, ensuring the stability of the three-dimensional reconstruction.

[0096] Corresponding to the three-dimensional reconstruction method provided above, the present application embodiment also provides a three-dimensional reconstruction device. Figure 8 As shown, the 3D reconstruction device 800 includes:

[0097] An extraction module 801 is used to extract features from the image of the object to be reconstructed to obtain a feature vector, wherein the feature vector is used to represent shape feature information of the object to be reconstructed;

[0098] A generating module 802, configured to generate a feature map according to the feature vector and a preset template for the object to be reconstructed, wherein the preset template is used to characterize the three-dimensional structure information of the object to be reconstructed;

[0099] The reconstruction module 803 is used to input the above-mentioned feature map into the trained graph convolutional neural network to obtain the three-dimensional reconstruction result of the above-mentioned object to be reconstructed.

[0100] Optionally, the graph convolutional neural network includes N functional modules connected in series;

[0101] The input of the first functional module is the input of the graph convolutional neural network, the output of the Nth functional module is the output of the graph convolutional neural network, and N is an integer greater than 2.

[0102] Among them, the above functional modules include a convolution unit, a normalization unit and an activation function unit.

[0103] Optionally, the first functional module includes at least three specified structures, which are sequentially connected in series, and the specified structure includes the convolution unit, the normalization unit and the activation function unit which are sequentially connected in series;

[0104] In the at least three specified structures mentioned above: the input of the first specified structure is the input of the first of the above functional modules, and the residual between the output of the last specified structure and the input of the first of the above functional modules is the output of the first of the above functional modules.

[0105] Optionally, the i-th functional module includes at least two specified structures, which are sequentially connected in series, and the specified structure includes the convolution unit, the normalization unit and the activation function unit which are sequentially connected in series, and the i is an integer greater than 1 and less than N;

[0106] In the above-mentioned at least two specified structures: the input of the first specified structure is the output of the i-1th above-mentioned functional module, and the residual between the output of the last specified structure and the output of the i-1th above-mentioned functional module is the output of the i-th above-mentioned functional module.

[0107] Optionally, the Nth functional module includes the convolution unit, the normalization unit, the activation function unit and the convolution unit connected in series in sequence;

[0108] In the Nth functional module: the input of the first convolution unit is the output of the N-1th functional module, and the residual between the output of the second convolution unit and the output of the N-1th functional module is the output of the Nth functional module.

[0109] Optionally, the convolution unit is a Chebyshev convolution unit.

[0110] Optionally, when the object to be reconstructed is a human body, the preset template is a human body grid map; the generating module 802 includes:

[0111] A construction unit, configured to construct a graph structure in a preset format based on the human body mesh graph, wherein the graph structure includes vertex information of the human body mesh graph;

[0112] The concatenation unit is used to merge and concatenate the feature vector with the graph structure to obtain the feature graph.

[0113] Optionally, the above-mentioned 3D reconstruction device 800 further includes:

[0114] A calculation module, used for calculating the total loss of the graph convolutional neural network based on mesh loss, three-dimensional joint loss, surface normal loss and surface edge loss during the training process of the graph convolutional neural network;

[0115] Among them, the above-mentioned mesh loss is used to describe the position difference between the real human body mesh and the predicted human body mesh;

[0116] The above 3D joint loss is used to describe the position difference between the real 3D joints of the human body and the predicted 3D joints of the human body;

[0117] The above surface normal loss is used to describe the angle difference between the normal vector of the triangular face of the real human mesh and the normal vector of the triangular face of the predicted human mesh;

[0118] The above surface edge loss is used to describe the length difference between the side lengths of the triangular faces of the real human mesh and the side lengths of the triangular faces of the predicted human mesh.

[0119] Optionally, the extraction module 801 includes:

[0120] A segmentation unit, used for segmenting the image based on the object to be reconstructed to obtain a local image;

[0121] An adjusting unit, used for adjusting the size of the partial image to a preset size;

[0122] The extraction unit is used to extract features from the resized local image by using an encoder of a convolutional neural network to obtain the feature vector.

[0123] As can be seen from the above, through the embodiment of the present application, the image of the object to be reconstructed is first feature extracted to obtain a feature vector for characterizing the shape feature information of the object to be reconstructed, and then the feature vector is combined with a preset template for the object to be reconstructed to generate a feature map, and finally the feature map is input into the trained graph convolutional neural network to obtain the three-dimensional reconstruction result of the object to be reconstructed. In the above process, through the combination of the feature vector and the preset template, the feature map finally generated not only contains the shape features of the object to be reconstructed, but also obtains the three-dimensional structure information of the object to be reconstructed displayed by the preset template. As a result, the trained graph convolutional neural network can better process the feature map, and the obtained three-dimensional reconstruction result will be more accurate, ensuring the stability of the three-dimensional reconstruction.

[0124] Corresponding to the three-dimensional reconstruction method provided above, the present application embodiment also provides an electronic device. Fig. 9 The electronic device 9 in the embodiment of the present application includes: a memory 901, one or more processors 902 ( Fig. 9 Only one is shown) and a computer program stored in the memory 901 and executable on the processor. The memory 901 is used to store software programs and units, and the processor 902 executes various functional applications and diagnoses by running the software programs and units stored in the memory 901 to obtain the resources corresponding to the above-mentioned preset events. Specifically, the processor 902 implements the following steps when running the above-mentioned computer program stored in the memory 901:

[0125] Extracting features from the image of the object to be reconstructed to obtain a feature vector, wherein the feature vector is used to represent shape feature information of the object to be reconstructed;

[0126] Generate a feature map according to the feature vector and a preset template for the object to be reconstructed, wherein the preset template is used to characterize the three-dimensional structure information of the object to be reconstructed;

[0127] The feature map is input into the trained graph convolutional neural network to obtain the three-dimensional reconstruction result of the object to be reconstructed.

[0128] Assuming that the above is the first possible implementation, in a second possible implementation provided on the basis of the first possible implementation, the graph convolutional neural network includes N functional modules connected in series;

[0129] The input of the first functional module is the input of the graph convolutional neural network, the output of the Nth functional module is the output of the graph convolutional neural network, and N is an integer greater than 2.

[0130] Among them, the above functional modules include a convolution unit, a normalization unit and an activation function unit.

[0131] In a third possible implementation provided on the basis of the second possible implementation, the first functional module includes at least three specified structures, which are sequentially connected in series, and the specified structure includes the convolution unit, the normalization unit, and the activation function unit which are sequentially connected in series;

[0132] In the at least three specified structures mentioned above: the input of the first specified structure is the input of the first of the above functional modules, and the residual between the output of the last specified structure and the input of the first of the above functional modules is the output of the first of the above functional modules.

[0133] In a fourth possible implementation provided on the basis of the second possible implementation, the i-th functional module includes at least two specified structures, which are sequentially connected in series, and the specified structure includes the convolution unit, the normalization unit, and the activation function unit which are sequentially connected in series, and the i is an integer greater than 1 and less than N.

[0134] In the above-mentioned at least two specified structures: the input of the first specified structure is the output of the i-1th above-mentioned functional module, and the residual between the output of the last specified structure and the output of the i-1th above-mentioned functional module is the output of the i-th above-mentioned functional module.

[0135] In a fifth possible implementation provided as a basis of the second possible implementation, the Nth functional module includes the convolution unit, the normalization unit, the activation function unit and the convolution unit connected in series in sequence;

[0136] In the Nth functional module: the input of the first convolution unit is the output of the N-1th functional module, and the residual between the output of the second convolution unit and the output of the N-1th functional module is the output of the Nth functional module.

[0137] In a sixth possible implementation provided on the basis of the second possible implementation, or on the basis of the third possible implementation, or on the basis of the fourth possible implementation, or on the basis of the fifth possible implementation, the convolution unit is a Chebyshev convolution unit.

[0138] In a seventh possible implementation provided on the basis of the first possible implementation, when the object to be reconstructed is a human body, the preset template is a human body grid map; generating a feature map according to the feature vector and the preset template for the object to be reconstructed includes:

[0139] Constructing a graph structure in a preset format based on the human body mesh graph, wherein the graph structure includes vertex information of the human body mesh graph;

[0140] The above feature vector is fused and spliced ​​with the above graph structure to obtain the above feature graph.

[0141] In an eighth possible implementation provided on the basis of the seventh possible implementation, the processor 902 further implements the following steps when running the computer program stored in the memory 901:

[0142] During the training process of the graph convolutional neural network, the total loss of the graph convolutional neural network is calculated based on the mesh loss, the three-dimensional joint loss, the surface normal loss and the surface edge loss;

[0143] Among them, the above-mentioned mesh loss is used to describe the position difference between the real human body mesh and the predicted human body mesh;

[0144] The above 3D joint loss is used to describe the position difference between the real 3D joints of the human body and the predicted 3D joints of the human body;

[0145] The above surface normal loss is used to describe the angle difference between the normal vector of the triangular face of the real human mesh and the normal vector of the triangular face of the predicted human mesh;

[0146] The above surface edge loss is used to describe the length difference between the side lengths of the triangular faces of the real human mesh and the side lengths of the triangular faces of the predicted human mesh.

[0147] In a ninth possible implementation provided on the basis of the first possible implementation, the feature extraction of the image of the object to be reconstructed to obtain the feature vector includes:

[0148] Segmenting the image based on the object to be reconstructed to obtain a local image;

[0149] Adjust the size of the partial image to a preset size;

[0150] The feature vector is obtained by extracting features from the resized local image using an encoder of a convolutional neural network.

[0151] It should be understood that in the embodiment of the present application, the processor 902 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0152] The memory 901 may include a read-only memory and a random access memory, and provide instructions and data to the processor 902. A part or all of the memory 901 may also include a non-volatile random access memory. For example, the memory 901 may also store information of a device category.

[0153] As can be seen from the above, through the embodiment of the present application, the image of the object to be reconstructed is first feature extracted to obtain a feature vector for characterizing the shape feature information of the object to be reconstructed, and then the feature vector is combined with a preset template for the object to be reconstructed to generate a feature map, and finally the feature map is input into the trained graph convolutional neural network to obtain the three-dimensional reconstruction result of the object to be reconstructed. In the above process, through the combination of the feature vector and the preset template, the feature map finally generated not only contains the shape features of the object to be reconstructed, but also obtains the three-dimensional structure information of the object to be reconstructed displayed by the preset template. As a result, the trained graph convolutional neural network can better process the feature map, and the obtained three-dimensional reconstruction result will be more accurate, ensuring the stability of the three-dimensional reconstruction.

[0154] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0155] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0156] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0157] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0158] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the associated hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The above-mentioned computer-readable storage medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer-readable memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media does not include electrical carrier signals and telecommunication signals.

[0160] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Extracting features from the image of the object to be reconstructed to obtain a feature vector, wherein the feature vector is used to characterize shape feature information of the object to be reconstructed, including contour features describing the boundary shape of the object to be reconstructed and regional features describing the internal shape of the object to be reconstructed; A feature map is generated according to the feature vector and a preset template for the object to be reconstructed. The preset template is used to characterize the three-dimensional structure information of the object to be reconstructed, which is the three-dimensional structure information of the object to be reconstructed in a specified posture. Different preset templates are set according to different types of objects to be reconstructed. When the object to be reconstructed is a human body, the preset template is a human body grid map. When the object to be reconstructed is a hand, the preset template is a hand grid map. The preset template is expressed in a graph structure and fused and spliced ​​with the feature vector to generate a feature map. The human body grid map is a standard template defined by the SMPL model. First, the human body grid map is represented as M smpl =(V,F), where V represents the vertex set of the human body mesh; F represents the triangular face set of the human body mesh, and three vertices constitute a triangular face; for the graph structure M of the general format smpl =(V,F) to convert and obtain the graph structure expression M in the preset format smpl =(V,A); where V still represents the vertex set; A represents the adjacency matrix of the human body mesh graph, which is used to indicate that the value of the elements in the adjacency matrix is ​​0 or 1. If the i-th vertex is connected to the j-th vertex, then (A)ij=1, otherwise (A)ij=0; The feature map is input into a trained graph convolutional neural network to obtain a three-dimensional reconstruction result of the object to be reconstructed.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The graph convolutional neural network includes N functional modules connected in series; The input of the first functional module is the input of the graph convolutional neural network, the output of the Nth functional module is the output of the graph convolutional neural network, and N is an integer greater than 2; Among them, the functional module includes a convolution unit, a normalization unit and an activation function unit.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The first functional module includes at least three designated structures, which are sequentially connected in series, and the designated structure includes the convolution unit, the normalization unit, and the activation function unit which are sequentially connected in series; In the at least three designated structures: the input of the first designated structure is the input of the first functional module, and the residual between the output of the last designated structure and the input of the first functional module is the output of the first functional module.

4. The three-dimensional reconstruction method according to claim 2, characterized in that: The i-th functional module includes at least two specified structures, which are sequentially connected in series. The specified structure includes the convolution unit, the normalization unit and the activation function unit which are sequentially connected in series. The i is an integer greater than 1 and less than N. In the at least two designated structures: the input of the first designated structure is the output of the i-1th functional module, and the residual between the output of the last designated structure and the output of the i-1th functional module is the output of the i-th functional module.

5. The three-dimensional reconstruction method according to claim 2, characterized in that: The Nth functional module includes the convolution unit, the normalization unit, the activation function unit and the convolution unit connected in series in sequence; In the Nth functional module: the input of the first convolution unit is the output of the N-1th functional module, and the residual between the output of the second convolution unit and the output of the N-1th functional module is the output of the Nth functional module.

6. The three-dimensional reconstruction method according to any one of claims 2 to 5, characterized in that: The convolution unit is a Chebyshev convolution unit.

7. The three-dimensional reconstruction method according to claim 1, characterized in that: During the training process of the graph convolutional neural network, a total loss of the graph convolutional neural network is calculated based on mesh loss, three-dimensional joint loss, surface normal loss and surface edge loss; Wherein, the mesh loss is used to describe the position difference between the real human body mesh and the predicted human body mesh; The three-dimensional joint loss is used to describe the position difference between the real three-dimensional joints of the human body and the predicted three-dimensional joints of the human body; The surface normal loss is used to describe the angle difference between the normal vector of the triangular face of the real human body mesh and the normal vector of the triangular face of the predicted human body mesh; The surface edge loss is used to describe the length difference between the side lengths of the triangular faces of the real human body mesh and the side lengths of the triangular faces of the predicted human body mesh.

8. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of extracting features from the image of the object to be reconstructed to obtain a feature vector includes: Segmenting the image based on the object to be reconstructed to obtain a local image; Adjusting the size of the partial image to a preset size; The feature vector is obtained by performing feature extraction on the local image after size adjustment using an encoder of a convolutional neural network.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images