Three-dimensional reconstruction method, device, equipment and storage medium

Abstract

The present application provides a three-dimensional reconstruction method, apparatus, device and storage medium, wherein the method includes: extracting features from a two-dimensional image to obtain a feature map, performing multi-perspective feature fusion and cross-perspective alignment on multiple feature maps to obtain an aligned feature coding sequence with global information, obtaining three-dimensional key points based on the aligned feature coding sequence, splicing the feature coding sequence, the three-dimensional key points and a preset three-dimensional model vertex set to obtain an input sequence, and obtaining a three-dimensional model of the target object at the target perspective based on the input sequence. Multi-perspective feature fusion can enable the generated feature coding sequence to have global context information, and cross-perspective alignment can ensure that the obtained feature coding sequence is strongly correlated with the target object, reduce the proportion of irrelevant noise, ensure the semantic consistency of feature coding at each perspective, and achieve accurate conversion of models at different perspectives, thereby improving the efficiency of three-dimensional modeling.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional model reconstruction, and in particular to a three-dimensional reconstruction method, apparatus, device and storage medium. Background Art

[0002] Human-computer interaction technology has been widely used in intelligent scenarios, and image-based three-dimensional human body reconstruction, as a key link in the perception of human-computer interaction behavior, has also become a research hotspot in recent years. The main purpose of this task is to use image data collected by visual perception devices to perform three-dimensional modeling of the human body, obtain a digital representation of the target subject, and then hand it over to downstream tasks for analysis and processing.

[0003] Currently, methods for image-based 3D human body modeling have emerged. However, due to the multi-solution nature of 3D postures, it is difficult for the model to achieve realistic reconstruction results. In addition, real scenes are complex and human postures are diverse. For example, the widespread phenomenon of portrait occlusion has also caused great trouble for 3D human body reconstruction methods.

[0004] Therefore, existing 3D human body modeling methods have problems with low accuracy and efficiency. Summary of the Invention

[0005] The purpose of this application is to provide a three-dimensional reconstruction method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problems of low accuracy and efficiency of the three-dimensional human body modeling methods in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a three-dimensional reconstruction method, the method comprising:

[0008] Acquire two-dimensional images of the target object under multiple viewing angles;

[0009] Inputting the two-dimensional images from the multiple perspectives into a pre-trained three-dimensional reconstruction network model, performing feature extraction on each of the two-dimensional images in the three-dimensional reconstruction network model to obtain a feature map corresponding to each of the two-dimensional images;

[0010] Performing multi-view feature fusion on each of the feature maps to obtain a feature coding sequence;

[0011] Performing cross-view alignment on the feature coding sequence to obtain an aligned feature coding sequence, where the aligned feature coding sequence is the feature coding sequence corresponding to the target view;

[0012] Obtaining three-dimensional key points under the target perspective according to the aligned feature coding sequence;

[0013] The feature coding sequence, the three-dimensional key points and the preset three-dimensional model vertex set are spliced to obtain an input sequence, and the three-dimensional model vertex set of the target object at the target perspective is obtained based on the input sequence. The three-dimensional model vertex set of the target object at the target perspective is output as the three-dimensional model of the target object at the target perspective.

[0014] Optionally, the 3D reconstruction network model includes: a multi-head attention mechanism module and a feature refinement module;

[0015] The multi-view feature fusion of each feature map to obtain a feature coding sequence includes:

[0016] The multi-head attention mechanism module models the contextual relationship information in each of the feature maps to obtain a coding sequence with contextual information;

[0017] The coding sequence with context information is input into the feature refinement module for standardization to obtain the feature coding sequence.

[0018] Optionally, performing cross-view alignment on the feature coding sequence to obtain an aligned feature coding sequence includes:

[0019] Align the key point information in the feature coding sequence to obtain the aligned feature coding sequence.

[0020] Optionally, the step of concatenating the feature coding sequence, the three-dimensional key points, and the preset vertex set to obtain an input sequence includes:

[0021] Splicing the three-dimensional key points and the preset vertex set to obtain an initial input sequence;

[0022] The characteristic coding sequence is concatenated with the initial input sequence to obtain the input sequence.

[0023] Optionally, obtaining a set of vertices of a three-dimensional model of the target object at the target perspective according to the input sequence includes:

[0024] Based on preset transit hyperparameters, upsampling processing is performed on the three-dimensional space coordinates in the input sequence to obtain a three-dimensional model vertex set of the target object at the target perspective.

[0025] Optionally, the method further includes:

[0026] Obtaining parameters of a camera corresponding to at least one viewing angle, and determining a transformation matrix between the cameras based on the parameters;

[0027] The three-dimensional model at the target viewing angle is converted into a three-dimensional model at a viewing angle corresponding to the conversion matrix according to the conversion matrix.

[0028] Optionally, the method further includes:

[0029] The initial model is trained based on the target loss function and preset samples to obtain the three-dimensional reconstruction network model, wherein the target loss function includes: a first sub-function, a second sub-function, a third sub-function and a fourth sub-function, and the first sub-function, the second sub-function, the third sub-function and the fourth sub-function are superimposed with corresponding weights to obtain the target loss function, the first sub-function is used to calculate the loss of key points, the second sub-function is used to calculate the loss of vertices of the three-dimensional model, the third sub-function is used to calculate the loss of cross-view alignment, and the fourth sub-function is used to calculate the loss of surface smoothness of the three-dimensional model.

[0030] In a second aspect, another embodiment of the present application provides a three-dimensional reconstruction device, the device comprising:

[0031] An acquisition module is used to: acquire two-dimensional images of a target object at multiple viewing angles;

[0032] a feature extraction module, configured to: input the two-dimensional images from the multiple perspectives into a pre-trained three-dimensional reconstruction network model, perform feature extraction on each of the two-dimensional images in the three-dimensional reconstruction network model, and obtain a feature map corresponding to each of the two-dimensional images;

[0033] A feature fusion module is used to: perform multi-view feature fusion on each of the feature maps to obtain a feature coding sequence;

[0034] A cross-view alignment module is used to: perform cross-view alignment on the feature coding sequence to obtain an aligned feature coding sequence, where the aligned feature coding sequence is the feature coding sequence corresponding to the target view;

[0035] A generating module, configured to obtain three-dimensional key points under the target viewing angle according to the aligned feature coding sequence;

[0036] An output module is used to: splice the feature coding sequence, the three-dimensional key points and a preset three-dimensional model vertex set to obtain an input sequence, and obtain a three-dimensional model vertex set of the target object at the target perspective based on the input sequence, and output the three-dimensional model vertex set of the target object at the target perspective as the three-dimensional model of the target object at the target perspective.

[0037] Optionally, the 3D reconstruction network model includes: a multi-head attention mechanism module and a feature refinement module;

[0038] The feature fusion module is further used for:

[0039] The multi-head attention mechanism module models the contextual relationship information in each of the feature maps to obtain a coding sequence with contextual information;

[0040] The coding sequence with context information is input into the feature refinement module for standardization to obtain the feature coding sequence.

[0041] Optionally, the cross-view alignment module is further configured to:

[0042] Align the key point information in the feature coding sequence to obtain the aligned feature coding sequence.

[0043] Optionally, the output module is further used to:

[0044] Splicing the three-dimensional key points and the preset vertex set to obtain an initial input sequence;

[0045] The characteristic coding sequence is concatenated with the initial input sequence to obtain the input sequence.

[0046] Optionally, the output module is further used to:

[0047] Based on preset transit hyperparameters, upsampling processing is performed on the three-dimensional space coordinates in the input sequence to obtain a three-dimensional model vertex set of the target object at the target perspective.

[0048] In the third aspect, another embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the above-mentioned three-dimensional reconstruction method.

[0049] In a fourth aspect, another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned three-dimensional reconstruction method are executed.

[0050] The beneficial effects of the present application are: by performing feature extraction and multi-perspective feature fusion on images from multiple perspectives, the generated feature coding sequence can be provided with global context information; by acquiring images from multiple perspectives, the accuracy of the generated three-dimensional model can be improved; and by cross-perspective alignment, the key point information of the target object from each perspective can be aligned, thereby ensuring that the obtained feature coding sequence is strongly correlated with the target object, reducing the proportion of irrelevant noise, ensuring the semantic consistency of the feature coding from each perspective, and realizing accurate conversion of models from different perspectives, thereby improving the efficiency of three-dimensional modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A schematic diagram showing an application scenario provided by an embodiment of the present application is shown;

[0053] Figure 2 A flowchart of a three-dimensional reconstruction method provided by an embodiment of the present application is shown;

[0054] Figure 3 A schematic diagram showing the operation of a three-dimensional reconstruction network model provided by an embodiment of the present application is shown;

[0055] Figure 4 A flow chart of obtaining a characteristic coding sequence provided by an embodiment of the present application is shown;

[0056] Figure 5 A flowchart of obtaining an input sequence provided by an embodiment of the present application is shown;

[0057] Figure 6 A flowchart of another three-dimensional reconstruction method provided by an embodiment of the present application is shown;

[0058] Figure 7 A schematic structural diagram of a three-dimensional reconstruction device provided in an embodiment of the present application is shown;

[0059] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0061] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0062] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0063] With the rapid development of computer vision in the field of artificial intelligence, how to use computer vision to achieve high-quality human-computer interaction has become a research hotspot. Image-based three-dimensional human body reconstruction technology uses image data collected by visual perception devices to perform three-dimensional modeling of the human body, obtain a digital representation of the target subject, and then hand it over to downstream tasks for analysis and processing.

[0064] Currently, methods for image-based 3D human body modeling have emerged. However, due to the multi-solution nature of 3D postures, it is difficult for the model to achieve realistic reconstruction results. In addition, real scenes are complex and human postures are diverse. For example, the widespread phenomenon of portrait occlusion has also caused great trouble for 3D human body reconstruction methods.

[0065] Therefore, how to stably and accurately reconstruct a three-dimensional human body model from images is one of the key issues in realizing highly interactive intelligent agents.

[0066] Based on the above problems, this application proposes a three-dimensional reconstruction method, which can be applied to a three-dimensional reconstruction network model in a three-dimensional human body reconstruction scenario. The executing subject can be, for example, an electronic device. By analyzing and modeling images from multiple perspectives, the multi-solution of two-dimensional images to three-dimensional modeling and the complexity of natural scenes on the model performance are reduced as much as possible, thereby improving the efficiency and accuracy of model modeling.

[0067] Figure 1 This is a schematic diagram of an application scenario given in this application, such as Figure 1 As shown, the user can input multiple two-dimensional images of the target object at different perspectives at the same time into the three-dimensional reconstruction network model. The three-dimensional reconstruction network model can analyze and reconstruct the multiple input images to obtain a three-dimensional model of the target object at a specified perspective.

[0068] Next, combine Figure 2 , the three-dimensional reconstruction method of this application is further explained, such as Figure 2 As shown, the method includes:

[0069] S201: Acquire two-dimensional images of a target object under multiple viewing angles.

[0070] Optionally, the target object can be a person who needs to be three-dimensionally modeled, or other living or non-living things. In the embodiment of the present application, three-dimensional modeling of the human body is used as an example, but this should not be used as a limitation. The three-dimensional reconstruction method of the present application can be applied to any object that needs to be three-dimensionally modeled.

[0071] Optionally, multiple cameras may be deployed around the target object to acquire two-dimensional images of the target object from multiple perspectives at the same time.

[0072] S202: Inputting two-dimensional images from multiple perspectives into a pre-trained three-dimensional reconstruction network model, performing feature extraction on each two-dimensional image in the three-dimensional reconstruction network model, and obtaining a feature map corresponding to each two-dimensional image.

[0073] Optionally, the pre-trained 3D reconstruction network model can be used to perform 3D modeling of target objects in the image based on multiple 2D images. Technicians can pre-train the model based on a large number of samples to improve the reliability of the model when used.

[0074] Optionally, the three-dimensional reconstruction network model can perform feature extraction on multiple input two-dimensional images. For example, a feature extraction network can be used to extract features from multiple input two-dimensional images to obtain feature maps that correspond one to one to the multiple two-dimensional images.

[0075] S203: Perform multi-view feature fusion on each feature map to obtain a feature coding sequence.

[0076] Optionally, the 3D reconstruction network model can perform multi-view feature fusion on each feature map to obtain a feature coding sequence.

[0077] Optionally, when the target object is a human body, the multi-view feature fusion can process the multiple feature maps obtained in the above step S202 to obtain a feature coding sequence containing global view information and human body semantic information.

[0078] Optionally, the feature coding sequence can represent the image features of the target object under multiple perspectives, and the feature coding sequence can include the features of the target object. Taking human body modeling as an example, the feature coding sequence can represent the image features under different perspectives, and the feature coding sequence can include human body posture and body shape information.

[0079] S204: performing cross-view alignment on the feature coding sequence to obtain an aligned feature coding sequence, where the aligned feature coding sequence is the feature coding sequence corresponding to the target view.

[0080] It should be noted that the feature coding sequence of the target object obtained in the above step S203 is only an integration of multi-view images, and lacks sufficient constraints to ensure that the focus of modeling is on the target object, rather than the background or other areas unrelated to the target object. Therefore, directly using the above feature coding sequence for modeling will result in a lot of noise, resulting in low modeling accuracy. Therefore, in this step, the intermediate task of cross-view alignment is used to reduce the proportion of irrelevant information in the modeling process.

[0081] Optionally, cross-view alignment can be to align the target objects at each view in the feature coding sequence so that the focus of the model at each view is on the target object, and obtain the aligned feature coding sequence based on the aligned feature maps at each view.

[0082] Optionally, the target viewing angle may be any viewing angle among multiple viewing angles determined by the user, and the aligned feature coding sequence may be an aligned feature coding sequence under any viewing angle specified by the user.

[0083] S205: Obtain three-dimensional key points under the target perspective based on the aligned feature coding sequence.

[0084] Optionally, the 3D reconstruction network model can obtain the 3D key points of the target object under the target perspective based on the aligned feature coding sequence under the target perspective.

[0085] Optionally, the three-dimensional key point may be a plurality of three-dimensional key points that characterize the posture of the target object.

[0086] Exemplarily, when the target object is a human body, the three-dimensional key points may be the three-dimensional coordinates of key points specified in a human skeleton point model, for example, key points describing human posture such as elbow joints and knee joints.

[0087] Optionally, the three-dimensional key points under the target perspective are obtained based on the aligned feature coding sequence. For example, the aligned feature coding sequence can be input into a fully connected network, and the sequence is hierarchically reduced in dimensionality until the three-dimensional coordinates of the key points are output.

[0088] S206: The feature coding sequence, the three-dimensional key points and the preset three-dimensional model vertex set are spliced to obtain an input sequence, and the three-dimensional model vertex set of the target object at the target perspective is obtained based on the input sequence. The three-dimensional model vertex set of the target object at the target perspective is output as the three-dimensional model of the target object at the target perspective.

[0089] Optionally, the preset three-dimensional model vertex set may be a preset vertex model of the target object. For example, when the target object is a human body, the preset three-dimensional model vertex set may be a 431 vertex model of the human body in a static state.

[0090] Optionally, the vertex set of the three-dimensional model of the target object at the target viewing angle may be, for example, a vertex set of the human body surface at a specified viewing angle.

[0091] Optionally, the feature coding sequence can be a feature coding sequence corresponding to the target perspective. The three-dimensional reconstruction network model can splice the feature coding sequence, three-dimensional key points and a preset three-dimensional model vertex set to obtain an input sequence, which may include the three-dimensional spatial position information of the target object. According to the input sequence, the three-dimensional model vertex set of the target object under the target perspective can be obtained. The three-dimensional reconstruction network model can display the three-dimensional model vertex set to obtain the three-dimensional model of the target object under the specified perspective.

[0092] It should be noted that the above steps S202 to S206 can all be performed in the three-dimensional reconstructed network model.

[0093] In an embodiment of the present application, a three-dimensional reconstruction network model can extract features from two-dimensional images of a target object at different perspectives at the same time to obtain a feature map, perform multi-perspective feature fusion and cross-perspective alignment on multiple feature maps to obtain a feature coding sequence aligned at the target perspective with global information, and obtain three-dimensional key points based on the aligned feature coding sequence. Finally, the feature coding sequence, the three-dimensional key points, and a preset set of three-dimensional model vertices are spliced to obtain an input sequence, and a three-dimensional model of the target object at the target perspective is obtained based on the input sequence. By performing feature extraction and multi-perspective feature fusion on images at multiple perspectives, the generated feature coding sequence can be provided with global context information. By obtaining images from multiple perspectives, the accuracy of the generated three-dimensional model can be improved. By cross-perspective alignment, the key point information of the target object at each perspective can be aligned, thereby ensuring that the obtained feature coding sequence is strongly correlated with the target object, reducing the proportion of irrelevant noise, ensuring the semantic consistency of the feature coding at each perspective, and achieving accurate conversion of models at different perspectives, thereby improving the efficiency of three-dimensional modeling.

[0094] like Figure 3 As shown, it is a schematic diagram of the operation of a three-dimensional reconstruction network model given in an embodiment of the present application, wherein the three-dimensional reconstruction network model 302 includes a multi-view feature encoder, a multi-view feature decoder and a conversion encoder. The multi-view feature encoder and the multi-view feature decoder in part 301 of the figure are used to perform multi-view feature fusion. The above-mentioned three-dimensional reconstruction network model may include: a multi-head attention mechanism module and a feature refinement module. The multi-head attention mechanism module and the feature refinement module can both exist in the multi-view feature encoder and the multi-view feature decoder.

[0095] Next, the steps of fusing the multi-view features of each feature map to obtain the feature coding sequence are described. Figure 4 As shown, the above step S203 includes:

[0096] S401: The multi-head attention mechanism module models the contextual relationship information in each feature map to obtain a coding sequence with contextual information.

[0097] Optionally, the multi-head attention mechanism module can be a conversion encoder and a conversion decoder including a multi-head attention mechanism, see Figure 3 , for example, it can be a multi-view feature encoder and a multi-view feature decoder including a multi-head attention mechanism.

[0098] The multi-view feature encoder can encode the input feature map to establish associations, and the corresponding multi-view feature decoder can output the associated feature maps as a coding sequence.

[0099] The following is an example of how the multi-head attention mechanism module models the contextual information in each feature map to obtain an encoding sequence.

[0100] For example, each feature map output by the feature extraction network can be input into the multi-head attention mechanism module to generate a coding sequence, and the multi-head attention mechanism is used in the multi-view feature encoder to model the contextual relationship of image features. It should be noted that since the original number of channels of the feature map is too high, it will cause a huge amount of computation, which will affect the training and operation efficiency of the model. Therefore, its dimension and shape can be adjusted through the fully connected layer first. This process can be expressed by the following formula (1):

[0101] X=Reshape(F) W X ={x1,x2,…,x L} (1)

[0102] Where F is the image feature encoding sequence before adjustment, and X is the image feature encoding sequence after dimension adjustment. It consists of L tokens, each of dimension d. Token, or x in the formula, is a configurable hyperparameter. This projection process not only effectively reduces the model's computational complexity but also enhances the model's expressive power during the encoding sequence transformation.

[0103] Next, the sequence X can be linearly transformed by multiplying it with a learnable parameter matrix to obtain the most important triples (Q, K, V) in the attention mechanism, which are the query sequence, key sequence, and value sequence, respectively. This triple can be used as the input of the conversion encoder or conversion decoder to complete the sequence transformation process. It should be noted that this application uses a multi-head attention mechanism to extract contextual relationships. Taking the query sequence Q in the triple as an example, its formal representation can be shown as follows (2):

[0104]

[0105] Where Q is the query sequence in triples, h is the number of subsets of the feature dimension, L is the number of tokens in equation (1), and d is the dimension of each token. According to equation (2), the multi-head attention mechanism actually splits the feature dimension in the traditional attention mechanism into h subsets. For each subset in the sequence, it is still obtained by linearly transforming the input sequence X. This process is similar to the method of obtaining sequence X from sequence F, and can also be calculated according to equation (3).

[0106]

[0107] This mapping relationship allows the model to expand from the feature space corresponding to X to the larger feature space corresponding to the triple, thereby improving the model's fitting ability. Furthermore, the multi-head attention mechanism's feature splitting operation enables the multi-view feature encoder and multi-view feature decoder to extract features from different subspaces of the input sequence, thereby achieving more accurate relationship modeling results.

[0108] After obtaining the triple (Q, K, V) as the formal input sequence, the multi-head attention mechanism can use point multiplication to perform the "query-match-value" operation on it, and then splice the calculation results of different subspaces together to finally output the feature encoding sequence Z with context information. This process can be expressed by the following formula (4).

[0109]

[0110] Where MultiHeadAtt(·) is the function label of multi-head attention, Softmax(·) is the probability normalization function commonly used in machine learning, is the key sequence, V i is a value sequence, Q i is the query sequence.

[0111] S402: Input the coding sequence with context information into the feature refinement module for standardization to obtain a feature coding sequence.

[0112] Optionally, after obtaining the coding sequence with context information, the 3D reconstruction network model can input the sequence into the feature refinement module to further refine the feature information in the coding sequence. The specific process can be shown in the following formula (5).

[0113]

[0114] LayerNorm(·) can normalize the feature dimensions within the sequence, ensuring that the model output value remains within the region of large gradient changes in the activation function. Dropout(·) can intentionally disable certain neurons at a certain rate during the training of the 3D reconstruction network model, thereby enhancing the model's robustness and achieving more stable and excellent performance during testing.

[0115] It is worth noting that the above multi-head attention mechanism module and feature refinement module can exist in Figure 3 In the multi-view feature encoder and the multi-view feature decoder, in the multi-view feature encoder, the sequence X can be obtained by encoding and transforming the image feature, and in the multi-view feature decoder, the sequence X can be a manually set learnable parameter.

[0116] In an embodiment of the present application, a multi-head attention mechanism is used to model contextual information on feature maps from multiple perspectives, and the resulting coding sequence with contextual information is standardized so that the resulting feature coding sequence has contextual information, thereby improving the accuracy of subsequent modeling based on the sequence.

[0117] Continue to refer to Figure 4 After the multi-view feature fusion, the multi-view feature decoder can also include a conversion encoder for performing three-dimensional modeling according to the input sequence, the preset vertex set and the three-dimensional key points to obtain the three-dimensional model of the target object under the target perspective.

[0118] In the above step S204, the feature coding sequence is aligned across viewing angles to obtain an aligned feature coding sequence, including:

[0119] Align the key point information in the feature coding sequence to obtain the aligned feature coding sequence.

[0120] Optionally, the three-dimensional reconstruction network model can obtain key point information therein according to the feature coding sequence. The key point information can be, for example, a skeleton point of the human body or a preset key point.

[0121] It should be noted that the feature coding sequence includes key point information under multiple perspectives. However, since the position of the target object in the feature map under multiple perspectives may not be the same, the key point information of each feature map can be aligned through cross-perspective alignment. It can be understood as aligning the parts that the model focuses on in the feature map under each perspective to the target object, thereby obtaining the aligned feature map. Based on the aligned feature map, the model can obtain the aligned coding sequence.

[0122] It is worth noting that the aligned feature coding sequence here can be a feature coding sequence at any specified viewing angle.

[0123] Next, we will describe the steps of combining the feature coding sequence, 3D key points and preset vertex sets to obtain the input sequence. Figure 5 As shown, the above step S206 includes:

[0124] S501: Concatenate the three-dimensional key points and the preset vertex set to obtain an initial input sequence.

[0125] Optionally, the preset vertex set can be, for example, a vertex model of a human body 431 in a silent state, and the three-dimensional key points can be the three-dimensional key points under the target perspective obtained in the above-mentioned S205 step. The three-dimensional key points and the preset vertex set can be spliced by connecting the coordinates of the three-dimensional key points with the vertex coordinates of the vertex model in the quantity dimension as the basic template of the input sequence, that is, the initial input sequence.

[0126] S502: Concatenate the feature coding sequence with the initial input sequence to obtain an input sequence.

[0127] Optionally, the feature coding sequence may be a feature coding sequence under the aligned target perspective, and the feature coding sequence and the initial input sequence may be spliced together, which may be an input sequence obtained by splicing the initial input sequence obtained in the above step S501 and the feature coding sequence in the dimension of the number of channels.

[0128] Exemplarily, the above step of constructing the input sequence can be expressed by the following formula (6):

[0129]

[0130] in That is, the three-dimensional key points and the vertex coordinates in the preset vertex set, is the feature coding sequence after linear transformation, the feature coding sequence The length of may not be consistent with the number of vertices on the human body surface, so it can be linearly transformed to align the length of the feature coding sequence with the number of vertices on the human body surface.

[0131] In an embodiment of the present application, by splicing three-dimensional key points with a preset vertex set, and then splicing the initial input sequence and feature coding sequence obtained after splicing, the posture information of the target object can be enhanced and the accuracy of subsequent modeling can be improved.

[0132] The following is a description of the steps for obtaining a set of vertices of a three-dimensional model of a target object at a target perspective according to the input sequence. The above step S206 includes:

[0133] Based on the preset transit hyperparameters, the 3D spatial coordinates in the input sequence are upsampled to obtain the 3D model vertex set of the target object under the target perspective.

[0134] Optionally, the vertex coordinates of the target object under the target perspective can be obtained from the input sequence. It should be noted that the input sequence represents only a subset of the surface vertices of the target object. By upsampling the input sequence, the full set of surface vertices of the target object can be obtained, thereby obtaining a three-dimensional model of the target object under the target perspective.

[0135] Optionally, after obtaining an input sequence with 3D spatial position information, the model can continuously refine the geometric input sequence through a multi-layer Transformer Encoder containing a fully connected network to obtain the 3D spatial coordinates of a subset of the target object's surface vertices. This is then restored to the complete set of surface vertices through upsampling, resulting in a 3D human body model from a specified primary viewpoint as the prediction result.

[0136] Optionally, preset transition hyperparameters can smoothly transition the predicted subset of target object surface vertices to the entire vertex set during the upsampling process, thereby improving the quality of the generated model.

[0137] In the embodiment of the present application, after obtaining the three-dimensional model at the target perspective, the three-dimensional model at any perspective can be obtained according to the three-dimensional model at the target perspective, such as Figure 6 As shown, the three-dimensional reconstruction method of the present application also includes:

[0138] S601: Obtain parameters of a camera corresponding to at least one viewing angle, and determine a conversion matrix between the cameras according to the parameters.

[0139] Optionally, before acquiring the multi-view input images in the above step S201 , the camera may be calibrated first to acquire the camera parameters of each view and determine the transformation matrix between each camera and other cameras.

[0140] S602: Converting the three-dimensional model at the target viewing angle into a three-dimensional model at a viewing angle corresponding to the conversion matrix according to the conversion matrix.

[0141] Optionally, the 3D reconstruction network model can convert the 3D model at the target perspective into a 3D model at a perspective corresponding to the transformation matrix according to the transformation matrix between the cameras.

[0142] After obtaining the camera parameters, in the embodiment of the present application, the 3D reconstruction network model can also predict the camera intrinsic parameters under each viewing angle, thereby realizing the conversion of 3D key points to 2D key points. The conversion process can be shown in the following formula (7).

[0143]

[0144] Where Π represents the projection mapping from 3D space to 2D plane. In this paper, the weak perspective projection model is used; K i It represents the intrinsic parameter of the camera corresponding to the i-th perspective; R i is the rotation matrix determined during camera calibration, It is a three-dimensional key point. It is worth noting that in some cases, the image will be cropped during data preprocessing in order to place the target object in the center of the image, which will cause the camera intrinsic parameters to become invalid. However, image preprocessing does not affect the correctness of the camera extrinsic parameters. Therefore, the embodiment of the present application can use camera extrinsic parameters to perform model conversion of target objects under different perspectives.

[0145] During the training phase of the 3D reconstruction network model, the 3D reconstruction method of the present application further includes:

[0146] The initial model is trained based on the target loss function and preset samples to obtain a three-dimensional reconstruction network model, wherein the target loss function includes: a first sub-function, a second sub-function, a third sub-function and a fourth sub-function. The first sub-function, the second sub-function, the third sub-function and the fourth sub-function are superimposed with corresponding weights to obtain the target loss function. The first sub-function is used to calculate the loss of key points, the second sub-function is used to calculate the loss of vertices of the three-dimensional model, the third sub-function is used to calculate the loss of cross-view alignment, and the fourth sub-function is used to calculate the loss of surface smoothness of the three-dimensional model.

[0147] Optionally, the preset samples may be, for example, labeled two-dimensional images at different viewing angles at different times.

[0148] For example, the objective loss function may be formally expressed as:

[0149]

[0150] in It can be the first sub-function mentioned above, which represents the deviation between the human key point position predicted by the model and the true value; It can be the second sub-function mentioned above, used to represent the deviation between the predicted three-dimensional human body surface vertex and its real position in space; It can be the third sub-function mentioned above, which is a numerical representation of the geometric constraints set for the model by the intermediate task of cross-view alignment; It can be the fourth sub-function mentioned above, which is a regularization term designed to improve the surface smoothness of the three-dimensional volume model of the human body output by the non-parametric method; the remaining coefficient quadruple (α, β, γ, μ) can be understood as the constraint strength of the loss values from the above four aspects on the model parameters.

[0151] Optionally, the first subfunction can be used to calculate the loss value between the key points predicted by the model and the true key points annotated by the sample. Exemplarily, the loss value can be, for example, the average first-order distance between the key points predicted by the model and the true key points annotated by the dataset.

[0152] Exemplarily, the calculation process of the first sub-function can be shown as follows (8):

[0153]

[0154] in are the predicted values of the model, and is the true value marked in the preset sample, and K is the number of key points. When calculating the key point loss function, because the sum of the first-order distances between the predicted key points and the true key points is proportional to the number of key points, we can first determine the number of key points and then calculate the mean as the final loss value.

[0155] Optionally, the second sub-function can be used to calculate the loss value between the vertex predicted by the calculation model and the actual vertex labeled by the sample.

[0156] Exemplarily, the calculation process of the second sub-function can be shown as follows (9):

[0157]

[0158] Among them, (m, M1, M2) represent the size of the vertex set of the three target object vertex model surfaces; and This corresponds to the vertex coordinates predicted by the model in the three sets; It is the true value of the three-dimensional vertex coordinate in the preset sample annotation.

[0159] Optionally, the third sub-function is used to calculate the loss of cross-view alignment and also constrain the feature encoding sequence with contextual information. The specific constraints can be divided into the following three aspects:

[0160] The directions of each sub-triangular facet of the 3D model surface generated during the 3D reconstruction network model training phase are constrained, and all predicted triangular faces are calculated to see if they meet expectations in terms of orientation. If not, the error value is included in the loss value to optimize the model parameters.

[0161] The sizes of all connecting edges between vertices on the surface of the 3D model generated during the 3D reconstruction network model training phase are constrained.

[0162] The consistency constraint constructed based on the Laplace loss calculates the deviation between the vertex and its true value, and optimizes the 3D reconstruction network model according to the error value between the vertex whose deviation exceeds the expected value and its true value.

[0163] Optionally, the fourth sub-function is used to calculate the loss of surface smoothness of the three-dimensional model. Exemplarily, the fourth sub-function may be a preset regularization term.

[0164] Based on the same inventive concept, the embodiments of the present application also provide a three-dimensional reconstruction device corresponding to the three-dimensional reconstruction method. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the above-mentioned three-dimensional reconstruction method in the embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0165] Reference Figure 7 FIG. 7 is a schematic diagram of a 3D reconstruction device provided in an embodiment of the present application, wherein the device includes: an acquisition module 701, a feature extraction module 702, a feature fusion module 703, a cross-view alignment module 704, a generation module 705, and an output module 706, wherein:

[0166] The acquisition module 701 is used to: acquire two-dimensional images of the target object under multiple viewing angles;

[0167] The feature extraction module 702 is configured to input two-dimensional images from multiple perspectives into a pre-trained three-dimensional reconstruction network model, extract features from each two-dimensional image in the three-dimensional reconstruction network model, and obtain a feature map corresponding to each two-dimensional image;

[0168] The feature fusion module 703 is used to perform multi-view feature fusion on each feature map to obtain a feature coding sequence;

[0169] The cross-view alignment module 704 is used to perform cross-view alignment on the feature coding sequence to obtain an aligned feature coding sequence, where the aligned feature coding sequence is the feature coding sequence corresponding to the target view;

[0170] The generating module 705 is used to obtain the three-dimensional key points under the target perspective according to the aligned feature code sequence;

[0171] Output module 706 is used to: splice the feature coding sequence, the three-dimensional key points and the preset three-dimensional model vertex set to obtain an input sequence, and obtain the three-dimensional model vertex set of the target object at the target perspective based on the input sequence, and output the three-dimensional model vertex set of the target object at the target perspective as the three-dimensional model of the target object at the target perspective.

[0172] Optionally, the 3D reconstruction network model includes: a multi-head attention mechanism module and a feature refinement module;

[0173] Optionally, the feature fusion module 703 is further configured to:

[0174] The multi-head attention mechanism module models the contextual relationship information in each feature map to obtain a coding sequence with contextual information;

[0175] The coding sequence with context information is input into the feature refinement module for standardization to obtain the feature coding sequence.

[0176] Optionally, the cross-view alignment module 704 is further configured to:

[0177] Align the key point information in the feature coding sequence to obtain the aligned feature coding sequence.

[0178] Optionally, the output module 706 is further configured to:

[0179] Concatenate the 3D key points and the preset vertex set to obtain the initial input sequence;

[0180] The feature encoding sequence is concatenated with the initial input sequence to obtain the input sequence.

[0181] Optionally, the output module 706 is further configured to:

[0182] Based on the preset transit hyperparameters, the 3D spatial coordinates in the input sequence are upsampled to obtain the 3D model vertex set of the target object under the target perspective.

[0183] Optionally, the 3D reconstruction apparatus may further include a conversion module, the conversion module being configured to:

[0184] Obtaining parameters of at least one camera corresponding to a viewing angle, and determining a transformation matrix between the cameras based on the parameters;

[0185] According to the transformation matrix, the three-dimensional model at the target perspective is transformed into a three-dimensional model at the perspective corresponding to the transformation matrix.

[0186] Optionally, the 3D reconstruction device may further include a training module, which is used to:

[0187] The initial model is trained based on the target loss function and preset samples to obtain a three-dimensional reconstruction network model, wherein the target loss function includes: a first sub-function, a second sub-function, a third sub-function and a fourth sub-function. The first sub-function, the second sub-function, the third sub-function and the fourth sub-function are superimposed with corresponding weights to obtain the target loss function. The first sub-function is used to calculate the loss of key points, the second sub-function is used to calculate the loss of vertices of the three-dimensional model, the third sub-function is used to calculate the loss of cross-view alignment, and the fourth sub-function is used to calculate the loss of surface smoothness of the three-dimensional model.

[0188] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0189] In an embodiment of the present application, a three-dimensional reconstruction network model can extract features from two-dimensional images of a target object at different perspectives at the same time to obtain a feature map, perform multi-perspective feature fusion and cross-perspective alignment on multiple feature maps to obtain a feature coding sequence aligned at the target perspective with global information, and obtain three-dimensional key points based on the aligned feature coding sequence. Finally, the feature coding sequence, the three-dimensional key points, and a preset set of three-dimensional model vertices are spliced to obtain an input sequence, and a three-dimensional model of the target object at the target perspective is obtained based on the input sequence. By performing feature extraction and multi-perspective feature fusion on images at multiple perspectives, the generated feature coding sequence can be provided with global context information. By obtaining images from multiple perspectives, the accuracy of the generated three-dimensional model can be improved. By cross-perspective alignment, the key point information of the target object at each perspective can be aligned, thereby ensuring that the obtained feature coding sequence is strongly correlated with the target object, reducing the proportion of irrelevant noise, ensuring the semantic consistency of the feature coding at each perspective, and achieving accurate conversion of models at different perspectives, thereby improving the efficiency of three-dimensional modeling.

[0190] The present application also provides an electronic device, such as Figure 8 As shown in FIG, a schematic diagram of the electronic device structure provided by an embodiment of the present application includes: a processor 81, a memory 82, and a bus. The memory 82 stores machine-readable instructions executable by the processor 81 (for example, Figure 7 The device includes an acquisition module 701, a feature extraction module 702, a feature fusion module 703, a cross-view alignment module 704, a generation module 705, and an output module 706 corresponding to the execution instructions, etc.). When the computer device is running, the processor 81 communicates with the memory 82 through a bus, and when the machine-readable instructions are executed by the processor 61, the above-mentioned three-dimensional reconstruction method is performed.

[0191] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned three-dimensional reconstruction method are executed.

[0192] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0193] In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0194] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquire two-dimensional images of the target object under multiple viewing angles; Inputting the two-dimensional images from the multiple perspectives into a pre-trained three-dimensional reconstruction network model, performing feature extraction on each of the two-dimensional images in the three-dimensional reconstruction network model to obtain a feature map corresponding to each of the two-dimensional images; Performing multi-view feature fusion on each of the feature maps to obtain a feature coding sequence; Performing cross-view alignment on the feature coding sequence to obtain an aligned feature coding sequence, where the aligned feature coding sequence is the feature coding sequence corresponding to the target view; Obtaining three-dimensional key points under the target perspective according to the aligned feature coding sequence; The feature coding sequence, the three-dimensional key points and the preset three-dimensional model vertex set are spliced to obtain an input sequence, and the three-dimensional model vertex set of the target object at the target perspective is obtained based on the input sequence. The three-dimensional model vertex set of the target object at the target perspective is output as the three-dimensional model of the target object at the target perspective.

2. The method according to claim 1, characterized in that The 3D reconstruction network model includes: a multi-head attention mechanism module and a feature refinement module; The multi-view feature fusion of each feature map to obtain a feature coding sequence includes: The multi-head attention mechanism module models the contextual relationship information in each of the feature maps to obtain a coding sequence with contextual information; The coding sequence with context information is input into the feature refinement module for standardization to obtain the feature coding sequence.

3. The method according to claim 1, characterized in that The step of performing cross-view alignment on the feature coding sequence to obtain an aligned feature coding sequence includes: Align the key point information in the feature coding sequence to obtain the aligned feature coding sequence.

4. The method according to claim 1, wherein The step of splicing the feature coding sequence, the three-dimensional key points, and the preset vertex set to obtain an input sequence includes: Splicing the three-dimensional key points and the preset vertex set to obtain an initial input sequence; The characteristic coding sequence is concatenated with the initial input sequence to obtain the input sequence.

5. The method according to claim 1, wherein Obtaining a set of vertices of a three-dimensional model of the target object at the target perspective according to the input sequence includes: Based on preset transit hyperparameters, upsampling processing is performed on the three-dimensional space coordinates in the input sequence to obtain a three-dimensional model vertex set of the target object at the target perspective.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining parameters of a camera corresponding to at least one viewing angle, and determining a transformation matrix between the cameras based on the parameters; The three-dimensional model at the target viewing angle is converted into a three-dimensional model at a viewing angle corresponding to the conversion matrix according to the conversion matrix.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The initial model is trained based on the target loss function and preset samples to obtain the three-dimensional reconstruction network model, wherein the target loss function includes: a first sub-function, a second sub-function, a third sub-function and a fourth sub-function, and the first sub-function, the second sub-function, the third sub-function and the fourth sub-function are superimposed with corresponding weights to obtain the target loss function, the first sub-function is used to calculate the loss of key points, the second sub-function is used to calculate the loss of vertices of the three-dimensional model, the third sub-function is used to calculate the loss of cross-view alignment, and the fourth sub-function is used to calculate the loss of surface smoothness of the three-dimensional model.

8. A three-dimensional reconstruction device, characterized in that: include: An acquisition module is used to: acquire two-dimensional images of a target object at multiple viewing angles; a feature extraction module, configured to: input the two-dimensional images from the multiple perspectives into a pre-trained three-dimensional reconstruction network model, perform feature extraction on each of the two-dimensional images in the three-dimensional reconstruction network model, and obtain a feature map corresponding to each of the two-dimensional images; A feature fusion module is used to: perform multi-view feature fusion on each of the feature maps to obtain a feature coding sequence; A cross-view alignment module is used to: perform cross-view alignment on the feature coding sequence to obtain an aligned feature coding sequence, where the aligned feature coding sequence is the feature coding sequence corresponding to the target view; A generating module, configured to obtain three-dimensional key points under the target viewing angle according to the aligned feature coding sequence; An output module is used to: splice the feature coding sequence, the three-dimensional key points and a preset three-dimensional model vertex set to obtain an input sequence, and obtain a three-dimensional model vertex set of the target object at the target perspective based on the input sequence, and output the three-dimensional model vertex set of the target object at the target perspective as the three-dimensional model of the target object at the target perspective.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the program instructions to perform the steps of the three-dimensional reconstruction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional reconstruction method according to any one of claims 1 to 7 are executed.

Images