Multi-view 3D reconstruction method, device, electronic device and readable storage medium

Abstract

The present application provides a multi-view 3D reconstruction method, device, electronic device and readable storage medium. After obtaining multiple groups of training samples each containing multiple training images collected from different viewpoints at the same time, each training image is fitted with a preset topological face model, and sample labels of each training image are obtained according to the fitting parameters and the topological face model. The constructed neural network model is trained using each group of training samples carrying sample labels until a 3D reconstruction model is obtained when the preset requirements are met. The training samples used in this solution are multiple training images at the same time, which can ensure the consistency of facial expressions, and can optimize the model in terms of facial shape and facial expressions. The sample labels of the training images are obtained based on the fitting topological face model, and the sample labels have standardized and accurate effects, thereby improving the training effect of the model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-view three-dimensional reconstruction method, device, electronic device and readable storage medium. Background Art

[0002] At present, three-dimensional reconstruction of human faces is required in many application scenarios. For example, in live broadcast application scenarios, three-dimensional reconstruction of human faces can be performed based on the host's image, which can be applied to control applications such as digital humans, face replacement, and AI three-dimensional animation effects.

[0003] In the prior art, when performing 3D reconstruction of a face, there is a solution that uses multiple face images collected at different times to constrain and optimize the reconstruction model. However, in this method, since facial expressions are constantly changing, although the model is optimized based on face images from different perspectives, it can only achieve optimization effects in terms of face shape, and the optimization effect on facial expressions is very poor. In addition, the labels of face images used for model training are based on manual annotation, which has the problem of inaccurate and non-standard label information. Summary of the invention

[0004] The purpose of the present application includes, for example, providing a multi-view three-dimensional reconstruction method, device, electronic device and readable storage medium, which can achieve accurate three-dimensional reconstruction from the aspects of face shape and facial expression.

[0005] The embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides a multi-view 3D reconstruction method, the method comprising:

[0007] Acquire multiple sets of training samples, each set of training samples includes multiple training images collected from different perspectives at the same time;

[0008] For each group of training samples, each of the training images is fitted with a preset topological face model, and a sample label of each of the training images is obtained according to the fitting parameters and the topological face model;

[0009] In an optional embodiment, the multiple training images included in each group of training samples are divided into first-view training images and second-view training images, and the neural network model includes a first branch model and a second branch model;

[0010] The step of training the constructed neural network model using each group of training samples carrying sample labels until a three-dimensional reconstruction model is obtained when preset requirements are met includes:

[0011] The first branch model is trained using the first-perspective training image carrying the sample label until the target first branch model is obtained when the preset requirements are met;

[0012] Based on the target first branch model, the second branch model is trained according to the first perspective training image and the second perspective training image until the target second branch model is obtained when the preset requirements are met.

[0013] In an optional embodiment, the step of training the second branch model based on the target first branch model and according to the first perspective training image and the second perspective training image until the target second branch model is obtained when the preset requirements are met includes:

[0014] Importing the first perspective training image and the second perspective training image into the target first branch model and the second branch model respectively;

[0015] Obtaining intermediate feature information of the first-view training image using the target first branch model;

[0016] The second branch model is trained by combining the intermediate feature information of the first perspective training image and the second perspective training image until a target second branch model is obtained when preset requirements are met.

[0017] In an optional embodiment, the step of training the constructed neural network model using each group of training samples carrying sample labels until a three-dimensional reconstruction model is obtained when preset requirements are met includes:

[0018] Importing each group of training samples into the constructed neural network model for processing, and outputting the output coefficients of each of the training samples;

[0019] Calculating a loss function value according to the output coefficient, the topological face model and the sample label;

[0020] After adjusting the model parameters of the neural network model, the training is continued until the loss function value meets the preset requirements, and the training is stopped to obtain a three-dimensional reconstruction model.

[0021] In an optional implementation, the step of calculating the loss function value according to the output coefficient, the topological face model and the sample label includes:

[0022] Constructing a prediction item based on the output coefficient and the topological face model, and taking the sample label carried by the training sample as a label item;

[0023] A loss function is constructed based on the prediction item and the label item, and the loss function value is calculated.

[0024] In an optional implementation, the topological face model includes a topological average face model, a topological face shape model and a topological face expression model, and the output coefficients include face shape coefficients, face expression coefficients and an affine transformation matrix;

[0025] The step of constructing a prediction item based on the output coefficient and the topological face model comprises:

[0026] Superimposing the topological average face model, the topological face shape model carrying the face shape coefficient, and the topological face expression model carrying the face expression coefficient to obtain a superimposed term;

[0027] The prediction term is constructed by multiplying the superposition term by the affine transformation matrix.

[0028] In an optional implementation, the step of fitting each of the training images to a preset topological face model includes:

[0029] Extract multiple core points of the topological face contained in the preset topological face model;

[0030] Extracting key points corresponding to each of the core points from the training faces contained in the training images;

[0031] Based on multiple key points and multiple core points, the training face and the topological face are fitted.

[0032] In an optional embodiment, the method further comprises:

[0033] Acquire an image to be reconstructed, wherein the image to be reconstructed includes a plurality of face images of a target object taken at the same time from different viewing angles;

[0034] A plurality of face images are imported into the three-dimensional reconstruction model for processing, and the three-dimensional reconstruction information of the face of the target object is output.

[0035] In an optional embodiment, the plurality of facial images include a first-view facial image and a second-view facial image, and the three-dimensional reconstruction model includes a first reconstruction model and a second reconstruction model;

[0036] The step of importing a plurality of face images into the three-dimensional reconstruction model for processing and outputting the three-dimensional reconstruction information of the face of the target object comprises:

[0037] Importing the first-view facial image and the second-view facial image into the first reconstruction model and the second reconstruction model respectively;

[0038] Using the first reconstruction model, obtaining first-perspective feature information corresponding to the first-perspective face image;

[0039] The second reconstruction model is used to output the three-dimensional reconstruction information of the face of the target object corresponding to the second-view facial image based on the first-view feature information and the second-view facial image.

[0040] In a second aspect, the present application provides a multi-view 3D reconstruction device, the device comprising:

[0041] An acquisition module is used to acquire multiple sets of training samples, each set of training samples includes multiple training images collected from different viewing angles at the same time;

[0042] A fitting module, for fitting each training image with a preset topological face model for each group of training samples, and obtaining a sample label of each training image according to the fitting parameters and the topological face model;

[0043] The training module is used to train the constructed neural network model using each group of training samples carrying sample labels until a three-dimensional reconstruction model is obtained when preset requirements are met.

[0044] In a third aspect, the present application provides an electronic device comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the aforementioned embodiments.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of the aforementioned embodiments are implemented.

[0046] The beneficial effects of the embodiments of the present application include, for example:

[0047] The present application provides a multi-view 3D reconstruction method, device, electronic device and readable storage medium. After obtaining multiple groups of training samples each containing multiple training images collected from different viewpoints at the same time, each training image is fitted with a preset topological face model, and sample labels of each training image are obtained according to the fitting parameters and the topological face model. The constructed neural network model is trained using each group of training samples carrying sample labels until a 3D reconstruction model is obtained when the preset requirements are met. The training samples used in this solution are multiple training images at the same time, which can ensure the consistency of facial expressions, and can optimize the model in terms of facial shape and facial expressions. The sample labels of the training images are obtained based on the fitting topological face model, and the sample labels have standardized and accurate effects, thereby improving the training effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of an application scenario of the multi-view 3D reconstruction method provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of a multi-view 3D reconstruction method provided in an embodiment of the present application;

[0051] Figure 3 for Figure 2 Flow chart of sub-steps included in step S120;

[0052] Figure 4 A schematic diagram of core point extraction in a topological face model provided in an embodiment of the present application;

[0053] Figure 5 for Figure 2 Flow chart of sub-steps included in step S130;

[0054] Figure 6 for Figure 5 Flow chart of sub-steps included in step S132;

[0055] Figure 7 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application;

[0056] Figure 8 for Figure 2 Another flowchart of the sub-steps included in step S130;

[0057] Fig. 9 for Figure 8 Flow chart of sub-steps included in step S134;

[0058] Fig.10 Another flow chart of the multi-view 3D reconstruction method provided in an embodiment of the present application;

[0059] Fig.11 for Fig.10 Flow chart of sub-steps included in step S220;

[0060] Fig.12 A schematic diagram of the structure of a three-dimensional reconstruction model provided in an embodiment of the present application;

[0061] Fig.13A schematic diagram of three-dimensional feature points after three-dimensional reconstruction information is restored according to an embodiment of the present application;

[0062] Fig.14 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0063] Fig.15 This is a functional module block diagram of the multi-view 3D reconstruction device provided in an embodiment of the present application.

[0064] Icons: 100 - electronic device; 110 - storage medium; 120 - processor; 130 - multi-view 3D reconstruction device; 131 - acquisition module; 132 - fitting module; 133 - training module; 140 - communication interface; 200 - image acquisition device. DETAILED DESCRIPTION

[0065] In many application scenarios, such as live broadcasting, there are application control requirements such as expression drive, virtual digital human, virtual 3D special effects, avater, etc. In these application scenarios, there are high requirements for the authenticity and accuracy of control. Therefore, it is very important to reconstruct a high-precision 3D face model while ensuring a certain degree of real-time performance.

[0066] In the implementation of 3D reconstruction, the acquisition equipment used may include, for example, a depth camera, an ordinary camera, etc. Depth information can be obtained based on the depth camera, and good reconstruction effects can be obtained based on point cloud reconstruction. However, this solution has high requirements for equipment, the cost of the depth camera is high, and the color camera of the depth camera generally has low accuracy, which is not conducive to texture reconstruction. Therefore, compared with the solution based on the depth camera, the solution based on the ordinary camera has the advantages of being easier to implement.

[0067] Among them, ordinary cameras include high-precision SLR cameras and ordinary cameras. Compared with high-precision SLR cameras, ordinary cameras are more convenient to implement and can meet certain business scenario requirements, such as high-precision expression driving, high-precision face texture acquisition, etc.

[0068] In the existing technology, the 3D reconstruction of human faces based on ordinary cameras is mainly to optimize and reconstruct the model through multiple collected images. In this solution, multiple images are generally used to optimize the shape of the face, that is, to optimize the appearance to a certain extent. However, since multiple images may be collected at different times, the facial expressions and lighting are not consistent, the effect of facial expressions during reconstruction is very poor, and the accuracy is greatly affected.

[0069] Based on the above research findings, an embodiment of the present application provides a multi-view three-dimensional reconstruction method, which can use multiple groups of training samples that each contain multiple training images collected from different viewpoints at the same time. The three-dimensional reconstruction model obtained by training can ensure the consistency of facial expressions and can be optimized in terms of facial shape and facial expressions. The sample labels of the training images are obtained based on the fitted topological face model, and the sample labels have standardized and accurate effects, thereby improving the training effect of the model.

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0071] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0072] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0073] In the description of the present application, it should be noted that the terms "first", "second", etc., if used, are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0074] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0075] See also Figure 1 , is a schematic diagram of an application scenario of the multi-view 3D reconstruction method provided in an embodiment of the present application, wherein the application scenario includes an electronic device 100 and multiple image acquisition devices 200 that are communicatively connected to the electronic device 100. The electronic device 100 may be a server or a terminal device. The multiple image acquisition devices 200 may include a device for acquiring two-dimensional images, such as a camera.

[0076] In this embodiment, each image acquisition device 200 can send the acquired image information or video information to the electronic device 100, and the electronic device 100 analyzes and processes the received image information or video information to obtain information that can be used to achieve three-dimensional reconstruction of the face.

[0077] Combination Figure 2 The embodiment of the present application also provides a multi-view 3D reconstruction method applicable to an electronic device 100 to achieve optimization of the 3D reconstruction model and 3D reconstruction of a human face. The electronic device 100 can be a server or a terminal device, such as a live broadcast server or a live broadcast terminal in a live broadcast application scenario. The method steps defined in the process related to the multi-view 3D reconstruction method can be implemented by the electronic device 100. Figure 2 The specific process shown is described in detail.

[0078] Step S110, obtaining multiple groups of training samples, each group of training samples includes multiple training images collected from different viewing angles at the same time.

[0079] Step S120: for each group of training samples, fit each of the training images with a preset topological face model, and obtain a sample label of each of the training images according to the fitting parameters and the topological face model.

[0080] Step S130, training the constructed neural network model using each group of training samples carrying sample labels until a three-dimensional reconstructed model is obtained when preset requirements are met.

[0081] In this embodiment, in the model training stage, training samples can be first collected. Multiple image acquisition devices 200 can be aimed at the target face at different viewing angles for acquisition. The so-called target face can be any face used as a model, and this embodiment does not limit it. In order to enrich the diversity of samples, multiple different target faces and images of the target face in different expression states can be collected as samples.

[0082] Multiple image acquisition devices 200 are used to acquire multiple training images at different viewing angles at the same time, and the multiple training images can be used as a group of training samples. There is no strict requirement on the viewing angle of the multiple training images, but there is a strict requirement on the time, that is, the multiple training images need to be acquired at the same time.

[0083] For the multiple training images in each group of training samples, since digital humans, AI models, etc. are generally driven from an intermediate perspective in applications, the multiple training images generally include training images captured from an intermediate perspective. In addition, training images captured from left perspectives, right perspectives, and other perspectives are generally included. In this way, more comprehensive three-dimensional information can be obtained based on training images from multiple different perspectives, and three-dimensional reconstruction information can be accurately constructed. In addition, it is ensured that three-dimensional reconstruction information from an intermediate perspective can be obtained to facilitate subsequent driving applications.

[0084] The image acquisition device 200 can only acquire training images. To train the model, the training images need to be labeled. In the existing method, samples are generally labeled manually, which is not standardized and accurate.

[0085] Based on this, in this embodiment, a topological face model (BFM) is used. The BFM topological face model is a model of three-dimensional expression of the face. This model is obtained by decoupling the three-dimensional face models of N people collected through PCA (Principal Component Analysis). The topological face model can be understood as a standard face model, and the face of any person can be adjusted to correspond to the topological face model.

[0086] Therefore, in this embodiment, any collected training image is fitted with a preset topological face model, and the standard topological face model is used as the standard, and the sample label of the training sample is obtained by fitting parameters and the topological face model. In other words, the sample label of the training image can be characterized based on the standardized topological face model fitting, so that the unified standardized topological face model is used as the criterion to ensure the standardization and accuracy of the sample label of the obtained training image.

[0087] In this embodiment, the sample labels of the training images may include the shape coefficients and expression coefficients of the faces in the training images, and the affine transformation matrix of the training images at corresponding viewing angles.

[0088] The constructed neural network model is trained using each group of training samples carrying sample labels to obtain a 3D reconstruction model. In this way, since the training samples used are multiple training images at the same time, the consistency of facial expressions can be guaranteed, and the model can be optimized in terms of facial shape and facial expressions. The sample labels of the training images are obtained based on the fitting topological face model, and the sample labels have the advantages of being standardized and accurate, thereby improving the training effect of the model.

[0089] See also Figure 3 In this embodiment, when fitting the training image and the topological face model, it can be achieved in the following ways:

[0090] Step S121, extracting multiple core points of the topological face included in the preset topological face model.

[0091] Step S122: extract key points corresponding to the core points from the training faces contained in the training images.

[0092] Step S123: fitting the training face and the topological face based on the multiple key points and the multiple core points.

[0093] A human face generally includes multiple feature points, and the features of a human face can be mainly represented by multiple key feature points. Therefore, in this embodiment, while ensuring the fitting accuracy, in order to reduce the amount of information processing, the fitting process can be performed by extracting some feature points on the human face.

[0094] Please refer to Figure 4 ,in, Figure 4 The face composed of dense points can represent the topological face in the topological face model, wherein the marked multiple points corresponding to the facial features and contours can be core points.

[0095] In this embodiment, the feature points on each face are labeled in the same indexing manner. Therefore, after extracting multiple core points on the topological face, the corresponding key points on the training face in the training image can be obtained according to the index labels of each core point.

[0096] Then, based on the core points on the topological face and the key points on the training face, the training face and the topological face are fitted.

[0097] In this embodiment, a method of fitting some feature points on the face is adopted to achieve the purpose of obtaining sample labels based on the coefficients of the topological face model.

[0098] In this embodiment, each group of training samples includes multiple training images, and the multiple training images can be divided into first-perspective training images and second-perspective training images. The first-perspective training images and the second-perspective training images can both include training images at multiple different perspectives, that is, the first perspective and the second perspective can include multiple perspectives, rather than being limited to one perspective. For example, the first-perspective training images can include training images at a left perspective and training images at a right perspective, and the second-perspective training images can be training images at a middle perspective.

[0099] Accordingly, the constructed neural network model may include a first branch model and a second branch model, wherein the first branch model may correspond to the first perspective training image, and the second branch model may correspond to the second perspective training image. When the first perspective training image includes a left perspective training image and a right perspective training image, the first branch model includes a left first branch model corresponding to the left perspective training image, and a right first branch model corresponding to the right perspective training image. If the second perspective training image is an intermediate perspective training image, the second branch model is a branch model corresponding to the intermediate perspective training image.

[0100] In face 3D reconstruction, it is generally necessary to ensure the accuracy of 3D reconstruction information under the middle view as much as possible, while face images from the side view can provide side face information to supplement features that cannot be learned from the front face. However, in the traditional method, face images from the side view and face images from the middle view are often imported into the model for learning and training. In this method, since the learning and training of the side view also starts from the initial state, the learning effect has not yet reached a good state, which is not very helpful for the feature compensation of the face image from the middle view, and it is easy to cause feature redundancy and affect the training efficiency.

[0101] Based on this, see Figure 5 In this embodiment, in the process of training the neural network model, it is achieved by the following methods:

[0102] Step S131, training the first branch model using the first-perspective training image carrying the sample label until the target first branch model is obtained when the preset requirements are met.

[0103] Step S132: Based on the target first branch model, the second branch model is trained according to the first perspective training image and the second perspective training image, until the target second branch model is obtained when the preset requirements are met.

[0104] In this embodiment, the second perspective training image and the second branch model can be set according to the needs. The second perspective can be the target perspective required for subsequent driving. For example, if the perspective required for subsequent driving is the middle perspective, the second perspective can be the middle perspective. If the perspective required for subsequent driving is the left perspective, the second perspective can be the left perspective. Often, when driving digital humans and AI models, the perspective generally used is the middle perspective. Therefore, in this embodiment, the second perspective training image can be an image of the middle perspective, and the first perspective training image can be an image of other side perspectives, such as left and right perspectives.

[0105] The first branch model can be trained using the first-perspective training images with sample labels. When the first branch model is trained to meet the preset requirements, the first branch model is initially fixed to obtain the target first branch model. The first branch model can be a lightweight network model of, but not limited to, MobilenetV3. The preset requirements can be training convergence, training iterations reaching a maximum set number, or training duration reaching a set maximum duration.

[0106] After obtaining a target first branch model that meets certain requirements, the second branch model is trained based on the target first branch model and in combination with the first-perspective training image and the second-perspective training image until the target second branch model meets the preset requirements.

[0107] Since the target first branch model has been pre-trained, the features extracted based on the target first branch model are more accurate. Combined with the more accurate features of the first-view training image to complete the feature information that cannot be learned from the second-view training image, the accuracy of the obtained target second branch model can be further improved.

[0108] In this embodiment, compared with the method of synchronously training the branch models corresponding to multiple perspective images from the initial state, although the synchronous training can eventually converge and the loss function is small, the above method can avoid the defects of feature redundancy and easy falling into local optimality in simultaneous re-training.

[0109] For details, see Figure 6 In this embodiment, when training the second branch model, it can be achieved by the following methods:

[0110] Step S1321: import the first perspective training image and the second perspective training image into the target first branch model and the second branch model respectively.

[0111] Step S1322: Use the target first branch model to obtain intermediate feature information of the first perspective training image.

[0112] Step S1323, training the second branch model in combination with the intermediate feature information of the first perspective training image and the second perspective training image until a target second branch model is obtained when preset requirements are met.

[0113] When training the first branch model, the first branch model is adjusted multiple times based on the sample labels of the first-view training image and the output of the first branch model to achieve the purpose of model optimization. The intermediate network layer of the first branch model can obtain the intermediate feature information of the first-view training image, and the intermediate feature information is then processed by the subsequent network layer of the first branch model to obtain the final output.

[0114] When training the second branch model, the required information of the first-perspective training image is the intermediate feature information of the first-perspective training image obtained by the intermediate network layer of the target first branch model.

[0115] The second-view training image is imported into the second branch model, and after being processed by the intermediate network layer of the second branch model, the intermediate feature information of the second-view training image can be obtained. The intermediate feature information of the first-view training image obtained by the target first branch model and the intermediate feature information of the second-view training image are combined, and then processed by the subsequent network layer of the second branch model to obtain the model output of the second branch model. Based on the model output of the second branch model and the sample labels of the second-view training image, the second branch model is trained multiple times, and when the preset requirements are met, the target second branch model is obtained.

[0116] In this embodiment, the second branch model is trained by combining the intermediate feature information of the first-view training image and the second-view training image, and the feature information under the first view can be used to complete the feature learning of the second view, so that the features learned by the trained second branch model are more comprehensive and more accurate. In addition, by first training the first branch model to a target first branch model that meets the preset requirements, and then using the intermediate feature information obtained by the target first branch model in the training of the second branch model, the redundancy of feature information can be avoided, the training can be prevented from falling into a local optimum, and the accuracy of model training can be further improved.

[0117] Please refer to Figure 7 In this embodiment, the first branch model includes the first branch model on the left and the first branch model on the right, and the second branch model is a model of the middle perspective. Each branch model may include multiple FC (fully connected) layers, wherein the dimension of each FC layer may be 240 dimensions, including 199 dimensions for regressing face shape coefficients, 29 dimensions for regressing expression coefficients, and 12 dimensions for regressing affine transformation matrices.

[0118] You can train alone first Figure 7 The left branch model and the right branch model in the figure, after the left branch model and the right branch model are trained to meet the preset requirements, the features obtained by the last FC layer of the left branch model and the right branch model are feature information with actual feature significance, which can help the middle perspective image to regress faster and better.

[0119] When training the branch model of the middle view, the left view training images are also imported into the left branch model, the right view training images are imported into the right branch model, and the middle view training images are imported into the middle branch model. 左2 and FC 右2 After the layer, the intermediate feature information is obtained, and the intermediate view training image is passed through FC 中2 After the layer, the intermediate feature information is obtained. 左2 and FC 右2 The intermediate feature information and FC of the layer中2 The intermediate feature information is combined and jointly processed by FC 中3 After the processing of the layers, the output of the intermediate branch model is obtained. The model parameters are adjusted using the sample labels of the intermediate view training images and the output of the intermediate branch model and trained multiple times until an intermediate branch model that meets the requirements is obtained.

[0120] See also Figure 8 In this embodiment, for any of the above branch models, when training is performed inside the branch model, the specific training process can be implemented in the following manner:

[0121] Step S133, importing each group of training samples into the constructed neural network model for processing, and outputting the output coefficients of each of the training samples.

[0122] Step S134, calculating a loss function value according to the output coefficient, the topological face model and the sample label.

[0123] Step S135, adjusting the model parameters of the neural network model and continuing the training until the loss function value meets the preset requirements, then stopping the training to obtain a three-dimensional reconstruction model.

[0124] In the training process of the model, the setting of the loss function also plays a very important role in the model training effect. In this embodiment, the sample labels of each group of training samples are obtained by fitting with the topological face model. After the neural network model performs feature extraction, feature abstraction and other processing on the imported training samples, the corresponding output coefficients can be output eventually. The model will be optimized by differentiating the real label from the model output. Similarly, the output coefficients of the model need to be combined with the topological face model to unify the standards between the real label and the output.

[0125] Therefore, please refer to Fig. 9 In this embodiment, the calculation of the loss function value can be achieved in the following way:

[0126] Step S1341, constructing a prediction item based on the output coefficient and the topological face model, and using the sample label carried by the training sample as a label item.

[0127] Step S1342: construct a loss function based on the prediction item and the label item, and calculate the loss function value.

[0128] In this embodiment, for each branch model, each branch model can process the imported training samples, so the training processing logic inside each branch model is the same. A loss function can be constructed based on the output coefficient of the model, the topological face model and the sample label of the training sample. The output coefficient and the topological face model are used as the prediction items of the model, and the sample label of the training sample is used as the label item. The loss function can characterize the difference between the prediction item and the label item.

[0129] The training process of the model is the process of minimizing the loss function. In multiple iterative trainings, the model parameters of the neural network model can be adjusted and trained in the direction of minimizing the loss function until the training is stopped when the preset requirements are met to obtain a three-dimensional reconstructed model.

[0130] In this embodiment, the topological face model can be decoupled into a topological average face model, a topological face shape model and a topological face expression model through PCA. Any three-dimensional face of a person can achieve the function of being like the target face by adjusting the face shape and face expression coefficients based on the topological average face model. The output coefficients of the neural network model may include face shape coefficients, face expression coefficients and affine transformation matrix.

[0131] In this embodiment, the topological average face model, the topological face shape model carrying the face shape coefficient, and the topological face expression model carrying the face expression coefficient are superimposed to obtain a superimposed term, and then the superimposed term is multiplied by the affine transformation matrix to construct a prediction term.

[0132] Optionally, the constructed loss function can be as follows:

[0133]

[0134] Among them, predict represents the prediction item and label represents the label item.

[0135] The specific composition of the prediction items can be shown as follows:

[0136] {(bfm mean +shape predict *shape std +exp predict *exp std )*affine matrix} pts_index

[0137] Among them, bfm mean Represents the topological average face model, shape std Represents the topological face shape model, exp std Represents the topological facial expression model, shape predict , exppredict Respectively represent the face shape coefficient and face expression coefficient output by the model, affine matrix Represents the affine transformation matrix. pts_index represents the core point index information extracted from the topological face model.

[0138] The specific structure of the tag item can be as follows:

[0139] mesh pts_index *affine matrix_label

[0140] Among them, affine matrix_label Represents the affine transformation matrix in the sample label of the training image, mesh pts_index Represents the index information of the key points in the training image. For the above multiple branch models, the mesh of each branch model pts_index are the same, the difference is affine matrix_label , because the representation of a face at different viewing angles is just a different projection matrix direction.

[0141] Each branch model in the neural network model is iteratively trained according to the above loss function to obtain a three-dimensional reconstruction model that meets the preset requirements.

[0142] The above process is the process of using training samples to train a three-dimensional reconstruction model. In this embodiment, multiple training images collected from different perspectives at the same time are used as a group of training samples to train a three-dimensional reconstruction model. This can ensure the consistency of facial expressions in the training images in a group of training samples, and can optimize the model in terms of facial shape and facial expression.

[0143] Moreover, the sample labels of the training images are obtained based on the fitted topological face model. The sample labels are standardized and accurate, which can improve the training effect of the model.

[0144] Furthermore, the adopted neural network model includes multiple branch models. By first training the branch model corresponding to the auxiliary perspective to meet certain requirements, and then combining the branch model that meets the requirements to obtain the intermediate feature information under the auxiliary perspective and the training image under the target perspective (such as the intermediate perspective), the branch model corresponding to the target perspective is trained. This can avoid the training from falling into the local optimum and improve the optimization effect of the branch model under the target perspective.

[0145] In this embodiment, each branch model does not share an encoder, but allows different branch networks to focus on learning training images from different perspectives, and then shares parameters at the end, so as to achieve better reconstruction effects.

[0146] The three-dimensional reconstruction model obtained above can be used to obtain three-dimensional reconstruction information of the face in actual application scenarios, so as to drive digital humans, AI models, etc. in live broadcast application scenarios based on the obtained three-dimensional reconstruction information.

[0147] See also Fig.10 In this embodiment, the acquisition of face 3D reconstruction information can be achieved by:

[0148] Step S210: obtaining an image to be reconstructed, wherein the image to be reconstructed includes a plurality of face images of a target object taken from different viewing angles at the same time.

[0149] Step S220: import the multiple face images into the three-dimensional reconstruction model for processing, and output the three-dimensional reconstruction information of the face of the target object.

[0150] In this embodiment, the image to be reconstructed may be, for example, a face image of a host obtained in a live broadcast scene. Similarly, the image to be reconstructed includes multiple face images captured from different perspectives at the same time. The image to be reconstructed is processed using the three-dimensional reconstruction model obtained above, and the obtained three-dimensional reconstruction information of the face may include, for example, face shape coefficients, face expression coefficients, and affine transformation matrices.

[0151] Based on the obtained 3D facial reconstruction information, the 3D facial feature points of the target object can be restored, and then a 3D facial model can be constructed based on the 3D facial feature points, and then the digital human to be driven can be driven based on the 3D facial model.

[0152] In this embodiment, the 3D reconstruction model is obtained by training the constructed neural network model, and also includes multiple branch models. The multiple face images included in the image to be reconstructed include the first-view face image and the second-view face image, and the 3D reconstruction model includes the first reconstruction model and the second reconstruction model. Fig.11 , the 3D reconstruction information of the face can be obtained by:

[0153] Step S221: import the first-view face image and the second-view face image into the first reconstruction model and the second reconstruction model respectively.

[0154] Step S222: using the first reconstruction model to obtain first-perspective feature information corresponding to the first-perspective facial image.

[0155] Step S223: using the second reconstruction model and based on the first viewing angle feature information and the second viewing angle facial image, outputting the three-dimensional facial reconstruction information of the target object corresponding to the second viewing angle facial image.

[0156] Take the second-view face image as an example of the target view, such as the face image in the middle view. Then the first-view face image can be an auxiliary view, such as the face image in the left view and the right view. The 3D reconstruction model is part of the neural network model. In detail, Fig.12 As shown in , the first reconstruction model in the three-dimensional reconstruction model may lack the last FC layer compared to the first branch model in the neural network model. The second reconstruction model is the same as the second branch model in the neural network model.

[0157] Because in actual 3D reconstruction, the 3D information of the image under the target perspective needs to be obtained in the end, and the feature information of the image under the auxiliary perspective is used as a supplement to the 3D information of the image under the target perspective. Therefore, the first reconstruction model does not need the final output result, but only needs to obtain the intermediate feature information.

[0158] During implementation, the first reconstruction model and the second reconstruction model may perform information processing synchronously. Fig.12 As shown in , after the first perspective face image is processed by the first two FC layers of the first reconstruction model, the first perspective feature information is obtained. After the second perspective face image is processed by the first two FC layers of the second reconstruction model, the corresponding second perspective feature information is obtained. The first perspective feature information and the second perspective feature information are processed by the last FC layer of the second reconstruction model together, and the 3D reconstruction information of the face corresponding to the second perspective face image is output.

[0159] In this embodiment, after restoring the three-dimensional points of the face using the obtained three-dimensional face reconstruction information, the resulting effect image can be as follows: Fig.13 as shown in .

[0160] See also Fig.14 , is a schematic diagram of an exemplary component of an electronic device 100 provided in an embodiment of the present application, and the electronic device 100 may be a live broadcast server or a live broadcast providing terminal in a live broadcast application scenario. The electronic device 100 may include a storage medium 110, a processor 120, a multi-view three-dimensional reconstruction device 130 and a communication interface 140. In this embodiment, the storage medium 110 and the processor 120 are both located in the electronic device 100 and the two are separately arranged. However, it should be understood that the storage medium 110 may also be independent of the electronic device 100 and can be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, it may be a cache and / or a general register.

[0161] The multi-view three-dimensional reconstruction device 130 can be understood as the above-mentioned electronic device 100, or the processor 120 of the electronic device 100, or can be understood as a software function module independent of the above-mentioned electronic device 100 or the processor 120 that implements the above-mentioned multi-view three-dimensional reconstruction method under the control of the electronic device 100.

[0162] like Fig.15 As shown, the multi-view 3D reconstruction device 130 may include an acquisition module 131, a fitting module 132 and a training module 133. The functions of each functional module of the multi-view 3D reconstruction device 130 are described in detail below.

[0163] An acquisition module 131 is used to acquire multiple sets of training samples, each set of training samples includes multiple training images collected from different viewing angles at the same time;

[0164] It can be understood that the acquisition module 131 can be used to execute the above step S110. The detailed implementation of the acquisition module 131 can refer to the above content related to step S110.

[0165] A fitting module 132 is used to fit each training image with a preset topological face model for each group of training samples, and obtain a sample label of each training image according to the fitting parameters and the topological face model;

[0166] It can be understood that the fitting module 132 can be used to execute the above step S120. The detailed implementation of the fitting module 132 can refer to the above content related to step S120.

[0167] The training module 133 is used to train the constructed neural network model using each group of training samples carrying sample labels until a three-dimensional reconstruction model is obtained when preset requirements are met.

[0168] It can be understood that the training module 133 can be used to execute the above step S130. The detailed implementation of the training module 133 can refer to the above content related to step S130.

[0169] In a possible implementation, the multiple training images included in each group of training samples are divided into first-view training images and second-view training images, and the neural network model includes a first branch model and a second branch model; the training module 133 can be specifically used for:

[0170] The first branch model is trained using the first-perspective training image carrying the sample label until the target first branch model is obtained when the preset requirements are met;

[0171] Based on the target first branch model, the second branch model is trained according to the first perspective training image and the second perspective training image until the target second branch model is obtained when the preset requirements are met.

[0172] In a possible implementation, the training module 133 may be used to train the target second branch model in the following manner:

[0173] Importing the first perspective training image and the second perspective training image into the target first branch model and the second branch model respectively;

[0174] Obtaining intermediate feature information of the first-view training image using the target first branch model;

[0175] The second branch model is trained by combining the intermediate feature information of the first perspective training image and the second perspective training image until a target second branch model is obtained when preset requirements are met.

[0176] In a possible implementation, the training module 133 may be specifically used for:

[0177] Importing each group of training samples into the constructed neural network model for processing, and outputting the output coefficients of each of the training samples;

[0178] Calculating a loss function value according to the output coefficient, the topological face model and the sample label;

[0179] After adjusting the model parameters of the neural network model, the training is continued until the loss function value meets the preset requirements, and the training is stopped to obtain a three-dimensional reconstruction model.

[0180] In a possible implementation, the training module 133 may be used to calculate the loss function value in the following manner:

[0181] Constructing a prediction item based on the output coefficient and the topological face model, and taking the sample label carried by the training sample as a label item;

[0182] A loss function is constructed based on the prediction item and the label item, and the loss function value is calculated.

[0183] In a possible implementation, the topological face model includes a topological average face model, a topological face shape model, and a topological face expression model, and the output coefficients include face shape coefficients, face expression coefficients, and an affine transformation matrix; the training module 133 can be used to construct prediction items in the following manner:

[0184] Superimposing the topological average face model, the topological face shape model carrying the face shape coefficient, and the topological face expression model carrying the face expression coefficient to obtain a superimposed term;

[0185] The prediction term is constructed by multiplying the superposition term by the affine transformation matrix.

[0186] In a possible implementation, the fitting module 132 may be specifically used for:

[0187] Extract multiple core points of the topological face contained in the preset topological face model;

[0188] Extracting key points corresponding to each of the core points from the training faces contained in the training images;

[0189] Based on multiple key points and multiple core points, the training face and the topological face are fitted.

[0190] In a possible implementation, the multi-view 3D reconstruction device 130 further includes a reconstruction module, which can be used to:

[0191] Acquire an image to be reconstructed, wherein the image to be reconstructed includes a plurality of face images of a target object taken at the same time from different viewing angles;

[0192] A plurality of face images are imported into the three-dimensional reconstruction model for processing, and the three-dimensional reconstruction information of the face of the target object is output.

[0193] In a possible implementation, the multiple face images include a first-view face image and a second-view face image, and the three-dimensional reconstruction model includes a first reconstruction model and a second reconstruction model; the reconstruction module can be specifically used for:

[0194] Importing the first-view facial image and the second-view facial image into the first reconstruction model and the second reconstruction model respectively;

[0195] Using the first reconstruction model, obtaining first-perspective feature information corresponding to the first-perspective face image;

[0196] The second reconstruction model is used to output the three-dimensional reconstruction information of the face of the target object corresponding to the second-view facial image based on the first-view feature information and the second-view facial image.

[0197] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0198] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the multi-view three-dimensional reconstruction method provided in the above embodiment is implemented.

[0199] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium is executed, the above-mentioned multi-view 3D reconstruction method can be executed. Regarding the process involved when the computer-readable storage medium and its executable instructions are executed, reference can be made to the relevant description in the above-mentioned method embodiment, which will not be described in detail here.

[0200] In summary, the multi-view 3D reconstruction method, device, electronic device 100 and readable storage medium provided in the embodiment of the present application, after obtaining multiple groups of training samples each containing multiple training images collected from different perspectives at the same time, fit each training image with a preset topological face model, obtain sample labels of each training image according to the fitting parameters and the topological face model, and train the constructed neural network model using each group of training samples carrying sample labels until a 3D reconstruction model is obtained when the preset requirements are met. The training samples used in this solution are multiple training images at the same time, which can ensure the consistency of facial expressions, optimize the model in terms of facial shape and facial expressions, and the sample labels of the training images are obtained based on the fitting topological face model, and the sample labels have standardized and accurate effects, thereby improving the training effect of the model.

[0201] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A multi-view 3D reconstruction method, characterized in that: The method comprises: Acquire multiple groups of training samples, each group of training samples includes multiple training images collected from different perspectives at the same time, and the multiple training images are divided into first-perspective training images and second-perspective training images; For each group of training samples, each of the training images is fitted with a preset topological face model, and a sample label of each of the training images is obtained according to the fitting parameters and the topological face model; The constructed neural network model is trained using each group of training samples carrying sample labels until a 3D reconstruction model is obtained when the preset requirements are met, including: The first branch model included in the constructed neural network model is trained using the first-perspective training image carrying the sample label until the target first branch model is obtained when the preset requirements are met; based on the target first branch model, and according to the first-perspective training image and the second-perspective training image, the second branch model included in the neural network model is trained until the preset requirements are met and the target second branch model is obtained.

2. The multi-view 3D reconstruction method according to claim 1, characterized in that: The step of training the second branch model included in the neural network model based on the target first branch model and according to the first perspective training image and the second perspective training image until the target second branch model is obtained when the preset requirements are met includes: Importing the first perspective training image and the second perspective training image into the target first branch model and the second branch model respectively; Obtaining intermediate feature information of the first-view training image using the target first branch model; The second branch model is trained by combining the intermediate feature information of the first perspective training image and the second perspective training image until a target second branch model is obtained when preset requirements are met.

3. The multi-view 3D reconstruction method according to claim 1, characterized in that: The step of training the constructed neural network model using each group of training samples carrying sample labels until a three-dimensional reconstruction model is obtained when preset requirements are met includes: Importing each group of training samples into the constructed neural network model for processing, and outputting the output coefficients of each of the training samples; Calculating a loss function value according to the output coefficient, the topological face model and the sample label; After adjusting the model parameters of the neural network model, the training is continued until the loss function value meets the preset requirements, and the training is stopped to obtain a three-dimensional reconstruction model.

4. The multi-view 3D reconstruction method according to claim 3, characterized in that: The step of calculating the loss function value according to the output coefficient, the topological face model and the sample label comprises: Constructing a prediction item based on the output coefficient and the topological face model, and taking the sample label carried by the training sample as a label item; A loss function is constructed based on the prediction item and the label item, and the loss function value is calculated.

5. The multi-view 3D reconstruction method according to claim 4, characterized in that: The topological face model includes a topological average face model, a topological face shape model and a topological face expression model, and the output coefficients include face shape coefficients, face expression coefficients and an affine transformation matrix; The step of constructing a prediction item based on the output coefficient and the topological face model comprises: Superimposing the topological average face model, the topological face shape model carrying the face shape coefficient, and the topological face expression model carrying the face expression coefficient to obtain a superimposed term; The prediction term is constructed by multiplying the superposition term by the affine transformation matrix.

6. The multi-view 3D reconstruction method according to claim 1, characterized in that: The step of fitting each of the training images with a preset topological face model comprises: Extract multiple core points of the topological face contained in the preset topological face model; Extracting key points corresponding to each of the core points from the training faces contained in the training images; Based on multiple key points and multiple core points, the training face and the topological face are fitted.

7. The multi-view 3D reconstruction method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire an image to be reconstructed, wherein the image to be reconstructed includes a plurality of face images of a target object taken at the same time from different viewing angles; A plurality of face images are imported into the three-dimensional reconstruction model for processing, and the three-dimensional reconstruction information of the face of the target object is output.

8. The multi-view 3D reconstruction method according to claim 7, characterized in that: The multiple face images include a first-view face image and a second-view face image, and the three-dimensional reconstruction model includes a first reconstruction model and a second reconstruction model; The step of importing a plurality of face images into the three-dimensional reconstruction model for processing and outputting the three-dimensional reconstruction information of the face of the target object comprises: Importing the first-view facial image and the second-view facial image into the first reconstruction model and the second reconstruction model respectively; Using the first reconstruction model, obtaining first-perspective feature information corresponding to the first-perspective face image; The second reconstruction model is used to output the three-dimensional reconstruction information of the face of the target object corresponding to the second-view facial image based on the first-view feature information and the second-view facial image.

9. A multi-view 3D reconstruction device, characterized in that: The device is used to implement the multi-view 3D reconstruction method according to any one of claims 1 to 8, comprising: An acquisition module is used to acquire multiple sets of training samples, each set of training samples includes multiple training images collected from different viewing angles at the same time; A fitting module, for fitting each training image with a preset topological face model for each group of training samples, and obtaining a sample label of each training image according to the fitting parameters and the topological face model; The training module is used to train the constructed neural network model using each group of training samples carrying sample labels until a three-dimensional reconstruction model is obtained when preset requirements are met.

10. An electronic device, characterized in that: The electronic device comprises one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine executable instructions to perform the method steps described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of claims 1 to 8 are implemented.

Images