Model training method and device and storage medium

Abstract

The embodiment of the invention provides a model training method and device and a storage medium, and the method comprises the steps: obtaining high-quality three-dimensional face data and a two-dimensional face image for training, and obtaining a preliminary face construction model; and then, incremental learning is performed on the preliminary model by using data with lower quality and more quantity, so that the generalization ability of the model is further improved. In addition, a human-in-the-loop method is introduced, model output is fed back through artist correction or user selection, and continuous optimization of the model is achieved. According to the method and the related equipment provided by the invention, the requirement of a three-dimensional face construction model for high-quality training data can be relieved, and the robustness and generalization of the model are improved on the basis of limited high-quality training data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a model training method, apparatus, and storage medium. Background Art

[0002] With the continuous progress of computer vision technology, three-dimensional (3D) face image construction technology has been widely applied in multiple fields, including but not limited to security monitoring, virtual reality, game entertainment, etc. In these applications, accurately and efficiently generating 3D face images has become one of the key technologies. However, existing 3D face construction methods face many challenges.

[0003] For example, in the field of 3D face construction based on deep learning, the shortage and quality issues of training data have become key factors restricting the performance improvement of 3D face construction models. Traditional 3D face construction methods based on deep learning usually rely on limited high-quality training data, but such data is often difficult to obtain and costly. At the same time, due to various factors in the data acquisition process, such as lighting conditions, device accuracy, etc., the quality of training data is uneven, bringing great difficulties to model training.

[0004] To solve this problem, the industry has adopted a method of reconstructing geometry and texture maps from a single photo, which can reconstruct geometry and texture maps only from one facial image, but the richness and accuracy of the model results still need to be improved. Summary of the Invention

[0005] Embodiments of this application disclose a model training method, apparatus, and storage medium, which can be applied in fields such as artificial intelligence and computer vision to construct 3D faces, and the method and related devices can improve the richness and accuracy of model results.

[0006] In a first aspect, embodiments of this application provide a model training method, which includes:

[0007] Obtain a plurality of first training data, where the first training data includes first 3D face data and first 2D face images;

[0008] Train a neural network using the plurality of first training data to obtain a first face construction model. The first 3D face data in the first training data is label data, and the first 2D face images in the first training data are feature data. The first face construction model is used to generate corresponding 3D face data according to the input 2D face images;

[0009] Obtain a plurality of second training data, where the second training data includes second three-dimensional face data and second two-dimensional face images corresponding to the second three-dimensional face data, and the quality of the first training data is higher than that of the second training data;

[0010] Train the first face construction model according to the plurality of second training data to obtain a second face construction model. The second three-dimensional face data in the second training data is label data, and the second two-dimensional face images in the second training data are feature data. The second face construction model is used to generate corresponding three-dimensional face data according to the input two-dimensional face image.

[0011] In the above embodiment, there is a corresponding relationship between the first three-dimensional face data and the first two-dimensional face image. For example, the first two-dimensional face image is the front view of the first three-dimensional face data, and the first three-dimensional face data is the three-dimensional construction of the face image shown in the first two-dimensional face image. Similarly, there is also a corresponding relationship between the second three-dimensional face data and the second two-dimensional face image. The above quality indicates the authenticity of the two-dimensional face image or the three-dimensional face data. The higher the authenticity, the higher the quality.

[0012] The above method makes full use of training data of different qualities. First, the model is trained with high-quality first training data to ensure the initial accuracy and robustness of the model. Then, through gradient training, a large amount of second training data with relatively low quality is input into the model for re-training, so as to eliminate the problem that the accuracy of the model results generated by the model trained with less high-quality data and more relatively low-quality data is poor. This process combines the advantages of high-quality data and a large amount of medium- and low-quality data, improving the generalization ability and expression ability of the model. The three-dimensional face construction model obtained thereby can effectively improve the accuracy and stability of the three-dimensional face construction model on the basis of less high-quality training data.

[0013] In another optional embodiment of the first aspect, the training the first face construction model according to the plurality of second training data to obtain a second face construction model includes:

[0014] Input the plurality of second training data into the first face construction model for training by means of incremental learning to obtain the second face construction model.

[0015] The above-mentioned incremental learning method can make full use of new training data, continuously optimize the model, maximize the advantages of gradient training, and thus improve the performance of the model. At the same time, since incremental learning is carried out on the basis of an existing model, it can maintain the stability and consistency of the model and avoid the problem of catastrophic forgetting of the model under the influx of a large amount of training data.

[0016] In yet another alternative embodiment of the first aspect, the method further comprises:

[0017] Receiving a target two-dimensional face image;

[0018] Inputting the target two-dimensional face image into the second face construction model to generate target three-dimensional face data corresponding to the target two-dimensional face image;

[0019] Displaying the target three-dimensional face data;

[0020] Receiving a first model correction instruction for adjusting the appearance of the target three-dimensional face data to obtain corrected target three-dimensional face data;

[0021] Using the target two-dimensional face image and the corrected target three-dimensional face data as training data to train the second face construction model.

[0022] In the above embodiment, the target two-dimensional face image is the input data input into the second face construction model. After being processed by the model, the corresponding target three-dimensional face data is obtained. By updating and adjusting the target three-dimensional face data, new and better training data is obtained, thereby realizing the update of the second face construction model. Optionally, the first model correction instruction can be obtained after an artist adjusts the appearance of the target three-dimensional face data. By the artist's aesthetic perspective, the way of correcting the three-dimensional face data output by the model indicates the deficiencies of the model, so that the model can be trained specifically to further improve the expression ability of the model.

[0023] In yet another alternative embodiment of the first aspect, the method further comprises:

[0024] Receiving a target two-dimensional face image;

[0025] Inputting the target two-dimensional face image into the second face construction model to generate target three-dimensional face data corresponding to the target two-dimensional face image;

[0026] Displaying the target three-dimensional face data;

[0027] Receiving a second model correction instruction for evaluating the generation quality of the target three-dimensional face data, the second model correction instruction including the evaluation direction and / or evaluation level of the generation quality of the target three-dimensional face data;

[0028] Training the second face construction model based on the second model correction instruction.

[0029] Optionally, the generation quality includes multiple aspects, such as authenticity, matching degree with the target two-dimensional face image, integrity, etc. The generated target three-dimensional face data is evaluated through the second model correction instruction to obtain the direction of the optimization model, and the adjustment degree of the model is determined through the evaluation direction and / or evaluation level of the generation quality of the target three-dimensional face data. For example, the parameter weights can be re-adjusted through the evaluation level, so as to achieve precise adjustment of the model and improve the expression ability of the model.

[0030] In this embodiment, by introducing human subjective judgment and experience, the model output is corrected and optimized, thereby improving the accuracy and practicality of the model. At the same time, through continuous iteration and optimization, the performance of the model can be further improved.

[0031] In another optional embodiment of the first aspect, the three-dimensional face data in the first training data is obtained by a three-dimensional surround shooting method. Exemplarily, the above three-dimensional surround shooting method includes light cage scanning.

[0032] In this embodiment, the above-mentioned three-dimensional surround shooting method can obtain more accurate and comprehensive three-dimensional face data, providing high-quality data support for the training of the model.

[0033] In another optional embodiment of the first aspect, the two-dimensional face images in the second training data are obtained by at least one of network data and open source databases.

[0034] In another optional embodiment of the first aspect, the obtaining of the multiple first training data includes:

[0035] Obtaining multiple three-dimensional face data;

[0036] Rendering the multiple three-dimensional face data to obtain two-dimensional face images respectively corresponding to the multiple three-dimensional face data; the multiple three-dimensional face data and the two-dimensional face images respectively corresponding to the three-dimensional face data jointly form multiple first training data.

[0037] The data generation method in this embodiment can effectively expand the training data set and improve the generalization ability of the model.

[0038] In another optional embodiment of the first aspect, the obtaining of the multiple second training data includes:

[0039] Obtaining multiple of the second two-dimensional face images;

[0040] The multiple second two-dimensional facial images are three-dimensionally reconstructed to generate the multiple second three-dimensional facial data corresponding to the multiple two-dimensional facial images respectively. The multiple second two-dimensional facial images and the second three-dimensional facial data constitute the multiple second training data.

[0041] In another optional implementation of the first aspect, acquiring the plurality of second two-dimensional facial images includes:

[0042] Acquire multiple initial two-dimensional face images;

[0043] Inputting the multiple initial two-dimensional facial images into a preset data evaluation model to obtain probability values corresponding to the multiple initial two-dimensional facial images, wherein the probability values are used to represent the degree of match between the initial two-dimensional facial images and a two-dimensional facial image that meets a preset first condition, wherein a higher probability value indicates a higher degree of match; the preset first condition may include one or more of uniform illumination, image occlusion below a preset first threshold, and the image being a frontal face;

[0044] The multiple initial two-dimensional facial images whose probability values are higher than a preset second threshold are used as the multiple second two-dimensional facial images.

[0045] This embodiment filters initial two-dimensional facial images to obtain two-dimensional facial images that meet certain conditions as training data. For example, this filtering process is used to exclude data from the initial two-dimensional facial images that do not meet the requirements, such as two-dimensional facial images that appear in profile or have significant facial occlusion, to ensure data attribute consistency. Alternatively, it can be used to eliminate poor-quality data, improve the quality of the training data, and thus further enhance model performance.

[0046] Furthermore, the initial two-dimensional face images are screened through the data evaluation model to ensure the consistency of the training data and improve the stability of the model results.

[0047] In another optional implementation of the first aspect, the second face construction model includes an adversarial network model, which includes a face feature encoder, a generator and a discriminator. The face feature encoder is used to generate a face feature vector based on a two-dimensional image, the generator is used to generate three-dimensional face data based on the face feature vector and random noise, and the discriminator is used to judge the generated three-dimensional face data and the label data in the training data, and output a judgment result.

[0048] In yet another alternative implementation of the first aspect, the second training data includes a first data set and a second data set. The first data set includes two-dimensional face images and three-dimensional face data, and the second data set includes two-dimensional face images and three-dimensional face data. The quality of the first data set is higher than that of the second data set.

[0049] In this implementation, by splitting the second training data twice, data sets of different qualities are obtained, optimizing the gradient training mentioned above, and ensuring that the training data with different quality gradients can be used for model training in a preset order.

[0050] In yet another alternative implementation of the first aspect, the quality of the training data is determined based on a quality assessment model, and the quality assessment model is used to determine the difference between the training data and the real data.

[0051] The training data includes first training data and second training data. Optionally, the three-dimensional face data in the first training data and the second training data is the data whose quality is determined by the quality assessment model. The quality assessment model is trained based on historical training data and real data, as well as the difference between the training data and the real data. The difference can be expressed numerically in the model expression. The real data is three-dimensional face data composed of two-dimensional data obtained by a special surrounding shooting method. The common way to evaluate the quality of three-dimensional face data is mainly determined qualitatively, that is, by subjective judgment of human eyes. However, this implementation provides a quantitative method to accurately evaluate the training data, reducing labor costs.

[0052] This method can objectively evaluate the quality of the data and provide strong data support for model training.

[0053] In yet another alternative implementation of the first aspect, the second three-dimensional face data in the second training data has undergone retopology processing, and the topological structure of the second three-dimensional face data is the same as that of the first three-dimensional face data in the first training data.

[0054] The processing method provided in this implementation can eliminate the topological structure differences between different training data and improve the training effect of the model.

[0055] In a second aspect, an embodiment of the present application provides a model training device, and the device includes modules or units for executing the method according to any one of the first aspect.

[0056] In a possible design, the device includes:

[0057] A first acquisition module, configured to acquire a plurality of first training data, where the first training data includes first three-dimensional face data and first two-dimensional face images;

[0058] A first training module, configured to train a neural network using the plurality of first training data to obtain a first face construction model. The first three-dimensional face data in the first training data is label data, and the first two-dimensional face images in the first training data are feature data. The first face construction model is configured to generate corresponding three-dimensional face data according to the input two-dimensional face images;

[0059] A second acquisition module, configured to acquire a plurality of second training data, where the second training data includes second three-dimensional face data and second two-dimensional face images corresponding to the second three-dimensional face data, and the quality of the first training data is higher than that of the second training data;

[0060] A second training module, configured to train the first face construction model according to the plurality of second training data to obtain a second face construction model. The second three-dimensional face data in the second training data is label data, and the second two-dimensional face images in the second training data are feature data. The second face construction model is configured to generate corresponding three-dimensional face data according to the input two-dimensional face images.

[0061] In a possible implementation manner, the second training module is specifically configured to:

[0062] Input the plurality of second training data into the first face construction model for training in an incremental learning manner to obtain the second face construction model.

[0063] In a possible implementation manner, the apparatus further includes:

[0064] A first receiving unit, configured to receive a target two-dimensional face image;

[0065] A first input module, configured to input the target two-dimensional face image into the second face construction model to generate target three-dimensional face data corresponding to the target two-dimensional face image;

[0066] A first display module, configured to display the target three-dimensional face data;

[0067] A second receiving module, configured to receive a first model correction instruction, where the first model correction instruction is used to adjust the appearance of the target three-dimensional face data to obtain corrected target three-dimensional face data;

[0068] The third training module is used to train the second face construction model with the target two-dimensional face image and the corrected target three-dimensional face data as training data.

[0069] In a possible implementation, the device further includes:

[0070] A third receiving module, configured to receive a target two-dimensional face image;

[0071] A second input module, configured to input the target two-dimensional face image into the second face construction model to generate target three-dimensional face data corresponding to the target two-dimensional face image;

[0072] A second display module, configured to display the target three-dimensional face data;

[0073] A fourth receiving module, configured to receive a second model correction instruction, where the second model correction instruction is used to evaluate the generation quality of the target three-dimensional face data, and the second model correction instruction includes an evaluation direction and / or an evaluation level of the generation quality of the target three-dimensional face data;

[0074] Based on the second model correction instruction, train the second face construction model.

[0075] In a possible implementation, the three-dimensional face data in the first training data is obtained by a three-dimensional surround shooting method.

[0076] In a possible implementation, the two-dimensional face images in the second training data are obtained by at least one of network data and an open source database.

[0077] In a possible implementation, the first obtaining module is specifically configured to:

[0078] Obtain a plurality of first three-dimensional face data;

[0079] Render the plurality of first three-dimensional face data to obtain first two-dimensional face images respectively corresponding to the plurality of first three-dimensional face data; the plurality of first three-dimensional face data and the first two-dimensional face images form a plurality of first training data.

[0080] In a possible implementation, the second obtaining module is specifically configured to:

[0081] Obtain a plurality of the second two-dimensional face images;

[0082] Perform three-dimensional reconstruction on the plurality of second two-dimensional face images to generate the plurality of second three-dimensional face data respectively corresponding to the plurality of two-dimensional face images, and the plurality of second two-dimensional face images and the second three-dimensional face data form the plurality of second training data.

[0083] In a possible implementation, the second acquisition module is specifically configured to:

[0084] Acquire a plurality of initial two-dimensional face images;

[0085] Input the plurality of initial two-dimensional face images into a preset data evaluation model to obtain probability values corresponding to the plurality of initial two-dimensional face images respectively. The probability value is used to characterize the matching degree between the initial two-dimensional face image and the two-dimensional face image that meets the preset first condition. The higher the probability value, the higher the matching degree. The preset first condition includes one or more of uniform illumination, a screen occlusion lower than a preset first threshold, and a frontal face in the image;

[0086] Use the plurality of initial two-dimensional face images with probability values higher than a preset second threshold as the plurality of second two-dimensional face images.

[0087] In a possible implementation, the second face construction model includes an adversarial network model. The adversarial network model includes a face feature encoder, a generator, and a discriminator. The face feature encoder is used to generate a face feature vector according to a two-dimensional image. The generator is used to generate three-dimensional face data according to the face feature vector and random noise. The discriminator is used to judge the generated three-dimensional face data and the label data in the training data, and output a judgment result.

[0088] In a possible implementation, the second training data includes a first data set and a second data set. The first data set includes two-dimensional face images and three-dimensional face data. The second data set includes two-dimensional face images and three-dimensional face data. The quality of the first data set is higher than that of the second data set.

[0089] In a possible implementation, the quality of the training data is determined based on a quality evaluation model. The quality evaluation model is used to determine the difference size between the training data and the real data.

[0090] In a possible implementation, the second three-dimensional face data in the second training data has undergone retopology processing, and the topological structure of the second three-dimensional face data is the same as that of the first three-dimensional face data in the first training data.

[0091] In a possible implementation, the device further includes:

[0092] A third acquisition module, configured to acquire two-dimensional face images;

[0093] A third input module, configured to input the two-dimensional face image into the second face construction model to obtain three-dimensional face data corresponding to the two-dimensional face image.

[0094] Regarding the technical effects brought by the second aspect and any possible implementation manner, reference may be made to the introduction of the technical effects corresponding to the first aspect and the corresponding implementation manners.

[0095] Optionally, in the model training device described in the second aspect and any possible implementation manner of the second aspect:

[0096] In one implementation manner, the model training device is a module in a data acquisition device or a data acquisition module. Optionally, the data acquisition device may be a data acquisition device for acquiring first training data. Exemplarily, the data acquisition device is a device for performing optical cage scanning.

[0097] In another implementation manner, the model training device is a data processing device other than the data acquisition device, or a module in the data processing device. When the model training device is a data processing device other than the data acquisition device, the data processing device may be a device for caching data, training a model, and communicating with the data acquisition device.

[0098] In a third aspect, an embodiment of the present application provides a model training device, which at least includes a memory and a processor. The memory is used to store a computer program, and the processor is used to call the computer program to implement the method described in the first aspect or any possible implementation manner of the first aspect.

[0099] In a fourth aspect, an embodiment of the present application provides a computing device cluster, which is characterized by including at least one computing device, and each computing device includes a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method as described in the first aspect and any possible implementation manner.

[0100] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program (which may also be referred to as code or instructions); when the computer program runs on a computer, the method described in the first aspect and any possible implementation manner is implemented.

[0101] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program (which may also be referred to as code or instructions); when the computer program runs, the computer is caused to execute the method described in the first aspect and any possible implementation manner.

[0102] In a seventh aspect, an embodiment of the present application provides a system, which includes a model training device and a data acquisition device. The model training communication device is configured to execute the method according to the first aspect and any possible implementation manner thereof. The data acquisition device is configured to obtain training data / initial training data through its own functions. The model training device and the data acquisition device may be the same device or different devices separately provided. When the model training device and the data acquisition device are different devices separately provided, the model training device and the data acquisition device are communicatively connected.

[0103] In addition, in the process of executing the method according to the first aspect and any possible implementation manner thereof, the processes of sending information, data, and / or obtaining information, data, etc. in the above method can be understood as the process of the processor outputting information and / or the process of the processor receiving input information. When outputting information, the processor may output the information to a transceiver (or a communication interface, or a sending module) for transmission by the transceiver. After the information is output by the processor, other processing may be required before it reaches the transceiver. Similarly, when the processor receives input information, the transceiver (or a communication interface, or a sending module) receives the information and inputs it to the processor. Further, after the transceiver receives the information, the information may require other processing before it is input to the processor.

[0104] Based on the above principle, for example, obtaining training data mentioned in the foregoing method can be understood as the processor receiving input training data.

[0105] Optionally, for operations such as transmission, sending, and receiving involved by the processor, if there is no special description, or if it does not conflict with its actual role or internal logic in the relevant description, they can generally be understood as operations such as the processor outputting, receiving, and inputting.

[0106] Optionally, in the process of executing the method according to the first aspect and any possible implementation manner thereof, the above-mentioned processor may be a processor dedicated to executing these methods or a processor that executes these methods by executing computer instructions in a memory, such as a general-purpose processor. The above-mentioned memory may be a non-transitory memory, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or separately provided on different chips. The embodiments of the present application do not limit the type of the memory and the setting manner of the memory and the processor.

[0107] In a possible implementation manner, the above at least one memory is located outside the device.

[0108] In yet another possible implementation, at least one of the memories is located within the device.

[0109] In yet another possible implementation, a part of at least one of the memories is located within the device, and another part of the memories is located outside the device.

[0110] In this application, the processor and the memory may also be integrated into one device, that is, the processor and the memory may also be integrated together. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Figure 1 is a schematic structural diagram of a model training system provided by an embodiment of this application;

[0112] Figure 2 is a schematic flowchart of a model training method provided by an embodiment of this application;

[0113] Figure 3 is a schematic structural diagram of a generative adversarial network model provided by an embodiment of this application;

[0114] Figure 4 is a schematic structural diagram of a face construction model provided by an embodiment of this application;

[0115] Figure 5 is a schematic diagram of a model training process provided by an embodiment of this application;

[0116] Figure 6 is a schematic diagram of a human-in-the-loop processing method provided by an embodiment of this application;

[0117] Figure 7 is a schematic flowchart of a model usage method provided by an embodiment of this application;

[0118] Figure 8 is a schematic structural diagram of a model training device provided by an embodiment of this application;

[0119] Figure 9 is a schematic structural diagram of a computing device provided by an embodiment of this application;

[0120] Figure 10 is a schematic structural diagram of a computing device cluster provided by an embodiment of this application;

[0121] Figure 11 is a schematic structural diagram of a computing device cluster provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0122] The embodiments of this application will be described below with reference to the accompanying drawings in the embodiments of this application.

[0123] In the fields of artificial intelligence, computer vision, etc., 3D face construction is widely used. This 3D face construction refers to constructing the 3D geometry and texture mapping of a face in a certain way. This technology is widely applied in the fields of computer vision, computer graphics, human-computer interaction, virtual reality, etc. Exemplarily, 3D face construction can be used for face recognition. Generally, the face data obtained through face recognition is two-dimensional. Through the method of 3D face construction, the two-dimensional face data can be extended to three-dimensional face data, thereby achieving accurate face matching and identifying the identity of a person. Therefore, the related solutions of 3D face construction have been widely applied in fields such as payment and security.

[0124] However, for the 3D face construction method based on deep learning, a large amount of training data is required to obtain a model with high accuracy. And the two-dimensional face images used as training data for training the 3D face construction model have uneven quality. Most of the background parts of these two-dimensional face images are complex and diverse, and the face parts are easily interfered by various factors such as occlusion, angle, and expression. In related technologies, when obtaining face data, it is generally obtained through publicly available datasets. Since many datasets require payment for use, and for Asian face data, the publicly available datasets are limited, and due to data privacy reasons, it is difficult to obtain a large amount of high-quality data, resulting in a relatively large cost for obtaining face data.

[0125] To alleviate the model's dependence on high-quality face data, a robust texture-guided geometric detail recovery method has been proposed. This method is a method for reconstructing geometry and texture from a single photo, which only requires one facial image to reconstruct geometry and texture. This method first uses the Style-Based Generator Adversarial Network version 2 (StyleGAN.v2) to repair the occluded facial regions, generate complete textures, and construct an accurate multi-view dataset of the target object. To estimate the detailed geometric structure, an implicit signed distance function is defined, and a physically based implicit renderer is used to reconstruct fine geometric details from the generated multi-view images. Then, based on the self-supervised decomposition of the diffuse normal and specular normal, detailed facial geometry as well as approximate diffuse albedo, diffuse shadow, and specular reflection shadow are obtained. However, this method uses the fitting result of the 3D Morphable Model (3DMM) as a self-supervised label, which leads to the reconstruction result depending on the fitting result. On the other hand, the model construction method based on 3DMM can generally ensure robustness, but the richness of the result is still limited by the parametric model and the quantity of high-quality data.

[0126] In view of the fact that there are still many limitations in the selection of training data for the current three-dimensional face construction model, the embodiments of the present application provide related architectures, devices, and methods below, so as to achieve the purpose of improving the richness and accuracy of the model results.

[0127] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the architecture of a model training system provided by an embodiment of the present application. The architecture at least includes a model training device 101 and one or more data acquisition devices 102. The model training device 101 communicates with the data acquisition device 102 in a wired or wireless manner. The data acquisition device 102 can send the captured first training data and second training data to the model training device 101; correspondingly, the model training device 101 trains the three-dimensional face construction model through the received first training data and second training data. It should be noted that the quality of the first training data is higher than the quality of the second training data.

[0128] Optionally, there are multiple data acquisition devices 102, for example, including a first data acquisition device and a second data acquisition device. The first data acquisition device is used to acquire the first training data, and the second data acquisition device is used to acquire the second training data. For example, the first data acquisition device is a device capable of performing light cage scanning, and the second data acquisition device is a device for acquiring network data or database data through the network. Among them, the second data acquisition device can be a cloud or a terminal device.

[0129] Optionally, there is only one data acquisition device 102. In this case, the data acquisition device 102 can acquire the first training data and the second training data.

[0130] Since the first training data and the second training data need to be processed to be obtained, in an optional implementation manner, the processing process of the first training data and the second training data is performed on the data acquisition device 102. In this case, the model training device 101 is only used to perform processing tasks related to model training. Optionally, since the first training data and the second training data have a sequence when performing model training, and since the model training device 101 is only used to perform processing tasks related to model training, therefore, the timing of inputting the first training data and the second training data into the model training device 101 for model training is controlled by the data acquisition device 102.

[0131] Optionally, the three-dimensional face construction model in the model training device 101 is a model constructed based on an architecture similar to 3DMM.

[0132] The model training device 101 can be a device with strong computing power. For example, a server, or a server cluster composed of multiple servers.

[0133] In an alternative embodiment, the system further includes one or more model usage devices 103. The model training device 101 communicates with the model usage device 103 in a wired or wireless manner. Thus, the model training device 101 can send the trained model (or network) for constructing a three-dimensional face to the model usage device 103. Correspondingly, the model usage device 103 predicts the three-dimensional face corresponding to the two-dimensional face to be predicted based on the received model.

[0134] The model usage device 103 can be a device that needs to perform image recognition (or detection), such as a handheld device (e.g., mobile phone, tablet computer, personal digital assistant, etc.), a vehicle-mounted device (e.g., car, bicycle, electric vehicle, airplane, ship, etc.), a wearable device (e.g., smart watch (such as iWatch, etc.), smart bracelet, pedometer, etc.), a smart home device (e.g., refrigerator, television, air conditioner, electric meter, etc.), a smart robot, a workshop device, and so on.

[0135] Optionally, the model usage device 103 can feedback the result predicted based on the model to the above-mentioned model training device 101, so that the model training device 101 can further train the model based on the prediction result of the model usage device 103. The retrained model can be sent to the model usage device 103 to update the original model.

[0136] In an alternative embodiment, the system further includes an application server. In the training phase, the model training device 101 uses the first training data to train the initial model to obtain the first face construction model. Subsequently, the model training device 101 uses the second training data to train the first face construction model to obtain the second face construction model. In the model usage phase, the second face construction model is stored in the application server. In one implementation manner, the application server can send the second face construction model to the model usage device 103. The model usage device 103 collects a two-dimensional face image and uses the second face construction model to process the collected two-dimensional face image to output a three-dimensional face image. The model usage device 103 performs subsequent processing based on the three-dimensional face graph. For example, the subsequent processing can be face matching processing (applied to face recognition). In another implementation manner, the model usage device 103 collects an image and sends the collected image to the application server. The application server uses the second face construction model to process the image to output a three-dimensional face image, and the application server performs subsequent processing based on the three-dimensional face image. For example, the subsequent processing can be face matching processing (applied to face recognition), and sends the processing result to the model usage device 103.

[0137] It can be understood that the above-mentioned model training device 101 and model using device 103 can be two separate devices or one device, such as a terminal in any of the above specific forms. The above-mentioned model training device 101 and application server can be two separate devices or one device, such as a server. This application does not limit this.

[0138] Next, the specific implementation process of constructing training data and training a model will be explained.

[0139] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a model training method provided by an embodiment of this application. This method can be implemented based on a model training device of any architecture in the Figure 1 model training system shown. This method includes but is not limited to the following steps:

[0140] Step S201: Obtain a plurality of first training data.

[0141] Among them, the first training data includes first three-dimensional face data and first two-dimensional face images. It should be noted that any one of the plurality of first training data includes one or more first two-dimensional face images and the corresponding first three-dimensional face data. The one or more first two-dimensional face images can be face data of the same user under different directions and different illuminations.

[0142] In the embodiment of this application, using a frontal two-dimensional face image for the subsequent generation process of the face construction model can improve the model generation efficiency; using multiple side two-dimensional face images at different angles for the subsequent generation process of the face construction model can provide more face regions, so that the texture information in the side face region of the texture map is more. And it can avoid the stretching of the generated facial texture in the side face region, reduce the resource consumption for processing the stretching situation, and at the same time improve the model generation accuracy.

[0143] In an alternative implementation, the first three-dimensional face data in the first training data needs to be obtained first. Specifically, in this implementation, the first training data is high-quality training data. Considering that it is difficult to fully display details such as texture and image depth in two-dimensional face images, it is difficult to obtain the high-quality three-dimensional face data through generation, rendering, etc. Therefore, the first three-dimensional face data in the first training data in this implementation is obtained by a three-dimensional surround shooting method. Exemplarily, the above three-dimensional surround shooting method includes light cage scanning. Through a shooting method similar to light cage scanning, a non-blind facial and even head image of the person being photographed can be obtained. The first three-dimensional face data obtained in this way has clear texture, uniform illumination, and obvious feature points. Of course, the first three-dimensional face data in the first training data can also come from open-source high-quality three-dimensional face data in addition to being obtained based on the above three-dimensional surround shooting method.

[0144] In the implementation of this application, high-quality first three-dimensional face data refers to a data set that can more accurately reflect the three-dimensional structural characteristics of a human face. These data are usually obtained through professional three-dimensional scanning devices or technologies, which can capture the minute undulations, concavities and convexities on the surface of the human face, as well as facial features at different angles, so as to construct an accurate human face construction model.

[0145] The above content involves high-quality first training data, where the quality indicates the authenticity of the two-dimensional face images or three-dimensional face data in the first training data. The first three-dimensional face data in this embodiment is collected by multiple cameras in light cage scanning, which saves the information of the human face from various perspectives and is closer to a real person.

[0146] When reconstructing three-dimensional data from two-dimensional data obtained through network channels, usually there is only a single photo, lacking a three-dimensional structure. It can only ensure a relatively high projection similarity of the three-dimensional data in the direction of the two-dimensional data, and the depth information in other directions is missing. Therefore, from the perspective of face reconstruction, the true value is relatively low. In addition, the two-dimensional pictures obtained through network channels may also have problems such as image compression or stylization processing, which will all reduce the quality of the constructed three-dimensional face data. Therefore, the quality of the two-dimensional face images obtained through network channels and the three-dimensional face data reconstructed from these two-dimensional face images is relatively low.

[0147] In summary, the process of three-dimensional face construction has high requirements for the quality of training data. In the process of defining training data by quality in this application, it is mainly carried out from multiple aspects such as the accuracy, integrity, resolution, and noise control of the training data. The following exemplarily illustrates high-quality three-dimensional face data. High-quality three-dimensional face data has the following characteristics:

[0148] Higher accuracy: 3D face data can accurately reflect the true 3D shape of the face, including the dimensions, proportions, and relative positions of various parts.

[0149] Better integrity: 3D face data can cover all aspects of the face, including various angles such as the front, side, upward view, and downward view, without any omissions or missing parts.

[0150] Higher resolution: 3D face data has sufficient resolution to capture fine features on the face surface, such as wrinkles and pores.

[0151] Better noise control: The noise in 3D face data is effectively controlled and will not interfere with the construction of the 3D model and subsequent applications.

[0152] Therefore, considering that there are defects in terms of accuracy and integrity in the 3D data obtained by re-modeling from 2D data acquired through network channels, when this application conducts the initial model training using high-quality heterogeneous data pairs, it selects to obtain high-quality first 3D face data through methods such as 3D scanning devices or technologies, and renders high-quality first 2D face images based on the first 3D face data.

[0153] Correspondingly, the characteristics of the high-quality 2D face images can be analogous to the characteristics of the above-mentioned high-quality 3D face data.

[0154] Optionally, both the first 3D face data and the first 2D face images in the first training data provided by this application are high-quality images / data, that is, they have characteristics such as high accuracy, good integrity, high resolution, and good noise control.

[0155] In an optional implementation manner, the first 2D face image in the first training data is rendered based on the first 3D face data in the first training data. Therefore, the first 2D face image corresponds to the first 3D face data. It should be noted that the corresponding relationship is used to refer to the same subject displayed in the 2D image and 3D data. For example, the 2D image shows the front image of the user, and the 3D data shows the 3D modeling of the user's head / face. The 2D image and 3D data with a corresponding relationship display the characteristics / images of the same subject in different dimensions. Therefore, there is a one-to-one correspondence between the first 2D face image and the first 3D face data in the first training data. Specifically, the obtaining of multiple first training data includes:

[0156] Obtain multiple first 3D face data;

[0157] Render the multiple first three-dimensional face data to obtain first two-dimensional face images respectively corresponding to the multiple first three-dimensional face data; the multiple first three-dimensional face data and the first two-dimensional face images form first training data. Exemplarily, the first training data is a data set, which includes a plurality of heterogeneous data pairs, and each heterogeneous data pair includes a certain first three-dimensional face data and a certain first two-dimensional face image. There is a corresponding relationship between the three-dimensional data and the two-dimensional data in the same data pair, that is, the displayed subject is the same subject.

[0158] Optionally, since the first two-dimensional face images obtained by rendering can be from various angles, in the embodiments of the present application, the front direction is preferentially rendered to obtain first two-dimensional face images with frontal faces.

[0159] Of course, in different implementation scenarios, the directions of the preferentially rendered first two-dimensional face images are different. For example, in the face recognition scenario, since most face recognition is based on frontal faces, when selecting the training data of the model, first two-dimensional face images with frontal faces are preferentially rendered. In other scenarios, since it is difficult to obtain frontal two-dimensional face images, most of them are side face images. Correspondingly, in the training data of the model, first two-dimensional face images at various side angles are preferentially rendered.

[0160] It should be noted that the above rendering process can occur on the model training device or on other devices except the model training device.

[0161] In another optional implementation manner, the first two-dimensional face images in the first training data are obtained during the process of obtaining the first three-dimensional face data in the first training data. Exemplarily, if the first three-dimensional face data is obtained based on the surround shooting method, then in this process, first two-dimensional face images that meet the requirements may be obtained without rendering. Of course, if the first three-dimensional face data is obtained through other channels, such as the first three-dimensional face data obtained from a database, there may be no corresponding two-dimensional face images that meet the training requirements, then the first three-dimensional face data still needs to be rendered to obtain corresponding and required first two-dimensional face images.

[0162] Since the first three-dimensional face data in the first training data may be obtained through different channels, in an optional implementation manner, the first three-dimensional face data in the first training data is unified into the same topological structure to ensure data consistency.

[0163] Step S202: Train the multiple first training data to obtain a first face construction model.

[0164] Among them, the first 3D face data in the first training data is label data, and the first 2D face image in the first training data is feature data.

[0165] In an optional implementation manner, the first face construction model uses a preset empirical model of the 3D face shape and an objective function matching the empirical model of the 3D face shape to construct the 3D face shape corresponding to the 2D face image.

[0166] The empirical model of the 3D face shape is a statistical model obtained by pre-training to describe the differences in different face shapes. Since the empirical model of the 3D face shape is constructed based on the prior knowledge of the face shape, in the art, the empirical model of the 3D face shape is also referred to as the prior model of the 3D face shape.

[0167] In the embodiments of the present application, the empirical model can be any empirical model of the 3D face shape, and no limitation is imposed thereon.

[0168] Optionally, the process of 3D face construction is actually a process of combining face features. Therefore, in the process of 3D face construction, it is necessary to first extract face features, train to obtain a face conversion network, and then input the face image obtained by segmenting the first 2D face image in the first training data into the face conversion network, so as to complete the conversion from the 2D face image to the 3D face image. Among them, the face features can include geometric features and characterization features of the face. The geometric features are specifically the distances, areas, and angles between the facial features of the face, and the characterization feature is to pass the face image through a neural network to obtain a feature vector with a specific dimension, and this vector can well characterize the face data.

[0169] Exemplarily, the following gives an example of using the method of supervised monocular 3D face reconstruction (Self-Supervised Monocular 3D Face Reconstruction, MGCnet) as the initial model. Taking MGCnet as an example, this algorithm mainly uses a deep learning network to estimate the parameters of the 3D face model to obtain a face construction model. First, according to the dataset of the face construction model, the model is trained to obtain a 3D face construction model. Then, the input 2D face image is analyzed for facial features to determine the coefficients corresponding to the 3D face on each basis vector, including the face shape coefficient and the face texture coefficient. Then, according to the above coefficients, the shape and texture of the 3D face corresponding to the 2D face image are adjusted and determined to determine the finally output 3D face image / 3D face data.

[0170] For example, a face construction model can be a universal model that uses a fixed number of points to represent a face. Its core concept is that faces can be matched one-to-one in three-dimensional space, and a universal model can be obtained by weighted linear addition of orthogonal basis vectors of many faces. In our three-dimensional space, each point (x, y, z) is actually the weighted addition of basis vectors in three directions of three-dimensional space: (1, 0, 0), (0, 1, 0), and (0, 0, 1), with the weights being x, y, and z, respectively. Similarly, each three-dimensional face can be represented in a basis vector space composed of all faces in a database. Solving the model for any three-dimensional face is essentially equivalent to solving the problem of finding the coefficients of each basis vector. Basic facial attributes include shape and texture, and each face can be represented as a linear superposition of shape and texture vectors. It should be noted that the three-dimensional face construction model can also be used to reconstruct the face through a reconstruction algorithm, thereby outputting a three-dimensional face.

[0171] In an optional embodiment, the first face construction model includes an adversarial network model, which can be found in Figure 3 , Figure 3 A structural schematic diagram of an adversarial network model provided in an embodiment of the present application, wherein the adversarial network model includes a facial feature encoder, a generator, and a discriminator. The facial feature encoder is used to generate a corresponding facial feature vector based on a two-dimensional image, the generator is used to generate three-dimensional facial data based on the facial feature vector and random noise, and the discriminator is used to judge the generated three-dimensional facial data and the label data in the training data, and output a judgment result. Optionally, the above-mentioned judgment process is based on the four aspects of accuracy, completeness, resolution, and noise control for quality assessment. It can be understood that if the generated three-dimensional facial data meets the requirements of the above four aspects, the discriminator outputs True, or outputs False if it does not meet the requirements.

[0172] Optionally, during the above training process, the loss function of the model is as follows:

[0173] minmaxV(D,G)=IE χ～pdata (χ)[logD(χ)]+IE z～pz(z) [log(1-D(G(z)))]

[0174] Where D is the discriminator, G is the generator, (x) is the sample sampled from the true data distribution p_data(x), (z) is the sample sampled from the noise distribution p_z(z), (D(x)) is the discriminator's predicted probability for the true data, D(G(z)) is the discriminator's predicted probability for the generated data, and G(z) is the fake data generated by the generator based on the noise (z).

[0175] In an alternative implementation, the process of obtaining the first two-dimensional face image is to generate a two-dimensional image through geometric transformation, projection transformation, perspective transformation, and window clipping of a first three-dimensional face data, and then through the obtained material and light and shadow information. The shading model describes how the color of a rendered object changes based on coefficients such as surface orientation, view direction, and lighting. In other words, the shading model can control the input data that makes up the material to create the final appearance. The first three-dimensional face data in the first training data is rendered under specific lighting to obtain the first two-dimensional face image, and the specific lighting can be ambient light, parallel light, point light, or other light. The above shading model can use direct and indirect lighting, as well as reflected highlights. That is to say, after the three-dimensional face data is irradiated by specific lighting and then rendered, the first two-dimensional face image includes the color image information formed by the reflection of the specific lighting by the first three-dimensional face data, and the shadow information that may be generated when the specific lighting irradiates the surface of the three-dimensional face data.

[0176] In another alternative implementation, the first two-dimensional face image is an image generated by rendering the three-dimensional face data without lighting. The first three-dimensional face data without lighting irradiation only outputs the self-luminescence of the model color, so the first two-dimensional face image rendered only includes the texture color of the self-luminescence of the model itself and does not include any shadow information.

[0177] In addition, assuming that the specific lighting is an ideal point light source, then obvious boundary shadows may be formed after the first three-dimensional face data is irradiated. Assuming that the specific lighting is a surface light source with a certain size, then shadows with model boundaries may be formed after the three-dimensional face data is irradiated.

[0178] Since the first two-dimensional face image contains shadow information, geometric features can be reflected on the image, such as the distance, area, and angle between the facial features, thereby highlighting the three-dimensional effect of the facial features of the first three-dimensional face data. Then when training the model based on the first two-dimensional face image and the first three-dimensional face data, a three-dimensional model with a more three-dimensional output result can also be obtained.

[0179] Step S203: Obtain a plurality of second training data.

[0180] The second training data includes second three-dimensional face data and second two-dimensional face images, and the quality of the second training data is lower than the quality of the first training data.

[0181] Based on the above description of the characteristics of high-quality data, it can be understood that the quality of the second training data is lower than that of the first training data, indicating that the data in the second training data is still relatively high in terms of accuracy, integrity, resolution, noise control, etc., but is lower than the first training data in one or more of these aspects.

[0182] In an optional implementation manner, the second two-dimensional face images in the second training data are obtained through at least one of network data, open-source databases, etc. Considering that the number of two-dimensional face images in network data or open-source databases is much larger than the number of three-dimensional face data, the second three-dimensional face data in the second training data is generated based on the two-dimensional face images, and the generation method can be any one of methods such as deep learning-driven face reconstruction FaceScape, Metric Face (MI CA), or 3DMM, which is not limited in this application.

[0183] There are differences in the quality of three-dimensional face data obtained through different paths. For example, the quality of three-dimensional face data obtained through three-dimensional scanning devices / three-dimensional scanning technologies is higher than that obtained from network channels. And the above second training data has a quality lower than that of the first training data. Correspondingly, the source of the second training data is different from that of the first training data. It can be understood that the data from network channels has low quality, so it is used as the second training data, and the data from three-dimensional scanning devices / three-dimensional scanning technologies has high quality, so it is used as the first training data.

[0184] In an optional implementation manner, the sources of the second training data and the first training data are the same. For example, both the second training data and the first training data are sourced from light cage scanning. After quality determination, the data with poor performance in terms of accuracy, integrity, resolution, and noise control is classified as the second training data, and the data with better performance is classified as the first training data.

[0185] In another optional implementation manner, the second three-dimensional face data in the second training data is obtained by distorting the first three-dimensional face data in the first training data. The types of distortion processing here include any one or more of face deformation, uneven illumination, color cast, motion blur, scaling blur, and noise. It should be emphasized that the face deformation is used to change the facial features of the first three-dimensional face data in the first training data. For example, changing the distance between the two eyes, etc., so as to obtain a face different from the first three-dimensional face data in the first training data.

[0186] Correspondingly, the second two-dimensional face image in the second training data can be generated based on the second three-dimensional face data in the second training data, or be the first two-dimensional face image of the first training data that has undergone distortion processing. Taking the second two-dimensional face image in the second training data being the first two-dimensional face image of the first training data that has undergone distortion processing as an example, the process of distortion processing is described below:

[0187] Specifically, the first training data is subjected to distortion processing according to different distortion types to obtain distorted face images, and the distorted face data represents the second training data, including two-dimensional face images and three-dimensional face data.

[0188] In this embodiment, the step of subjecting the first two-dimensional face image in the first training data to distortion processing according to different distortion types to obtain distorted face images may include: using image processing techniques to perform distortion processing on the first two-dimensional face image according to different distortion types and different distortion degrees to obtain the distorted face images; where the distortion types include any one or more of face deformation, uneven illumination, color cast, motion blur, scaling blur, and noise. That is, using image processing techniques to perform distortion processing of different distortion types and different distortion degrees on the first two-dimensional face image. It can be understood that in this embodiment, multiple first two-dimensional face images are obtained, and each image can be subjected to distortion processing of one or more distortion types respectively, and the distortion degree can be adjusted in terms of strength according to settings or can be random.

[0189] In this embodiment, the step of subjecting the first two-dimensional face image to distortion processing according to different distortion types and corresponding distortion degrees to obtain the distorted face images may include: migrating the illumination component of the non-uniform illumination image onto the first two-dimensional face image and adjusting the image brightness component to obtain the distorted face image with uneven illumination; and / or, adjusting the chrominance component of the first two-dimensional face image to obtain the distorted face image with color cast; and / or, performing a convolution operation on the high-quality image to obtain the distorted face image with motion blur; and / or, performing a scaling operation on the high-quality image to obtain the distorted face image with scaling blur; and / or, adding Gaussian noise to the high-quality image to obtain the distorted face image containing noise. That is, to generate distorted face images of different distortion types, corresponding distortion processing operations are used, and a high-quality image can be subjected to multiple distortion types of processing to form a mixed distortion.

[0190] It can be understood that since it is difficult to collect distorted images of the same human face under different distortion types and different distortion degrees, in this embodiment, through an image processing method, different types of distortions such as human face deformation, uneven illumination, color cast, blur, and noise are simulated, so as to obtain the distortion feature map labels corresponding to various distortions. Exemplarily, illumination distortion includes uneven image brightness caused by illumination and the situations of over-bright and over-dark image brightness, which are caused by different light source positions and illumination intensities. For illumination distortion, in this embodiment, the illumination component of the non-uniform illumination image is migrated to the image with uniform illumination in the first two-dimensional human face image and the image brightness component is adjusted to obtain images of various illumination distortion types. Further, image blur distortion includes image motion blur and image scaling blur, which are caused by human face movement and too small images respectively. Image noise is mainly Gaussian noise, which is mainly caused by environmental noise and the noise of circuit components themselves. In this embodiment, motion blur convolution, image scaling, and adding Gaussian noise are respectively used for processing to obtain the corresponding distorted images.

[0191] It can be seen that in this embodiment, the influencing factors of image quality in the actual scenario are classified as human face deformation, illumination distortion, color distortion, blur, noise, etc., and the distorted images in the actual scenario are regarded as the superposition of single or multiple types of distortions such as human face deformation, illumination distortion, color distortion, blur, and noise. Through the image processing method, the distortion type and distortion degree of the image are automatically controlled, greatly enriching the data type and data volume, enriching the training data, solving the problem that it is difficult to collect model training data, and facilitating the training of the human face construction model.

[0192] Optionally, the second three-dimensional human face data in the second training data has undergone retopology processing, and the topological structure of the second three-dimensional human face data is the same as that of the first three-dimensional human face data in the first training data.

[0193] In an optional implementation manner, the quality of the training data is determined based on a quality assessment model, and the quality assessment model is used to determine the difference size between the training data and the real data.

[0194] The training data includes first training data and second training data. Optionally, the data whose quality is determined by the quality assessment model is the three-dimensional facial data in the first training data and the second training data. The quality assessment model is trained based on historical training data and real data, as well as the difference between the training data and the real data. The difference can be expressed numerically in the model expression. Exemplarily, the real data is three-dimensional facial data composed of two-dimensional data obtained by a special surround shooting method, or three-dimensional facial data manually pinched based on the entity of a real user, or three-dimensional facial data manually estimated based on a two-dimensional facial image displayed as a front face. It should be noted that the process of determining the quality of the training data can occur in a device other than the model training device. In this case, the model training device does not need to perceive the quality level and only needs to receive the input of the training data. Optionally, the process of determining the quality of the training data occurs in the data acquisition device, that is, the quality assessment model is set in the data acquisition device.

[0195] The common method of evaluating the quality of three-dimensional face data is mainly to determine it qualitatively, that is, through subjective judgment by the human eye. However, this embodiment provides a quantitative method to accurately evaluate the training data and reduce labor costs.

[0196] Based on the above description of quality, high-quality 3D facial data has the following characteristics: accuracy, 3D facial data can accurately reflect the true 3D shape of the face, including the size, proportion and relative position of each part; completeness, 3D facial data can cover all aspects of the face, including front, side, upward and downward angles, without missing or missing parts; resolution, 3D facial data has sufficient resolution to capture subtle features on the face surface, such as wrinkles and pores; noise control, the noise in the 3D facial data is effectively controlled and will not interfere with the construction of the 3D model and subsequent applications.

[0197] Therefore, in the embodiments of the present application, a construction idea of a quality assessment model is provided, which is as follows:

[0198] The idea of building a quality assessment model includes evaluating the quality of training data based on four aspects: accuracy, completeness, resolution, and noise control. The specific evaluation ideas can be as follows:

[0199] First, the accuracy is evaluated based on the training data.

[0200] Specifically, based on the correspondence between the three-dimensional face data and the two-dimensional face images in the training data, the accuracy of the training data is determined. In this embodiment, the training is mainly carried out with a single two-dimensional face image and a single three-dimensional face data as a set of training data. Considering that although some textures and details may be blocked or not displayed in the image display of a single two-dimensional face image, there are still some details shown. Therefore, the accuracy of the three-dimensional face data in the training data can be determined according to the shown partial details and the characteristics of the face. The characteristics of the face include the symmetry of the face. Exemplarily, for example, wrinkles. If the single two-dimensional face image in the training data shows a side face, there may be extended wrinkles on the other side that is not shown in the image. Of course, in the two-dimensional face images of the training data in the embodiments of the present application, the two-dimensional face images show a front face, and the above-mentioned side face is only an example.

[0201] Optionally, the training data further includes a control group, which includes two-dimensional face images in other directions except the front face images used as training data. When the two-dimensional face images in the training data show a front face, it may be difficult to intuitively obtain the sizes of the facial feature parts. For example, it is difficult to show the accurate height of the nasal peak in a front face, but the height of the nasal peak can be shown in the two-dimensional face image of the side belonging to the same person. The accuracy of the three-dimensional face data in the same group can be determined through the two-dimensional face images of the control group and the training data.

[0202] Furthermore, in the process of determining the accuracy of the three-dimensional face data, specific parameters of the face can be determined from the two-dimensional face images in the same group (including a single front two-dimensional face image and / or the control group). The specific parameters include the sizes, proportions, and relative positions of each part. Then, the accuracy is evaluated based on the difference between the specific parameters shown by the three-dimensional face data and the specific parameters of the face determined based on the two-dimensional face images in the same group (including a single front two-dimensional face image and / or the control group). The greater the difference, the lower the accuracy. For example, in the specific parameters of the face shown in the three-dimensional face data, the proportion of the nose in the head is, when viewed from the front, the nose occupies 1 / 5 of the longitudinal division of the head, and when viewed from the side, the nose occupies 1 / 3 of the transverse division of the face. While in the two-dimensional face images in the same group (including a single front two-dimensional face image and / or the control group), the proportion of the nose in the head is, when viewed from the front, the nose occupies 1 / 5 of the longitudinal division of the head, and when viewed from the side, the nose occupies 1 / 4 of the transverse division of the face. This undoubtedly indicates an error in the height of the nasal peak in the three-dimensional face data, and the accuracy of the corresponding training data will be relatively low. If the accuracy is evaluated in the form of a score, the accuracy score of this training data is relatively low.

[0203] Second, evaluate based on the integrity of the training data.

[0204] The integrity of the training data is mainly reflected in all angles of the 3D face data, such as the front, side, upward view, and downward view, without any omission or missing part. In an optional implementation, based on the comparison result between the facial features shown in the real data and the 3D face data in the training data, the integrity of the training data is determined. For example, the facial features shown in the real data include a wrinkle with a longitudinal distance of 3.1 cm and a transverse length of 5 cm between the eyebrows, but the 3D face data in the training data does not show such a wrinkle, indicating that there is a lack of this facial feature. It should be noted that in practical applications, most wrinkles are not straight lines but have multiple curvatures. If the wrinkle shown in the real data has multiple curvatures, while the 3D face data in the training data shows this wrinkle but the curvature of the shown wrinkle is different from that in the real data, it is considered that the integrity of the training data in terms of wrinkles is good, but there is a problem with accuracy. The above example is only used to illustrate the integrity of facial features and is not an actual case. Of course, if the facial features shown in the 3D face data in the training data include a wrinkle with a longitudinal distance of 3.1 cm and a transverse length of 5 cm between the eyebrows, the wrinkle has no curvature and is specifically shown at the corners of the eyes, while the facial features shown in the 3D face data in the real data include a wrinkle with a longitudinal distance of 3.1 cm and a transverse length of 5 cm between the eyebrows, and the wrinkle has multiple curvatures and is specifically shown at the forehead, it can also illustrate that there is an error between the training data and the real data, which represents that the integrity and accuracy of the training data are relatively low.

[0205] Specifically, according to the importance degree of the facial features, the reference score, that is, the full score, of the facial features is determined. Exemplarily, among the facial features, the importance degree of the facial features of the five sense organs is greater than that of the detail parts, so the reference score of the relevant parameters of the facial features of the five sense organs is higher than that of the relevant parameters of the detail parts.

[0206] Third, evaluate based on the resolution of the training data.

[0207] The 3D face data in the training data needs to have sufficient resolution to capture the subtle features on the face surface, such as wrinkles, pores, etc. Therefore, the subtle features on the face surface shown in the real data can be compared with the subtle features on the face surface shown in the 3D face data in the training data, and the evaluation process is similar to the above integrity evaluation or accuracy evaluation process.

[0208] Fourth, evaluate based on the noise control level of the training data.

[0209] The noise in the two-dimensional face images and three-dimensional face data in the training data needs to be effectively controlled to avoid interference with the construction of the face construction model and subsequent applications. Optionally, the noise in the two-dimensional face images and three-dimensional face data in the training data is processed, and the degree of noise control of the processed two-dimensional face images and three-dimensional face data is evaluated. The training data with higher noise is determined as the training data with lower quality.

[0210] In summary, the quality evaluation model in the embodiment of the present application determines the quality of each training data according to the evaluation results of the above four aspects and the corresponding weight ratios. The quality can be specifically displayed in the form of a quality score. The higher the quality score, the higher the quality of the corresponding training data.

[0211] In an optional implementation manner, if the sources of the three-dimensional face data prepared before model training are numerous and mixed, it is necessary to distinguish the quality of the three-dimensional face data, and then divide the first training data and the second training data. The three-dimensional face data is distinguished according to the quality score of each three-dimensional face data, so as to obtain the first training data and the second training data. It should be noted that when the quality score of a certain three-dimensional face data is high and it is assigned to the first training data, the two-dimensional face image corresponding to the three-dimensional face data is also assigned to the first training data and still has a corresponding relationship. It can be understood that the two-dimensional face image and the corresponding three-dimensional face data have a common real user (the portrait of the same person). Therefore, to ensure data consistency, when the corresponding relationship between a certain three-dimensional face data and a certain two-dimensional face image is determined, the two are bound. Even in the subsequent process, when the quality of the three-dimensional face data changes (for example, after topological processing, the three-dimensional face data is deformed) and is assigned to other training data, the corresponding two-dimensional face image is still bound to the three-dimensional face data and is also assigned to other training data.

[0212] In an alternative embodiment, any training data in the second training data includes one or more two-dimensional face images and corresponding three-dimensional face data, where the corresponding relationship indicates that the two-dimensional face image and the three-dimensional face data represent the same user. As can be seen from the above, the amount of data applied in the process of model training for the second training data with quality lower than the first training data is huge, and three-dimensional face data is relatively rare while two-dimensional face images are relatively common. Therefore, in the embodiments of the present application, a method of generating three-dimensional face data from two-dimensional face images is adopted to obtain a large amount of medium- and low-quality three-dimensional face data. The specific method of generating three-dimensional face data can be a method in the prior art or a future method for generating three-dimensional face data, which will not be listed one by one here. In an alternative embodiment, the obtaining of multiple second training data includes: obtaining multiple second two-dimensional face images; performing three-dimensional reconstruction on the multiple second two-dimensional face images to generate the multiple second three-dimensional face data respectively corresponding to the multiple two-dimensional face images, and the multiple second two-dimensional face images and the second three-dimensional face data form the multiple second training data.

[0213] Considering that in the actual application process, there may be a large amount of medium- and low-quality training data, and when dealing with the noise in the medium- and low-quality training data, the process is complex and the resource consumption is large. In an alternative embodiment, the second training data is screened to ensure the consistency and stability of the training data. The specific process is as follows:

[0214] The obtaining of multiple two-dimensional face images includes:

[0215] Obtaining multiple initial two-dimensional face images;

[0216] Screening the multiple initial two-dimensional face images to obtain multiple two-dimensional face images.

[0217] Furthermore, considering that most of the directly available two-dimensional face images (initial two-dimensional face images) are images obtained by photographing people with a camera, or video streams obtained by shooting with a camera and then segmented to obtain video images. Since the acquisition conditions of the above images are under natural light, there are unknown-angle and unknown-intensity lighting and shadow features on these images, and the situation is complex. In order to achieve accurate screening, fixed conditions are set to screen the data and improve the effect of model training. In an alternative embodiment, the screening of the multiple initial two-dimensional face images to obtain multiple two-dimensional face images includes:

[0218] Input the multiple initial two-dimensional face images into a preset data evaluation model to obtain probability values corresponding to the multiple initial two-dimensional face images respectively. The probability value is used to characterize the matching degree between the initial two-dimensional face image and a two-dimensional face image that meets a preset first condition. The higher the probability value, the higher the matching degree. The preset first condition includes one or more of uniform illumination, a screen occlusion lower than a preset first threshold, and a frontal face image.

[0219] Use the multiple initial two-dimensional face images with probability values higher than a preset second threshold as the multiple two-dimensional face images.

[0220] Step S204: Train the first face construction model according to the multiple second training data to obtain a second face construction model.

[0221] The second three-dimensional face data in the second training data is label data, and the second two-dimensional face image in the second training data is feature data. The second face construction model is used to generate a three-dimensional face image.

[0222] Optionally, the second face construction model is a model obtained by updating the first face construction model. The two have the same main architecture. Before training the model with the second training data, correct the parameters or other detailed architectures in the model to avoid the problem of catastrophic forgetting of the model that may be caused by the data volume contained in the second training data being much larger than the data volume contained in the first training data. Therefore, in an optional implementation manner, the step of training the first face construction model according to the multiple second training data to obtain a second face construction model, the specific model structure can be referred to Figure 4 , Figure 4 which is a schematic structural diagram of a face construction model provided by an embodiment of the present application.

[0223] In the implementation manner of the present application, the multiple second training data are input into the first face construction model for training by means of incremental learning to obtain the second face construction model.

[0224] Incremental learning is a machine learning method that can solve a common problem in model training, namely "catastrophic forgetting". For general machine learning models, especially deep learning methods based on backpropagation, when training a new task, their performance on the old task usually drops significantly. This is mainly because traditional models assume that the data distribution is fixed or stationary, and when the data becomes a continuous data stream, the acquisition of new knowledge will interfere with the old knowledge, resulting in a rapid decline in model performance, or even completely covering or forgetting the knowledge learned before.

[0225] In the embodiments of the present application, the goal of applying incremental learning is to find a balance between stability and plasticity under the condition of limited computing and storage resources. Incremental learning alleviates the problem of catastrophic forgetting through different strategies, such as using replay mechanisms, parameter isolation, regularization, or dynamically expanding the network. These methods enable the model to learn new knowledge while retaining knowledge of past tasks without storing a large amount of old data. Therefore, through incremental learning, the model can more effectively process continuous data streams and maintain memory of old tasks when training on new tasks, thereby enhancing its anti-forgetting ability.

[0226] Specifically, first, a neural network module is trained using a small amount of high-quality heterogeneous data pairs (first training data) to ensure the basic generation ability of the model, that is, to ensure the lower limit of the model. Secondly, when each batch of new medium- and low-quality heterogeneous data pairs (second training data) arrives, incremental learning is performed, and the newly arrived data enhances the generalization and expressiveness of the model. Optionally, the incremental learning is domain incremental learning, which mainly highlights the anti-forgetting ability of the model. In specific implementation, a distillation loss can be introduced into the loss function to make the prediction of the old model on new data close to the prediction of the new model on new data. In the embodiments of the present application, the training process of the model can be referred to Figure 5 , and the first training data with higher quality is preferentially input into the initial model for training to obtain the first face construction model, and then the second training data is input into the first face construction model obtained by the above training through incremental learning for training until the model converges to obtain the second face construction model.

[0227] In an optional embodiment, the second face construction model includes an adversarial network model, and the adversarial network model includes a face feature encoder, a generator, and a discriminator. The face feature encoder is used to generate a corresponding face feature vector according to a two-dimensional image, the generator is used to generate three-dimensional face data according to the face feature vector and random noise, and the discriminator is used to determine whether the three-dimensional face data is three-dimensional face data that meets a preset second condition.

[0228] In order to further improve the efficiency and effect of model training, the medium- and low-quality second training data is split again to obtain training data with multiple quality gradients. In an optional embodiment, according to the quality scores output by the above quality evaluation model, the second training data is divided into at least two data sets. Taking the division of the second training data into two data sets as an example, the second training data includes a first data set and a second data set. The first data set includes two-dimensional face images and three-dimensional face data, and the second data set includes two-dimensional face images and three-dimensional face data. The quality of the first data set is higher than that of the second data set.

[0229] It should be noted that dividing the second training data into two data sets in this embodiment is for the purpose of explaining this process. In actual applications, the second training data can be divided into multiple data sets of different qualities, and the data sets are input into the model for training in descending order of quality. Generally speaking, the lower the quality of the training data, the larger the quantity it includes. The specific process is similar to the above training process and will not be elaborated here.

[0230] Step S205: Process the result output by the face construction model through the human-in-the-loop method to obtain feedback data.

[0231] In an alternative embodiment, the human-in-the-loop method includes user selection and artist correction. The user selection is used to represent that the user evaluates the result output by the face construction model to obtain an evaluation result, and the model is updated through this evaluation result. The artist correction is used to represent that the artist corrects the result output by the face construction model to obtain a correction result, and the model is updated through this correction result. It should be noted that the artist here refers to the group of people who are capable of making positive corrections to three-dimensional faces, rather than the group of people whose occupation is an artist. The above human-in-the-loop method can effectively introduce human subjective judgment and experience to correct and optimize the model output, thereby improving the accuracy and practicality of the model. At the same time, through continuous iteration and optimization, the performance of the model can be further improved.

[0232] In an alternative embodiment, the above face construction model includes a first face construction model and / or a second face construction model. This means that the face construction model can be the first face construction model or the second face construction model. If the face construction model is the first face construction model, it means that the human-in-the-loop method is carried out during the model training process. Correspondingly, if the face construction model is the second face construction model, it means that the human-in-the-loop method is carried out after the model training is completed.

[0233] The following takes the face construction model as the second face construction model as an example to exemplarily illustrate step S205, which specifically includes:

[0234] Receive the target two-dimensional face image; input the target two-dimensional face image into the second face construction model to generate target three-dimensional face data corresponding to the target two-dimensional face image; display the target three-dimensional face data; receive the first model correction instruction / the second model correction instruction.

[0235] The first model correction instruction is used to adjust the appearance of the target three-dimensional face data to obtain the corrected target three-dimensional face data.

[0236] Optionally, the first model correction instruction includes the corrected target three-dimensional face data, which is used to update the model. It can be understood that the corrected target three-dimensional face data is the above-mentioned feedback data. The corrected target three-dimensional face data is used to retrain the second face construction model. Optionally, the first model correction instruction is input by the above-mentioned artist after correcting the target three-dimensional face data.

[0237] The second model correction instruction is used to evaluate the generation quality of the target three-dimensional face data. The second model correction instruction includes the evaluation direction and / or evaluation level of the generation quality of the target three-dimensional face data. In this case, the above-mentioned feedback data is the evaluation direction and / or evaluation level of the generation quality of the target three-dimensional face data. Optionally, the generation quality includes multiple aspects. For example, authenticity, matching degree with the target two-dimensional face image, integrity, etc. The generated target three-dimensional face data is evaluated by the second model correction instruction to obtain the direction for optimizing the model, and the adjustment degree of the model is clarified through the evaluation direction and / or evaluation level of the generation quality of the target three-dimensional face data. For example, the parameter weights can be readjusted through the evaluation level, so as to achieve precise adjustment of the model and improve the expression ability of the model.

[0238] Optionally, the evaluation direction includes positive evaluation and negative evaluation. Positive evaluation is used to represent the user's recognition of the generation quality of the target three-dimensional face data. Correspondingly, negative evaluation is used to represent the user's non-recognition of the generation quality of the target three-dimensional face data. The evaluation level is used to represent the degree of the user's recognition / non-recognition of the generation quality of the target three-dimensional face data. Exemplarily, the evaluation level includes strong, medium, and weak. The higher the evaluation level, the higher the degree of the user's recognition / non-recognition of the generation quality of the target three-dimensional face data. It should be noted that the evaluation level can also be displayed in the form of a score.

[0239] In an optional implementation manner, in order to improve the model quality, the human-in-the-loop method occurs during the model training process, that is, the above-mentioned face construction model is the first face construction model. In this case, the process of model update is similar to the model update process in the case where the above-mentioned face construction model is the first face construction model, and the first model correction instruction and / or the second model correction instruction can also be obtained. It should be noted that in addition to the first model correction instruction and / or the second model correction instruction, a third model correction instruction can also be obtained. The third model correction instruction includes the user's adjustment suggestions for the model parameters and / or weights. Correspondingly, the feedback data includes the adjustment suggestions for the parameters and / or weights of the first face construction model. Exemplarily, the adjustment suggestions include suggesting to reduce the parameter weights involved in generating the target three-dimensional face data.

[0240] In an alternative embodiment, see Figure 6 , Figure 6 A schematic diagram of a human-in-the-loop processing method provided in an embodiment of the present application. Figure 6 The figure shows the data selection options, training strategy options, and scoring of the generated results when the user trains the model. It should be noted that the scoring of the generated results represents the judgment of the model results by the human-in-the-loop method, and is also the feedback data for the model.

[0241] Step S206: training the face construction model using the feedback data to update the face construction model.

[0242] Based on the above, it can be seen that the feedback data includes the corrected target 3D facial data, as well as the evaluation direction and / or evaluation level. Since the feedback data includes various types, the process of training and updating the face construction model using the feedback data can also be divided into multiple cases. The following exemplifies the process of updating and training the model using feedback data through three cases, as follows:

[0243] Case 1: The feedback data is the corrected target 3D face data.

[0244] In Case 1, the model is updated and trained by inputting the corrected target three-dimensional face data and the feature data used to generate the target three-dimensional face data into the face construction model.

[0245] Case 2: The feedback data includes evaluation direction and evaluation level, and the evaluation direction is positive and the evaluation level is strong.

[0246] In Case 2, the evaluation direction is positive, and the evaluation level is strong, which means that the user has a high degree of recognition of the generation quality of the target three-dimensional face data. Therefore, the model is updated by increasing the parameter weights involved in generating the target three-dimensional face data. In this case, the higher the evaluation level, the higher the numerical adjustment of the parameter weight.

[0247] Case 3: The feedback data includes evaluation direction and evaluation level, and the evaluation direction is negative and the evaluation level is medium.

[0248] In Case 3, the evaluation direction is negative, and the evaluation level is medium, which means that the user has a low degree of recognition of the generation quality of the target three-dimensional face data. Therefore, the model is updated by lowering the parameter weights involved in generating the target three-dimensional face data. In this case, the lower the evaluation level, the lower the numerical adjustment of the parameter weight.

[0249] The above-mentioned model can be a first face construction model or a second face construction model.

[0250] In an alternative implementation, in the case of a negative evaluation direction, interact with the user to obtain the user's true feedback, thereby correcting the target 3D face data, and using the corrected target 3D face data as a new training sample.

[0251] In an alternative implementation, in the case of model update training by inputting the corrected target 3D face data, in an incremental learning manner, input the target 3D face data into the model for training, update the weights of the neural network, so that the model gradually learns the user's preferences and aesthetic standards, and generates 3D face data that better meets the user's expectations.

[0252] The following shows the usage process of the model. For details, please refer to Figure 7 , Figure 7 which is a schematic flowchart of a model usage method provided by an embodiment of the present application, and can be specifically applied to Figure 1 the model usage device in the architecture shown. The specific process is as follows:

[0253] Step S701: The model usage device acquires a two-dimensional face image.

[0254] The two-dimensional face image here may be a two-dimensional face image captured by the corresponding model usage device, and the two-dimensional face images in the above training data belong to different time stages. They may be the same two-dimensional face image or may not be the same two-dimensional face image.

[0255] Step S702: The model usage device inputs the two-dimensional face image into the second face construction model to obtain 3D face data corresponding to the two-dimensional face image.

[0256] The method provided by the embodiments of this application is implemented on the basis of the drawbacks of the existing method for constructing a face model using deep learning: the lack of high-quality training data leads to insufficient robustness and poor expression ability of the trained face construction model. Considering that high-quality training data requires high labor costs and other costs to obtain, and the amount of data required for model training is huge, directly training the model with a large amount of high-quality training data is too costly. Therefore, the above method makes full use of training data of different qualities. First, the model is trained with high-quality first training data to ensure the initial accuracy and robustness of the model. Then, through two methods of gradient training and incremental learning, a large amount of second training data of medium and low quality is input into the model for retraining. The two types of data complement each other, and the two methods complement each other. It not only combines the advantages of high-quality data and a large amount of medium and low-quality data, improves the generalization ability and expression ability of the model, but also ensures that the resulting 3D face construction model can effectively improve the accuracy and stability of the 3D face construction model on the basis of less high-quality training data.

[0257] The above has elaborated in detail the method of the embodiments of this application. Below, a device corresponding to the method of the above embodiments is provided.

[0258] It should be understood that for the device provided in the embodiments of this application, the division of the units therein is only a logical function division. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated.

[0259] This application also provides a model training device, as Figure 8 shown, including: a first acquisition module, a first training module, a second acquisition module, and a second training module.

[0260] The first acquisition module is used to acquire a plurality of first training data, where the first training data includes first 3D face data and first 2D face images;

[0261] The first training module is used to train a neural network with the plurality of first training data to obtain a first face construction model. The first 3D face data in the first training data is label data, and the first 2D face images in the first training data are feature data. The first face construction model is used to generate corresponding 3D face data according to the input 2D face images;

[0262] The second acquisition module is used to acquire a plurality of second training data. The second training data includes second 3D face data and second 2D face images corresponding to the second 3D face data. The quality of the first training data is higher than that of the second training data;

[0263] A second training module, configured to train the first face construction model according to the multiple pieces of second training data, so as to obtain a second face construction model. The second 3D face data in the second training data is label data, and the second 2D face image in the second training data is feature data. The second face construction model is configured to generate corresponding 3D face data according to the input 2D face image.

[0264] Among them, the first acquisition module, the first training module, the second acquisition module, and the second training module can all be implemented by software or by hardware. Exemplarily, next, taking the first acquisition module as an example, the implementation manner of the first acquisition module will be introduced. Similarly, the implementation manners of the first training module, the second acquisition module, and the second training module can refer to the implementation manner of the first acquisition module.

[0265] As an example of a software functional unit, the first acquisition module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above computing instance may be one or more. For example, the first acquisition module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running this code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers for running this code may be distributed in the same availability zone (AZ), or in different AZs. Each AZ includes one data center or multiple geographically proximate data centers. Usually, one region may include multiple AZs.

[0266] Similarly, the multiple hosts / virtual machines / containers for running this code may be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Usually, one VPC is set within one region. For cross-region communication between two VPCs within the same region and between VPCs in different regions, a communication gateway needs to be set in each VPC, and the interconnection between VPCs is realized through the communication gateway.

[0267] As an example of a hardware functional unit, the first acquisition module may include at least one computing device, such as a server. Alternatively, the first acquisition module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). Among them, the above PLD may be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0268] The multiple computing devices included in the first acquisition module may be distributed in the same region or in different regions. The multiple computing devices included in the first acquisition module may be distributed in the same availability zone (AZ) or in different AZs. Similarly, the multiple computing devices included in the first acquisition module may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, the multiple computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0269] It should be noted that in other embodiments, the first acquisition module may be used to execute any step in the model training method, the first training module may be used to execute any step in the model training method, the second acquisition module may be used to execute any step in the model training method, and the second training module may be used to execute any step in the model training method. The steps to be implemented by the first acquisition module, the first training module, the second acquisition module, and the second training module can be specified as needed. The entire function of the model training device is realized by the first acquisition module, the first training module, the second acquisition module, and the second training module respectively implementing different steps in the model training method.

[0270] Based on the above model training system, a possible implementation of the model training system will be described below. The model training system includes:

[0271] A model training device for performing model training based on first training data and second training data, where the quality of the first training data is higher than the quality of the second training data.

[0272] A data acquisition device for generating the first training data and / or the second training data.

[0273] Both the model training device and the data acquisition device can be implemented by software or by hardware. Exemplarily, the implementation manner of the model training device will be introduced next. Similarly, the implementation manner of the data acquisition device can refer to the implementation manner of the model training device.

[0274] As an example of a software functional unit, the model training device may include code running on a computing instance. Among them, the computing instance may be at least one of computing devices such as a physical host (computing device), a virtual machine, a container, etc. Further, the above computing devices may be one or more. For example, the model training device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers for running the code may be distributed in the same AZ or in different AZs, and each AZ includes one data center or multiple geographically proximate data centers. Among them, generally one region may include multiple AZs.

[0275] Similarly, the multiple hosts / virtual machines / containers for running the code may be distributed in the same VPC or in multiple VPCs. Among them, generally one VPC is set within one region. For cross-region communication between two VPCs within the same region and between VPCs in different regions, communication gateways need to be set in each VPC, and the interconnection between VPCs is realized through the communication gateways.

[0276] As an example of a hardware functional unit, the model training device may include at least one computing device, such as a server, etc. Or, the model training device may also be a device implemented by ASIC or PLD, etc. Among them, the above PLD may be implemented by CPLD, FPGA, GAL or any combination thereof.

[0277] The multiple computing devices included in the model training device may be distributed in the same region or in different regions. The multiple computing devices included in the model training device may be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the model training device may be distributed in the same VPC or in multiple VPCs. Among them, the multiple computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0278] It should be noted that the data acquisition device may be a device related to optical cage scanning.

[0279] This application also provides a computing device 100. AsFigure 9 As shown, computing device 100 includes: bus 104, processor 105, memory 106, and communication interface 107. The processor 105, memory 106, and communication interface 107 communicate with each other via bus 104. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0280] The bus 104 can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 only one line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus. The bus 104 can include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 105, communication interface 107).

[0281] The processor 105 can include any one or more of processors such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Microprocessor (MP), or a Digital Signal Processor (DSP).

[0282] The memory 106 can include volatile memory, such as Random Access Memory (RAM). The processor 105 can also include non-volatile memory, such as Read-Only Memory (ROM), flash memory, a Hard Disk Drive (HDD), or a Solid State Drive (SSD).

[0283] The memory 106 stores executable program code, and the processor 105 executes the executable program code to respectively implement the functions of the foregoing first acquisition module, first training module, second acquisition module, and second training module, thereby implementing the model training method. That is, instructions for executing the model training method are stored on the memory 106.

[0284] Alternatively, executable code is stored in the memory 106, and the processor 105 executes the executable code to implement the functions of the foregoing model training device and data acquisition device respectively, thereby implementing the model training method. That is to say, instructions for executing the model training method are stored on the memory 106.

[0285] The communication interface 107 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or communication networks.

[0286] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0287] As Figure 10 shown, the computing device cluster includes at least one computing device 100. Instructions for executing the model training method may be stored in the memory 106 of one or more of the computing devices 100 in the computing device cluster.

[0288] In some possible implementation manners, partial instructions for executing the model training method may also be stored in the memory 106 of one or more of the computing devices 100 in the computing device cluster respectively. In other words, a combination of one or more computing devices 100 may jointly execute the instructions for executing the model training method.

[0289] It should be noted that the memories 106 in different computing devices 100 in the computing device cluster may store different instructions for respectively implementing partial functions of the model training device. That is to say, the instructions stored in the memories 106 of different computing devices 100 may implement the functions of one or more of the first acquisition module, the first training module, and the second acquisition module.

[0290] In some possible implementation manners, one or more computing devices in the computing device cluster may be connected through a network. Wherein, the network may be a wide area network or a local area network, etc. Figure 11 Shows a possible implementation manner. As Figure 11 shown, two computing devices 100A and 100B are connected through a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation manner, instructions for implementing the functions of the first acquisition module and the first training module are stored in the memory 106 of the computing device 100A. At the same time, instructions for implementing the functions of the first training module and the second acquisition module are stored in the memory 106 of the computing device 100B.

[0291] Figure 11 The connection mode between the computing device clusters shown can be considered that since the model training method provided in this application needs to store a large amount of second training data and perform multi-batch model training, it is considered to hand over the functions implemented by the second acquisition module and the second training module to the computing device 100B for execution.

[0292] It should be understood that Figure 11 the functions of the computing device 100A shown in can also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be completed by multiple computing devices 100.

[0293] The embodiments of this application also provide another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to Figure 10 and Figure 11 the connection mode of the described computing device cluster. The difference is that the same instructions for executing the model training method can be stored in the memory 106 of one or more computing devices 100 in this computing device cluster.

[0294] In some possible implementation manners, the memory 106 of one or more computing devices 100 in this computing device cluster can also store partial instructions for executing the model training method respectively. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the model training method.

[0295] It should be noted that the memories 106 in different computing devices 100 in the computing device cluster can store different instructions for implementing partial functions of the model training system. That is, the instructions stored in the memories 106 of different computing devices 100 can implement the functions of one or more of the model training device and the data acquisition device.

[0296] The embodiments of this application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it causes at least one computing device to execute the model training method.

[0297] Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive), etc. The computer-readable storage medium includes instructions that direct the computing device to execute the model training method.

[0298] In the embodiments of the present application, words such as "exemplarily" or "for example" are used to give examples, illustrations, or explanations. Any embodiment or design solution described as "exemplarily" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0299] In the embodiments of the present application, "at least one" mentioned refers to one or more, and "a plurality" refers to two or more. "At least one of the following" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c may represent: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, and c may be single or multiple. "And / or" describes the association relationship of associated objects and indicates that there can be three relationships. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B may be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0300] Also, unless otherwise stated, the ordinal numbers such as "first" and "second" used in the embodiments of the present application are used to distinguish multiple objects and are not used to limit the order, timing, priority, or importance of multiple objects. For example, the first device and the second device are only for ease of description and do not indicate differences in the structure, importance, etc. of the first device and the second device. In some embodiments, the first device and the second device may also be the same device.

[0301] As used in the above embodiments, depending on the context, the term "when..." may be interpreted to mean "if...", "after...", "in response to determining...", or "in response to detecting...". The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the concept and principles of the present application shall be included in the protection scope of the present application.

[0302] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk, an optical disc, etc.

Claims

1. A model training method, characterized in that, The method includes: Obtaining a plurality of first training data, where the first training data includes first three-dimensional face data and first two-dimensional face images; Training a neural network using the plurality of first training data to obtain a first face construction model. The first three-dimensional face data in the first training data is label data, and the first two-dimensional face images in the first training data are feature data. The first face construction model is used to generate corresponding three-dimensional face data according to the input two-dimensional face images; Obtaining a plurality of second training data, where the second training data includes second three-dimensional face data and second two-dimensional face images corresponding to the second three-dimensional face data, and the quality of the first training data is higher than that of the second training data; Training the first face construction model according to the plurality of second training data to obtain a second face construction model. The second three-dimensional face data in the second training data is label data, and the second two-dimensional face images in the second training data are feature data. The second face construction model is used to generate corresponding three-dimensional face data according to the input two-dimensional face images.

2. The method according to claim 1, characterized in that, The training the first face construction model according to the plurality of second training data to obtain a second face construction model includes: Inputting the plurality of second training data into the first face construction model for training by means of incremental learning to obtain the second face construction model.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Receiving a target two-dimensional face image; Inputting the target two-dimensional face image into the second face construction model to generate target three-dimensional face data corresponding to the target two-dimensional face image; Displaying the target three-dimensional face data; Receiving a first model correction instruction for adjusting the appearance of the target three-dimensional face data to obtain corrected target three-dimensional face data; Using the target two-dimensional face image and the corrected target three-dimensional face data as training data to train the second face construction model.

4. The method according to claim 1 or 2, characterized in that, The method further includes: Receiving a target two-dimensional face image; Inputting the target two-dimensional face image into the second face construction model to generate target three-dimensional face data corresponding to the target two-dimensional face image; Displaying the target three-dimensional face data; Receiving a second model correction instruction for evaluating the generation quality of the target three-dimensional face data. The second model correction instruction includes an evaluation direction and / or an evaluation level for the generation quality of the target three-dimensional face data; Training the second face construction model based on the second model correction instruction.

5. The method according to any one of claims 1-4, characterized in that, The obtaining a plurality of second training data includes: Obtaining a plurality of the second two-dimensional face images; Performing three-dimensional reconstruction on the plurality of second two-dimensional face images to generate the plurality of second three-dimensional face data respectively corresponding to the plurality of two-dimensional face images. The plurality of second two-dimensional face images and the second three-dimensional face data constitute the plurality of second training data.

6. The method according to claim 5, characterized in that, The obtaining a plurality of the second two-dimensional face images includes: Obtaining a plurality of initial two-dimensional face images; Input the multiple initial two-dimensional face images into a preset data evaluation model to obtain probability values respectively corresponding to the multiple initial two-dimensional face images. The probability value is used to characterize the matching degree between the initial two-dimensional face image and a two-dimensional face image that meets a preset first condition. The higher the probability value, the higher the matching degree. The preset first condition includes one or more of uniform illumination, a screen occlusion lower than a preset first threshold, and a frontal face in the image. Use the multiple initial two-dimensional face images with probability values higher than a preset second threshold as the multiple second two-dimensional face images.

7. The method according to any one of claims 1-6, characterized in that, The second face construction model includes an adversarial network model. The adversarial network model includes a face feature encoder, a generator, and a discriminator. The face feature encoder is used to generate a face feature vector according to a two-dimensional image. The generator is used to generate three-dimensional face data according to the face feature vector and random noise. The discriminator is used to judge the generated three-dimensional face data and the label data in the training data and output a judgment result.

8. The method according to any one of claims 1-7, characterized in that, The second three-dimensional face data in the second training data has undergone retopology processing, and the topological structure of the second three-dimensional face data is the same as that of the first three-dimensional face data in the first training data.

9. A model training device, characterized in that Including: A first acquisition module, configured to acquire multiple first training data, where the first training data includes first three-dimensional face data and first two-dimensional face images. A first training module, configured to train a neural network using the multiple first training data to obtain a first face construction model. The first three-dimensional face data in the first training data is label data, and the first two-dimensional face images in the first training data are feature data. The first face construction model is used to generate corresponding three-dimensional face data according to the input two-dimensional face image. A second acquisition module, configured to acquire multiple second training data. The second training data includes second three-dimensional face data and second two-dimensional face images corresponding to the second three-dimensional face data, and the quality of the first training data is higher than that of the second training data. A second training module, configured to train the first face construction model according to the multiple second training data to obtain a second face construction model. The second three-dimensional face data in the second training data is label data, and the second two-dimensional face images in the second training data are feature data. The second face construction model is used to generate corresponding three-dimensional face data according to the input two-dimensional face image.

10. The device according to claim 9, characterized in that, The second training module is specifically configured to: Input the multiple second training data into the first face construction model for training in an incremental learning manner to obtain the second face construction model.

11. The device according to claim 9 or 10, characterized in that, The device further includes: A first receiving unit, configured to receive a target two-dimensional face image. A first input module, configured to input the target two-dimensional face image into the second face construction model to generate target three-dimensional face data corresponding to the target two-dimensional face image. A first display module, configured to display the target three-dimensional face data. A second receiving module, configured to receive a first model correction instruction for adjusting the appearance of the target three-dimensional face data to obtain corrected target three-dimensional face data; A third training module, configured to use the target two-dimensional face image and the corrected target three-dimensional face data as training data to train the second face construction model.

12. The device according to claim 9 or 10, characterized in that, The apparatus further includes: A third receiving module, configured to receive a target two-dimensional face image; A second input module, configured to input the target two-dimensional face image into the second face construction model to generate target three-dimensional face data corresponding to the target two-dimensional face image; A second display module, configured to display the target three-dimensional face data; A fourth receiving module, configured to receive a second model correction instruction for evaluating the generation quality of the target three-dimensional face data, where the second model correction instruction includes an evaluation direction and / or an evaluation level of the generation quality of the target three-dimensional face data; Based on the second model correction instruction, train the second face construction model.

13. The device according to any one of claims 9-12, characterized in that, The second obtaining module is specifically configured to: Obtain a plurality of the second two-dimensional face images; Perform three-dimensional reconstruction on the plurality of second two-dimensional face images to generate the plurality of second three-dimensional face data respectively corresponding to the plurality of two-dimensional face images, where the plurality of second two-dimensional face images and the second three-dimensional face data form the plurality of second training data.

14. The device according to claim 13, characterized in that, The second obtaining module is specifically configured to: Obtain a plurality of initial two-dimensional face images; Input the plurality of initial two-dimensional face images into a preset data evaluation model to obtain probability values respectively corresponding to the plurality of initial two-dimensional face images, where the probability value is used to characterize the matching degree between the initial two-dimensional face image and a two-dimensional face image that meets a preset first condition, and the higher the probability value, the higher the matching degree; the preset first condition includes one or more of uniform illumination, a screen occlusion lower than a preset first threshold, and a frontal face image; Use the plurality of initial two-dimensional face images with probability values higher than a preset second threshold as the plurality of second two-dimensional face images.

15. A cluster of computing devices, characterized in that, Including at least one computing device, each computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-8.

16. A computer program product containing instructions, characterized in that, When the instructions are run by the computing device cluster, the computing device cluster executes the method according to any one of claims 1-8.

17. A computer-readable storage medium, characterized in that, Including computer program instructions, when the computer program instructions are executed by the computing device cluster, the computing device cluster executes the method according to any one of claims 1-8.

Images