A training method, device and storage medium for a three-dimensional object model

Abstract

The present application discloses a training method, device and storage medium for a three-dimensional object model, which can be applied to the field of maps. By obtaining unlabeled data; then performing self-supervisory training on a preset neural network based on the unlabeled data to obtain a first neural network; and obtaining labeled data; then performing supervised training on the first neural network based on the labeled data to obtain a second neural network; further, the unlabeled data and the labeled data are combined to obtain fused data, and the second neural network is trained to obtain a target neural network. Thus, an accurate reconstruction model training process is achieved. While learning the distribution of the object shape using unlabeled data, the projection parameters are decoupled through labeled data training, and the object flattening problem during the reconstruction process is alleviated, thereby improving the accuracy of reconstruction model training.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, device and storage medium for a three-dimensional object model. Background Art

[0002] With the rapid development of computer technology, face recognition has appeared in people's lives. Transactions and verifications can be performed through faces. However, how to generate a three-dimensional face model has become a difficult problem.

[0003] Generally, you can input an image into a designed network (such as a VGG network), use the network to predict the 3DMM coefficient, pose coefficient, and illumination coefficient, convert these coefficients into a 3D model and project them onto the image, and then calculate the gradient information based on the difference between the projected image and the original image (loss function). The gradient information adjusts the weights of the neural network, and the network is trained with a large number of images until it converges.

[0004] However, due to the differences between three-dimensional models and two-dimensional images, the recognition results obtained by the above method may be synthesized into flat results, affecting the accuracy of the reconstruction model training. Summary of the invention

[0005] In view of this, the present application provides a three-dimensional object model training method, which can effectively improve the accuracy of reconstruction model training.

[0006] In a first aspect, the present application provides a training method for a three-dimensional object model, which can be applied to a system or program including a model training function in a terminal device, and specifically includes:

[0007] Acquire unlabeled data, wherein the unlabeled data includes an unlabeled object image;

[0008] Performing self-supervisory training on a preset neural network based on the unlabeled object image to obtain a first neural network;

[0009] Acquire annotation data, wherein the annotation data includes an object image annotated with a reconstructed object;

[0010] Performing supervised training on the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network;

[0011] Fusing the unlabeled data with the labeled data to obtain fused data;

[0012] The second neural network is trained based on the fused data to obtain a target neural network, and the target neural network is used to perform projection reconstruction on the target object.

[0013] Optionally, in some possible implementations of the present application, the performing self-supervisory training on a preset neural network based on the unlabeled object image to obtain a first neural network includes:

[0014] Acquire a first projection image obtained based on projection parameters corresponding to the preset neural network;

[0015] determining a first loss parameter based on a difference in color information corresponding to the first projection image and the unlabeled object image;

[0016] Get preset key points;

[0017] Determine a second loss parameter based on the preset key point and the difference between the preset key point and the corresponding projection;

[0018] The preset neural network is self-supervised trained according to the first loss parameter and the second loss parameter to obtain the first neural network.

[0019] Optionally, in some possible implementations of the present application, the performing supervised training on the first neural network based on the object image labeled with the reconstructed object to obtain the second neural network includes:

[0020] Obtaining a reconstruction model based on projection parameters corresponding to the first neural network;

[0021] determining a third loss parameter based on a difference in vertex distance between the reconstructed model and the reconstructed object;

[0022] determining a fourth loss parameter based on a difference in vertex normals between the reconstructed model and the reconstructed object;

[0023] The second neural network is supervisedly trained according to the third loss parameter and the fourth loss parameter to obtain the second neural network.

[0024] Optionally, in some possible implementations of the present application, the method further includes:

[0025] Acquire a second projection image based on projection parameters corresponding to the first neural network;

[0026] determining a fifth loss parameter based on a difference in color information corresponding to the second projection image and the object image labeled with the reconstructed object;

[0027] Obtaining a preset key point to determine a sixth loss parameter based on the preset key point and a difference between a projection corresponding to the preset key point;

[0028] The parameters of the second neural network are adjusted according to the fifth loss parameter and the sixth loss parameter.

[0029] Optionally, in some possible implementations of the present application, the method further includes:

[0030] Get the key points of the model;

[0031] The reconstructed object including the annotations is aligned with the reconstructed model based on the model key points to update the reconstructed object.

[0032] Optionally, in some possible implementations of the present application, fusing the unlabeled data with the labeled data to obtain fused data includes:

[0033] determining a fusion ratio based on the annotation quality of the object image annotated with the reconstructed object;

[0034] The unlabeled data and the labeled data are fused according to the fusion ratio to obtain the fused data.

[0035] Optionally, in some possible implementations of the present application, the target object is a face model, and the number of the object images labeled with the reconstructed object is less than the number of the unlabeled object images.

[0036] A second aspect of the present application provides a three-dimensional object model training device, comprising:

[0037] An acquisition unit, configured to acquire unlabeled data, wherein the unlabeled data includes an unlabeled object image;

[0038] A training unit, configured to perform self-supervisory training on a preset neural network based on the unlabeled object image to obtain a first neural network;

[0039] The acquisition unit is further used to acquire annotation data, wherein the annotation data includes an object image annotated with a reconstructed object;

[0040] The training unit is further configured to perform supervised training on the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network;

[0041] A fusion unit, used for fusing the unlabeled data with the labeled data to obtain fused data;

[0042] The training unit is further used to train the second neural network based on the fusion data to obtain a target neural network, and the target neural network is used to perform projection reconstruction on the target object.

[0043] Optionally, in some possible implementations of the present application, the training unit is specifically used to obtain a first projection image based on projection parameters corresponding to the preset neural network;

[0044] The training unit is specifically configured to determine a first loss parameter based on a difference in color information corresponding to the first projection image and the unlabeled object image;

[0045] The training unit is specifically used to obtain preset key points;

[0046] The training unit is specifically used to determine a second loss parameter based on the preset key point and the difference between the projections corresponding to the preset key point;

[0047] The training unit is specifically used to perform self-supervised training on the preset neural network according to the first loss parameter and the second loss parameter to obtain the first neural network.

[0048] Optionally, in some possible implementations of the present application, the training unit is specifically used to obtain a reconstruction model based on the projection parameters corresponding to the first neural network;

[0049] The training unit is specifically configured to determine a third loss parameter based on the difference in vertex distance between the reconstructed model and the reconstructed object;

[0050] The training unit is specifically configured to determine a fourth loss parameter based on the difference in vertex normals between the reconstructed model and the reconstructed object;

[0051] The second neural network is supervisedly trained according to the third loss parameter and the fourth loss parameter to obtain the second neural network.

[0052] Optionally, in some possible implementations of the present application, the training unit is specifically used to obtain a second projection image based on the projection parameters corresponding to the first neural network;

[0053] The training unit is specifically configured to determine a fifth loss parameter based on a difference in color information corresponding to the second projection image and the object image labeled with the reconstructed object;

[0054] The training unit is specifically used to obtain a preset key point to determine a sixth loss parameter based on the preset key point and a difference between a projection corresponding to the preset key point;

[0055] The training unit is specifically used to adjust the parameters of the second neural network according to the fifth loss parameter and the sixth loss parameter.

[0056] Optionally, in some possible implementations of the present application, the training unit is specifically used to obtain key points of the model;

[0057] The training unit is specifically used to determine first position information of the model key point in the reconstructed object including the annotation;

[0058] The training unit is specifically used to determine the second position information of the model key point in the reconstructed model;

[0059] The training unit is specifically configured to approximately adjust the first position information toward the second position information so that the reconstructed object including the annotation is aligned with the reconstructed model, and update the reconstructed object.

[0060] Optionally, in some possible implementations of the present application, the fusion unit is specifically configured to determine a fusion ratio based on a labeling quality of the object image labeled with the reconstructed object;

[0061] The fusion unit is specifically configured to fuse the unlabeled data and the labeled data according to the fusion ratio to obtain the fused data.

[0062] The third aspect of the present application provides a computer device, comprising: a memory, a processor and a bus system; the memory is used to store program code; the processor is used to execute the training method of the model described in the first aspect or any one of the first aspects according to the instructions in the program code.

[0063] The fourth aspect of the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the training method of the model described in the first aspect or any one of the first aspects.

[0064] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the model provided in the first aspect or various optional implementations of the first aspect.

[0065] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0066] By acquiring unlabeled data, which includes unlabeled object images; then performing self-supervisory training on a preset neural network based on the unlabeled object images to obtain a first neural network; and acquiring labeled data, which includes an object image labeled with a reconstructed object; then performing supervised training on the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network; further fusing the unlabeled data and the labeled data to obtain fused data; and then training the second neural network based on the fused data to obtain a target neural network, which is used to project and reconstruct the target object. Thus, an accurate reconstruction model training process is achieved. While learning the distribution of the object shape using unlabeled data, the projection parameters are decoupled through labeled data training, and the object flattening problem in the reconstruction process is alleviated, thereby improving the accuracy of reconstruction model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0068] Figure 1 A diagram of the network architecture that runs the model's training system;

[0069] Figure 2 A flow chart of a three-dimensional object model training process provided in an embodiment of the present application;

[0070] Figure 3 A flowchart of a three-dimensional object model training method provided in an embodiment of the present application;

[0071] Figure 4 A schematic diagram of a scenario of a three-dimensional object model training method provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of a scenario of another three-dimensional object model training method provided in an embodiment of the present application;

[0073] Figure 6 A flowchart of another three-dimensional object model training method provided in an embodiment of the present application;

[0074] Figure 7 A schematic diagram of the structure of a three-dimensional object model training device provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0076] Fig. 9 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The embodiment of the present application provides a training method and related device for a three-dimensional object model, which can be applied to a system or program in a terminal device that includes a model training function, by obtaining unlabeled data, which includes an unlabeled object image; then performing self-supervisory training on a preset neural network based on the unlabeled object image to obtain a first neural network; and obtaining labeled data, which includes an object image labeled with a reconstructed object; then performing supervised training on the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network; further fusing the unlabeled data and the labeled data to obtain fused data; and then training the second neural network based on the fused data to obtain a target neural network, which is used to project and reconstruct the target object. Thus, an accurate training process for the reconstruction model is achieved, because while learning the distribution of the object shape using unlabeled data, the projection parameters are decoupled through labeled data training, and the problem of object flattening during the reconstruction process is alleviated, thereby improving the accuracy of the reconstruction model training.

[0078] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0079] First, some terms that may appear in the embodiments of the present application are explained.

[0080] 3D Deformable Face Model (3DMM): is a general 3D face parametric model that uses a fixed number of points to represent a face. Its core idea is that faces can be matched one-to-one in 3D space and can be derived from the weighted linear addition of many other faces by orthogonal basis.

[0081] 2D facial key point detection: The purpose is to automatically locate a set of predefined facial reference points (such as eye corners and mouth corners).

[0082] Iterative Closest Points (ICP): The ICP algorithm includes corresponding point search and pose solution. Its purpose is to find the matching relationship between point sets, and the result of the solution is the translation and rotation between the two point sets.

[0083] It should be understood that the model training method provided in the present application can be applied to a system or program in a terminal device that includes a model training function, such as face recognition. Specifically, the model training system can be run on Figure 1 In the network architecture shown in Figure 1 As shown in FIG. 1 , it is a network architecture diagram of the model training system. As can be seen from the figure, the model training system can provide a model training process with multiple information sources, that is, obtaining object information through interactive operations on the terminal side, and collecting and training object information on the server; it can be understood that Figure 1 A variety of terminal devices are shown in FIG. 1 . The terminal devices may be computer devices. In actual scenarios, more or fewer types of terminal devices may participate in the model training process. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1 One server is shown in the figure, but in actual scenarios, multiple servers may be involved, especially in scenarios of multi-model training interaction. The specific number of servers depends on the actual scenario.

[0084] In this embodiment, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart voice interaction device, smart home appliance, car terminal, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and the terminal and the server can be connected to form a blockchain network, which is not limited in this application.

[0085] It can be understood that the training system of the above-mentioned model can run on a personal mobile terminal, for example: as an application such as face recognition, it can also run on a server, and it can also run on a third-party device to provide model training to obtain the training processing results of the model of the information source; the specific model training system can be run in the above-mentioned device in the form of a program, and can also be run as a system component in the above-mentioned device, and can also be used as a cloud service program. This embodiment can be applied to cloud technology, autonomous driving and other scenarios. The specific operation mode depends on the actual scenario and is not limited here.

[0086] With the rapid development of computer technology, face recognition has appeared in people's lives. Transactions and verifications can be performed through faces. However, how to generate a three-dimensional face model has become a difficult problem.

[0087] Generally, you can input an image into a designed network (such as a VGG network), use the network to predict the 3DMM coefficient, pose coefficient, and illumination coefficient, convert these coefficients into a 3D model and project them onto the image, and then calculate the gradient information based on the difference between the projected image and the original image (loss function). The gradient information adjusts the weights of the neural network, and the network is trained with a large number of images until it converges.

[0088] However, due to the differences between three-dimensional models and two-dimensional images, the recognition results obtained by the above method may be synthesized into flat results, affecting the accuracy of the reconstruction model training.

[0089] In addition, a fully supervised solution can also be used, that is, inputting pictures into a designed network (such as a VGG network), the network predicts the 3DMM coefficients, and directly synthesizing a 3D face model based on the predicted 3DMM coefficients, calculating the difference between the reconstructed 3D face model and the labeled 3D face model, and calculating the gradient information. The gradient information adjusts the weights of the neural network. This method requires a large amount of labeled data. However, if there is too little labeled data to be collected, it is easy to overfit, resulting in performance degradation; if labeling is performed, the cost is very high due to the large amount of data; if a synthetic data set solution is used, the accuracy is too low.

[0090] In order to solve the above problems, this application proposes a training method for a three-dimensional object model. Figure 2 In the training process framework of the model shown, Figure 2As shown, it is a flow chart of the training of a three-dimensional object model provided by an embodiment of the present application. The user obtains object information (such as face information) through interactive operations of the terminal, and then trains the model on the server, that is, based on the deep learning method, a convolutional neural network model is trained, a face picture is input, and the reconstruction result of the 3D face is output. In this embodiment, a large number of face pictures are first collected, and a convolutional neural network model is pre-trained in a self-supervised manner, so that it can learn the distribution of faces through a large number of face pictures and obtain reasonable results; then, open source annotation data (that is, each annotated picture has a corresponding 3D face model) is collected, which is usually a very small amount, and the convolutional neural network model is continued to be trained as supervision, and the shape, posture, texture and illumination of the reconstructed model are decoupled as much as possible in this process; finally, the collected unannotated face pictures and annotated pictures are fused and supervised together to train the final network model.

[0091] This embodiment can be applied to the Intelligent Traffic System (ITS), also known as the Intelligent Transportation System, which effectively and comprehensively applies advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthens the connection between vehicles, roads and users, and thus forms a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy.

[0092] It can be understood that the method provided by the present application can be written as a program to serve as a processing logic in a hardware system, or as a training device for a three-dimensional object model, and the above processing logic is implemented in an integrated or external manner. As an implementation method, the training device of the model obtains unlabeled data, which includes unlabeled object images; then performs self-supervisory training on a preset neural network based on the unlabeled object images to obtain a first neural network; and obtains labeled data, which includes an object image labeled with a reconstructed object; then supervises the training of the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network; further fuses the unlabeled data and the labeled data to obtain fused data; and then trains the second neural network based on the fused data to obtain a target neural network, which is used to project and reconstruct the target object. Thereby, an accurate training process of the reconstruction model is realized. While learning the distribution of the object shape using unlabeled data, the projection parameters are decoupled through labeled data training, and the problem of object flatness during the reconstruction process is alleviated, thereby improving the accuracy of the reconstruction model training.

[0093] It is understandable that in the specific implementation of this application, related data such as user information (including but not limited to facial information) is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0094] The solution provided in the embodiments of the present application involves the deep learning technology of artificial intelligence, which is specifically described by the following embodiments:

[0095] Combined with the above process architecture, the following will introduce the training method of the model in this application. Please refer to Figure 3 , Figure 3 A flowchart of a training method for a three-dimensional object model provided in an embodiment of the present application. The management method can be executed by a server or a terminal. The embodiment of the present application at least includes the following steps:

[0096] 301. Obtain unlabeled data.

[0097] In this embodiment, the unlabeled data includes unlabeled object images, such as collecting a large number of unlabeled face images; wherein the object can be a face or other three-dimensional model, and the face is taken as an example for explanation here.

[0098] This embodiment can be applied to the reconstruction of the faces of characters in 3D games / 3D film and television works / short videos. The 3D face model reconstruction technology based on 3DMM is relatively mature, and many entertainment products provide face photo character shaping functions that mostly use this technology directly or indirectly. This embodiment can effectively improve the effect of the reconstructed 3D face model based on a single face photo input, making it more similar to the original image.

[0099] 302. Perform self-supervisory training on a preset neural network based on the unlabeled object image to obtain a first neural network.

[0100] In this embodiment, self-supervised training is performed on a preset neural network based on unlabeled object images, that is, a convolutional neural network model is pre-trained in a self-supervised manner, so that it can learn the distribution of faces through a large number of face images, obtain reasonable results, and give full play to the advantages of self-supervised training.

[0101] Specifically, for training the first neural network, a first projection image obtained based on projection parameters corresponding to a preset neural network can be first obtained; a first loss parameter can then be determined based on the difference in color information corresponding to the first projection image and the unlabeled object image; and preset key points can be obtained; a second loss parameter can be further determined based on the preset key points and the difference in projections corresponding to the preset key points; and the preset neural network can be self-supervised trained according to the first loss parameter and the second loss parameter to obtain the first neural network.

[0102] Among them, the first loss parameter L1 is photo loss (difference in color information), and the specific formula is as follows:

[0103]

[0104] Among them, R(h(f), p, s) represents the image projected according to the predicted 3DMM coefficient f, pose coefficient p and illumination coefficient s, h(f) is the 3D face model reconstructed according to the 3DMM coefficients, N represents the total number of image pixels, and i represents the i-th pixel.

[0105] In addition, the second loss parameter L2 is the difference between the key point projection result predefined on the 3D model and the key point detected in the original image. The specific formula is as follows:

[0106]

[0107] Among them, Vi is the i-th key point detected on the 2D image I, Mi is the i-th predefined key point on the 3D model h(f), N is the total number of key points, and T(*) is the projection function (projected according to the pose coefficient p and the 3D model).

[0108] Furthermore, by adding up the various loss functions, we can get the training loss for the preset neural network.

[0109] 303. Obtain labeled data.

[0110] In this embodiment, the labeled data includes object images labeled with the reconstructed object; wherein, the number of object images labeled with the reconstructed object is less than the number of unlabeled object images, that is, this embodiment has a small amount of labeled data and a large amount of unlabeled data, thereby saving training costs while ensuring the training effect.

[0111] It can be understood that the order of magnitude corresponding to the number of images of object images with reconstructed objects labeled is smaller than the order of magnitude corresponding to the number of images of unlabeled object images. For example, the order of magnitude of the number of images of object images with reconstructed objects labeled is in the tens, while the order of magnitude corresponding to the number of images of unlabeled object images is in the thousands, thereby facilitating data collection.

[0112] Specifically, the annotated data can be annotated in real time by relevant personnel, downloaded from a public database, or extracted based on historical identification information. The specific data source depends on the actual scenario.

[0113] 304. Perform supervised training on the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network.

[0114] In this embodiment, open source annotation data is collected to continue training the neural network, and the decoupling of the projection parameters is achieved by inputting pictures and outputting adjusted projection parameters 3DMM coefficient f, pose coefficient p and illumination coefficient s.

[0115] Specifically, the process of supervised training of the first neural network can first obtain a reconstructed model based on the projection parameters corresponding to the first neural network; then determine the third loss parameter based on the difference in vertex distance between the reconstructed model and the reconstructed object; and determine the fourth loss parameter based on the difference in vertex normals between the reconstructed model and the reconstructed object; and then supervised training the second neural network according to the third loss parameter and the fourth loss parameter to obtain the second neural network.

[0116] Among them, the third loss parameter L3 is the vertex distance difference, and the specific formula is as follows:

[0117]

[0118] Among them, h(f)i represents the i-th 3D vertex of the reconstructed 3D face model h(f), hi' represents the i-th 3D vertex of the labeled 3D face model h', and N is the total number of 3D vertices.

[0119] In addition, for the fourth loss parameter L4, which is the vertex normal difference, cosine similarity can be used to measure, solve each 3D vertex normal, and then calculate the loss. The specific formula is as follows:

[0120]

[0121] Wherein, Di represents the normal of the i-th 3D vertex of the reconstructed 3D face model h(f), Di' represents the normal of the i-th 3D vertex of the labeled 3D face model h', and N is the total number of 3D vertices.

[0122] It should be noted that before calculating L3 and L4, the annotated 3D face model h' can be first aligned to the reconstructed 3D face model h(f) according to the predefined key points. Here, the existing ICP algorithm is used, and this is the case for each iteration. That is, first obtain the model key points; then determine the first position information of the model key points in the reconstructed object containing the annotations; and determine the second position information of the model key points in the reconstructed model; and then adjust the first position information approximately to the second position information, so that the reconstructed object containing the annotations is aligned with the reconstructed model, and update the reconstructed object to update the reconstructed object, thereby improving the accuracy of the loss parameter calculation.

[0123] Furthermore, the loss parameters can also be color information and key point loss. Specifically, a second projection image obtained based on the projection parameters corresponding to the first neural network can be first obtained; then a fifth loss parameter can be determined based on the difference in color information corresponding to the second projection image and the object image labeled with the reconstructed object; further, preset key points can be obtained to determine a sixth loss parameter based on the preset key points and the difference in projections corresponding to the preset key points; and the parameters of the second neural network can be adjusted according to the fifth loss parameter and the sixth loss parameter.

[0124] Among them, the fifth loss parameter L5 is photo loss (difference in color information), and the specific formula is as follows:

[0125]

[0126] Among them, R(h', p, s) represents the image projected according to the annotated 3D face model h', pose coefficient p and illumination coefficient s, N represents the total number of image pixels, and i represents the i-th pixel.

[0127] In addition, the sixth loss parameter L6 is the difference between the predefined key point projection result on the annotated 3D face model h' and the key point detected in the original image. The specific formula is as follows:

[0128]

[0129] Among them, Vi is the i-th key point detected on the 2D image I, Mi' is the i-th predefined key point on the 3D model h', N is the total number of key points, and T(*) is the projection function (projection based on pose coefficient p and 3D model).

[0130] Furthermore, by adding up the various loss functions, we can get the training loss for the preset neural network.

[0131] On the one hand, this embodiment uses 3DMM coefficients to reconstruct a 3D face model h(f), calculates the difference between the reconstructed 3D face model h(f) and the annotated 3D face model h', takes the distance difference L3 between the 3D vertices and the angle difference L4 between the normals of the 3D vertices, and improves the problem of flat face through this constraint. On the other hand, the annotated 3D face model h' is projected onto the image based on the predicted pose coefficient p and illumination coefficient s, and the color information difference L5 and the key point difference L6 are calculated. Through this constraint, the prediction of the pose coefficient and the illumination coefficient is further refined and effectively decoupled. Therefore, step 304 must continue training on the basis of the pre-training in step 302 to effectively reduce the difficulty of training.

[0132] Below, we will explain the process of training a convolutional neural network to predict 3DMM coefficients using a scenario diagram. Figure 4 As shown, Figure 4 A scene diagram of a training method for a three-dimensional object model provided in an embodiment of the present application; the figure shows that a large number of face pictures are collected, and a convolutional neural network model is pre-trained in a self-supervised manner, so that it can learn the distribution of faces through a large number of face pictures, obtain reasonable results, and give full play to the advantages of self-supervised training; then collect open source annotation data to continue training the neural network, on the one hand, calculate the difference between the reconstructed 3D face model and the annotated 3D face model, take the distance between the 3D vertices and the angle between the normals of the 3D vertices, and improve the problem of flat faces through this constraint. On the other hand, the annotated 3D face model is projected onto the picture according to the predicted pose coefficient and illumination coefficient, and the color information difference and key point difference are calculated. Through this constraint, the prediction of the pose coefficient and illumination coefficient is further improved, and it is effectively decoupled.

[0133] 305. Fuse the unlabeled data with the labeled data to obtain fused data.

[0134] In this embodiment, the purpose of fusion is to prevent the training of the first two steps from falling into the local optimal result so as to perform further training.

[0135] Specifically, the fusion process can be carried out according to a certain ratio, for example, randomly extracting unlabeled images and labeled images, and fusing them in a 1:1 ratio for training (for example, 16 unlabeled images and 16 labeled images). This is just an example, and the actual fusion ratio can be 2:1, 3:1, etc.

[0136] In addition, the fusion ratio can also be adjusted, that is, the fusion ratio is first determined based on the annotation quality of the object image annotated with the reconstructed object; then the unannotated data and the annotated data are fused according to the fusion ratio to obtain the fused data, thereby improving the quality of the fused data.

[0137] 306. Train the second neural network based on the fused data to obtain a target neural network.

[0138] In this embodiment, the target object is a human face model, and the corresponding target neural network is used to perform projection reconstruction on the target object, that is, to perform projection reconstruction on the human face model.

[0139] Specifically, the process of training the second neural network based on the fused data is to simultaneously predict the 3DMM coefficient f, pose coefficient p, and illumination coefficient s of all input images, calculate the loss function and gradient, and adjust the network weights until convergence. The loss function of the unlabeled image takes the color information difference L1 and the key point difference L2; the loss function of the labeled image takes the 3D vertex distance difference L3, the 3D vertex normal difference L4, the color difference L5 of the projected image, and the key point difference L6, and finally all the loss functions are added.

[0140] It is understandable that the motivation of fusion is to prevent the training of the first two steps from falling into local optimal results. Training based on the first two steps can effectively reduce the difficulty of training on the one hand; on the other hand, it can alleviate the problem of data imbalance and prevent the network from learning mainly in a self-supervised manner. Introducing labeled data training while conducting self-supervised training.

[0141] Next, a scenario is used to illustrate the direct iterative improvement of the 3DMM coefficients in an optimized manner, such as Figure 5 As shown, Figure 5 A scene diagram of another method for training a three-dimensional object model provided in an embodiment of the present application; that is, randomly extracting unlabeled images and labeled images, merging them in a 1:1 ratio (for example, 16 unlabeled images and 16 labeled images), and all 3DMM coefficients, pose coefficients, and illumination coefficients, and calculating the loss function and gradient, and adjusting the network weights until convergence. The loss function of the unlabeled image takes the color information difference and the key point difference; the loss function of the labeled image takes the 3D vertex distance difference, the 3D vertex normal difference, the color difference of the projected image, and the key point difference. The problem of data imbalance can be alleviated, and labeled data can be introduced at the same time as self-supervised training for simultaneous supervised training.

[0142] This embodiment makes full use of the advantages of self-supervised training and the small amount of annotated data available, effectively learns the distribution of face shapes from a large number of pictures, decouples the 3DMM coefficient, pose coefficient and illumination coefficient as much as possible, and alleviates the common face flatness problem of single photo reconstruction. Since no additional data collection (annotation) is required, the cost is effectively reduced while improving the reconstruction of photo faces.

[0143] Next, the training process of step 302, step 304, and step 306 is described. The training process of each step is similar. The specific process of training the convolutional neural network is as follows:

[0144] Input: collected unlabeled images I, collected labeled images I' and corresponding labeled 3D face data h';

[0145] Output: Convolutional neural network weight w;

[0146] process:

[0147] 1. Initialize the convolutional neural network weight w;

[0148] 2. Loop N steps (1)-(4):

[0149] (1) Randomly select an image and input it into the convolutional neural network, outputting the 3DMM parameter f; pose coefficient p; illumination coefficient s;

[0150] (2) Get the projection image based on the parameters or ;

[0151] (3) Calculate the objective function L at this time;

[0152] (4) Use optimizer F to update w

[0153] The optimizer F can use the Adam optimizer. Represents the gradient of the objective function L for the network weight w.

[0154] This embodiment is based on a deep learning method, which can give full play to the advantages of self-supervised training and make full use of the small amount of annotated data available. It can effectively learn the distribution of face shapes from a large number of pictures, while trying to decouple the 3DMM coefficient, pose coefficient and illumination coefficient, and alleviate the common face flatness problem of single photo reconstruction. Since no additional data collection (annotation) is required, the cost is effectively reduced while improving the reconstruction of photo faces. It can be applied to the reconstruction of faces of characters in 3D games / 3D film and television works / short videos, so that the effect of the 3D face model reconstructed from the face photo is more similar to the original image.

[0155] In combination with the above embodiments, it can be known that by obtaining unlabeled data, the unlabeled data includes unlabeled object images; then self-supervising training is performed on a preset neural network based on the unlabeled object images to obtain a first neural network; and labeled data is obtained, the labeled data includes an object image labeled with a reconstructed object; then supervised training is performed on the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network; the unlabeled data and the labeled data are further fused to obtain fused data; and then the second neural network is trained based on the fused data to obtain a target neural network, which is used to project and reconstruct the target object. Thus, an accurate reconstruction model training process is achieved. While learning the distribution of the object shape using unlabeled data, the projection parameters are decoupled through labeled data training, and the object flattening problem during the reconstruction process is alleviated, thereby improving the accuracy of reconstruction model training.

[0156] The above embodiment introduces the training process of the reconstruction model, and in the actual process, dynamic adjustment can also be performed. The following describes this scenario. Figure 6 , Figure 6 A flowchart of another method for training a three-dimensional object model provided in an embodiment of the present application, wherein the embodiment of the present application at least comprises the following steps:

[0157] 601. Obtain a target face image.

[0158] In this embodiment, the target facial image may be a facial image collected in real time or a facial image in a database, and the specific data source depends on the actual scenario.

[0159] 602. Input the target face image into the target neural network to obtain a target reconstructed image.

[0160] In this embodiment, the target neural network is the network trained by steps 301-306. The specific training process can be referred to and will not be described here.

[0161] 603. Evaluate the quality of the target reconstructed image to adjust the fusion ratio.

[0162] In this embodiment, the quality of the target reconstructed image can be evaluated by the similarity between the target reconstructed image and the original image. The higher the similarity, the better the quality. If the similarity is lower than a threshold (eg, 0.8), the fusion ratio needs to be adjusted, such as increasing the ratio of the labeled image.

[0163] 604. Train the target neural network based on the adjusted fusion image.

[0164] In this embodiment, by dynamically adjusting the fused image, the target neural network can dynamically adapt to different face models, thereby improving the reconstruction accuracy of the target neural network.

[0165] This embodiment is based on a deep learning method, which can give full play to the advantages of self-supervised training and make full use of the small amount of annotated data available, effectively learn the distribution of face shapes from a large number of pictures, and try to decouple the 3DMM coefficient, pose coefficient and illumination coefficient, and alleviate the common face flatness problem of single photo reconstruction. Since no additional data collection (annotation) is required, the accuracy of photo face reconstruction is improved while the cost is effectively reduced; and the accuracy of photo face reconstruction is further improved through dynamic parameter adjustment.

[0166] In order to better implement the above solution of the embodiment of the present application, the following also provides related devices for implementing the above solution. Figure 7 , Figure 7 A schematic diagram of the structure of a three-dimensional object model training device provided in an embodiment of the present application, wherein the model training device 700 includes:

[0167] An acquisition unit 701 is used to acquire unlabeled data, where the unlabeled data includes an unlabeled object image;

[0168] A training unit 702, configured to perform self-supervisory training on a preset neural network based on the unlabeled object image to obtain a first neural network;

[0169] The acquisition unit 701 is further configured to acquire annotation data, wherein the annotation data includes an object image annotated with a reconstructed object;

[0170] The training unit 702 is further configured to perform supervised training on the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network;

[0171] A fusion unit 703, configured to fuse the unlabeled data with the labeled data to obtain fused data;

[0172] The training unit 702 is further used to train the second neural network based on the fusion data to obtain a target neural network, and the target neural network is used to perform projection reconstruction on the target object.

[0173] Optionally, in some possible implementations of the present application, the training unit 702 is specifically configured to obtain a first projection image based on projection parameters corresponding to the preset neural network;

[0174] The training unit 702 is specifically configured to determine a first loss parameter based on a difference in color information corresponding to the first projection image and the unlabeled object image;

[0175] The training unit 702 is specifically used to obtain preset key points;

[0176] The training unit 702 is specifically configured to determine a second loss parameter based on the preset key point and a difference between the preset key point and the projection corresponding to the preset key point;

[0177] The training unit 702 is specifically configured to perform self-supervised training on the preset neural network according to the first loss parameter and the second loss parameter to obtain the first neural network.

[0178] Optionally, in some possible implementations of the present application, the training unit 702 is specifically used to obtain a reconstruction model based on the projection parameters corresponding to the first neural network;

[0179] The training unit 702 is specifically configured to determine a third loss parameter based on the difference in vertex distance between the reconstructed model and the reconstructed object;

[0180] The training unit 702 is specifically configured to determine a fourth loss parameter based on the difference in vertex normals between the reconstructed model and the reconstructed object;

[0181] The second neural network is supervisedly trained according to the third loss parameter and the fourth loss parameter to obtain the second neural network.

[0182] Optionally, in some possible implementations of the present application, the training unit 702 is specifically configured to obtain a second projection image based on the projection parameters corresponding to the first neural network;

[0183] The training unit 702 is specifically configured to determine a fifth loss parameter based on a difference in color information corresponding to the second projection image and the object image labeled with the reconstructed object;

[0184] The training unit 702 is specifically used to obtain a preset key point to determine a sixth loss parameter based on the preset key point and a difference between the preset key point and a projection corresponding to the preset key point;

[0185] The training unit 702 is specifically configured to adjust parameters of the second neural network according to the fifth loss parameter and the sixth loss parameter.

[0186] Optionally, in some possible implementations of the present application, the training unit 702 is specifically used to obtain model key points;

[0187] The training unit 702 is specifically configured to determine first position information of the model key point in the reconstructed object including the annotation;

[0188] The training unit 702 is specifically used to determine the second position information of the model key point in the reconstructed model;

[0189] The training unit 702 is specifically configured to approximately adjust the first position information toward the second position information so that the reconstructed object including the annotation is aligned with the reconstructed model, and update the reconstructed object.

[0190] Optionally, in some possible implementations of the present application, the fusion unit 703 is specifically configured to determine a fusion ratio based on the annotation quality of the object image annotated with the reconstructed object;

[0191] The fusion unit 703 is specifically configured to fuse the unlabeled data and the labeled data according to the fusion ratio to obtain the fused data.

[0192] By acquiring unlabeled data, which includes unlabeled object images; then performing self-supervisory training on a preset neural network based on the unlabeled object images to obtain a first neural network; and acquiring labeled data, which includes an object image labeled with a reconstructed object; then performing supervised training on the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network; further fusing the unlabeled data and the labeled data to obtain fused data; and then training the second neural network based on the fused data to obtain a target neural network, which is used to project and reconstruct the target object. Thus, an accurate reconstruction model training process is achieved. While learning the distribution of the object shape using unlabeled data, the projection parameters are decoupled through labeled data training, and the object flattening problem in the reconstruction process is alleviated, thereby improving the accuracy of reconstruction model training.

[0193] The present application also provides a terminal device, such as Figure 8 The figure is a schematic diagram of the structure of another terminal device provided by the embodiment of the present application. For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, etc., taking the mobile phone as an example:

[0194] Figure 8FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 8 The mobile phone includes: a radio frequency (RF) circuit 810, a memory 820, an input unit 830, a display unit 840, a sensor 850, an audio circuit 860, a wireless fidelity (WiFi) module 870, a processor 880, and a power supply 890. Those skilled in the art will appreciate that Figure 8 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0195] Combine the following Figure 8 A detailed introduction to the various components of the mobile phone:

[0196] The RF circuit 810 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 880 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 810 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 810 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0197] The memory 820 can be used to store software programs and modules. The processor 880 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 820. The memory 820 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 820 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0198] The input unit 830 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 830 may include a touch panel 831 and other input devices 832. The touch panel 831, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 831 or near the touch panel 831 using any suitable object or accessory such as a finger, stylus, etc., and the air touch operation within a certain range on the touch panel 831), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 831 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 880, and can receive and execute the command sent by the processor 880. In addition, the touch panel 831 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 831, the input unit 830 may also include other input devices 832. Specifically, the other input devices 832 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0199] The display unit 840 may be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 840 may include a display panel 841. Optionally, the display panel 841 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 831 may cover the display panel 841. When the touch panel 831 detects a touch operation on or near it, it is transmitted to the processor 880 to determine the type of touch event. Subsequently, the processor 880 provides a corresponding visual output on the display panel 841 according to the type of touch event. Although in Figure 8 In the embodiment, the touch panel 831 and the display panel 841 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 831 and the display panel 841 can be integrated to realize the input and output functions of the mobile phone.

[0200] The mobile phone may also include at least one sensor 850, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 841 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 841 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; As for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0201] The audio circuit 860, the speaker 861, and the microphone 862 can provide an audio interface between the user and the mobile phone. The audio circuit 860 can transmit the received audio data to the speaker 861 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 862 converts the collected sound signal into an electrical signal, which is received by the audio circuit 860 and converted into audio data, and then the audio data is output to the processor 880 for processing, and then sent to another mobile phone through the RF circuit 810, or the audio data is output to the memory 820 for further processing.

[0202] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 870. It provides users with wireless broadband Internet access. Figure 8A WiFi module 870 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the embodiment.

[0203] The processor 880 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 820 and calling data stored in the memory 820, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 880 may include one or more processing units; optionally, the processor 880 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 880.

[0204] The mobile phone also includes a power supply 890 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 880 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0205] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0206] In the embodiment of the present application, the processor 880 included in the terminal also has the function of executing each step of the above-mentioned page processing method.

[0207] The present application also provides a server. Fig. 9 , Fig. 9 : is a structural diagram of a server provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 922 (for example, one or more processors) and memory 932, and one or more storage media 930 (for example, one or more mass storage devices) storing application programs 942 or data 944. Among them, the memory 932 and the storage medium 930 can be short-term storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 922 can be configured to communicate with the storage medium 930 and execute a series of instruction operations in the storage medium 930 on the server 900.

[0208] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems 941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0209] The steps performed by the management device in the above embodiment can be based on the Fig. 9 The server structure shown.

[0210] In an embodiment of the present application, a computer-readable storage medium is also provided, in which a model training instruction is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned Figures 3 to 6 The illustrated embodiment describes the steps performed by the training device of the model in the method.

[0211] The present application also provides a computer program product including a model training instruction, which, when executed on a computer, enables the computer to execute the above-mentioned Figures 3 to 6 The illustrated embodiment describes the steps performed by the training device of the model in the method.

[0212] The present application also provides a three-dimensional object model training system, and the model training system may include Figure 7 A training device for the model in the described embodiments, or Figure 8 The terminal device in the described embodiment, or Fig. 9 The server described.

[0213] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0214] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0215] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0216] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0217] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a model training device, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.

[0218] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a three-dimensional object model, characterized in that: include: Acquire unlabeled data, wherein the unlabeled data includes an unlabeled object image; Performing self-supervisory training on a preset neural network based on the unlabeled object image to obtain a first neural network; Acquire annotation data, wherein the annotation data includes an object image annotated with a reconstructed object; The first neural network is supervisedly trained based on the object image labeled with the reconstructed object to obtain the second neural network, which includes: obtaining a reconstruction model obtained based on the projection parameters corresponding to the first neural network, the projection parameters including 3DMM coefficients, Pose coefficients and illumination coefficients; determining a third loss parameter based on the difference in vertex distances between the reconstruction model and the reconstructed object; determining a fourth loss parameter based on the difference in vertex normals between the reconstruction model and the reconstructed object; supervising the second neural network according to the third loss parameter and the fourth loss parameter to obtain the second neural network; obtaining a second projection image obtained based on the projection parameters corresponding to the first neural network; determining a fifth loss parameter based on the difference in color information corresponding to the second projection image and the object image labeled with the reconstructed object; obtaining preset key points to determine a sixth loss parameter based on the preset key points and the difference in projections corresponding to the preset key points; adjusting parameters of the second neural network according to the fifth loss parameter and the sixth loss parameter; Fusing the unlabeled data with the labeled data to obtain fused data; The second neural network is trained based on the fused data to obtain a target neural network, and the target neural network is used to perform projection reconstruction on the target object.

2. The method according to claim 1, characterized in that The self-supervised training of a preset neural network based on the unlabeled object image to obtain a first neural network includes: Acquire a first projection image obtained based on projection parameters corresponding to the preset neural network; determining a first loss parameter based on a difference in color information corresponding to the first projection image and the unlabeled object image; Get preset key points; Determine a second loss parameter based on the preset key point and the difference between the preset key point and the corresponding projection; The preset neural network is self-supervised trained according to the first loss parameter and the second loss parameter to obtain the first neural network.

3. The method according to claim 1, characterized in that The method further comprises: Get the key points of the model; Determining first position information of the model key point in the reconstructed object including the annotation; Determine second position information of the model key point in the reconstructed model; The first position information is approximately adjusted toward the second position information so that the reconstructed object including the annotation is aligned with the reconstructed model, and the reconstructed object is updated.

4. The method according to claim 1, characterized in that The fusing the unlabeled data and the labeled data to obtain fused data includes: determining a fusion ratio based on the annotation quality of the object image annotated with the reconstructed object; The unlabeled data and the labeled data are fused according to the fusion ratio to obtain the fused data.

5. The method according to claim 1, characterized in that: The target object is a face model, and the order of magnitude corresponding to the number of the object images with the reconstructed object labeled is smaller than the order of magnitude corresponding to the number of the object images without the label.

6. A training device for a three-dimensional object model, characterized in that: include: An acquisition unit, configured to acquire unlabeled data, wherein the unlabeled data includes an unlabeled object image; A training unit, configured to perform self-supervisory training on a preset neural network based on the unlabeled object image to obtain a first neural network; The acquisition unit is further used to acquire annotation data, wherein the annotation data includes an object image annotated with a reconstructed object; The training unit is also used to perform supervised training on the first neural network based on the object image labeled with the reconstructed object to obtain a second neural network, which includes: obtaining a reconstruction model obtained based on the projection parameters corresponding to the first neural network, the projection parameters including 3DMM coefficients, Pose coefficients and illumination coefficients; determining a third loss parameter based on the difference in vertex distance between the reconstruction model and the reconstructed object; determining a fourth loss parameter based on the difference in vertex normals between the reconstruction model and the reconstructed object; performing supervised training on the second neural network according to the third loss parameter and the fourth loss parameter to obtain the second neural network; obtaining a second projection image obtained based on the projection parameters corresponding to the first neural network; determining a fifth loss parameter based on the difference in color information corresponding to the second projection image and the object image labeled with the reconstructed object; obtaining preset key points to determine a sixth loss parameter based on the preset key points and the difference in projections corresponding to the preset key points; and adjusting parameters of the second neural network according to the fifth loss parameter and the sixth loss parameter. A fusion unit, used for fusing the unlabeled data with the labeled data to obtain fused data; The training unit is further used to train the second neural network based on the fusion data to obtain a target neural network, and the target neural network is used to perform projection reconstruction on the target object.

7. The device according to claim 6, characterized in that The training unit is specifically used for: Acquire a first projection image obtained based on projection parameters corresponding to the preset neural network; determining a first loss parameter based on a difference in color information corresponding to the first projection image and the unlabeled object image; Get preset key points; Determine a second loss parameter based on the preset key point and the difference between the preset key point and the corresponding projection; The preset neural network is self-supervised trained according to the first loss parameter and the second loss parameter to obtain the first neural network.

8. The device according to claim 6, characterized in that The training unit is specifically used for: Get the key points of the model; Determining first position information of the model key point in the reconstructed object including the annotation; Determine second position information of the model key point in the reconstructed model; The first position information is approximately adjusted toward the second position information so that the reconstructed object including the annotation is aligned with the reconstructed model, and the reconstructed object is updated.

9. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store program code; the processor is used to execute the training method of the model described in any one of claims 1 to 5 according to the instructions in the program code.

10. A computer program product, comprising a computer program / instructions, wherein the computer program / instructions are stored in a computer-readable storage medium, characterized in that: The computer program / instructions in the computer-readable storage medium implement the steps of the model training method described in any one of claims 1 to 5 when executed by a processor.

Images