A Generative Adversarial Network Inversion Method Combined with Binary Loss Constraint

Abstract

The present invention relates to a generative adversarial network inversion method combined with a binary loss constraint, which can solve the accuracy, stability and unique feature information missing problems in the current generative adversarial network inversion. The method comprises the following steps: using TriPlaneNet to construct an inversion network structure, and accurately inverting the features or noise of the original input from the input image. In the data set preprocessing stage, the unique identity ID information obtained according to the category is added to the training set label, so that the training set has the label information required for constructing the binary loss. At the same time, a binary loss function is defined, and the Euclidean distance error loss is used to measure the difference between the feature vectors obtained by inverting different pictures to constrain the difference between the features. The binary loss is introduced into the loss function of TriPlaneNet, and the network training begins. This innovative method enables the present invention to achieve accurate inversion of the generated image while retaining the unique features, bringing new breakthroughs in the field of three-dimensional model inversion of images.

Description

Technical Field

[0001] The present invention relates to an inversion method of a generative adversarial network (GAN), and in particular to an inversion method of a generative adversarial network combined with a binary loss constraint. Background Art

[0002] Generative adversarial networks (GANs) are used to generate realistic images or perform image transformations. Recently, thanks to the rapid development of neural radiance fields (Neff), GANs have introduced a variety of high-resolution and high-fidelity generative modeling methods for human heads, such as the EG3D model. At the same time, the inverse problem must be solved to re-render or modify existing images or videos. However, the inversion problem of GANs, that is, reconstructing the noise or features of the original input from the generated images, has always been a challenging research direction.

[0003] Traditional inversion methods such as PSP usually have some problems, such as ambiguity, unreality and loss of unique features. These problems lead to inaccuracy and unreliability of the inversion results. In the problem of inverting two-dimensional images into three-dimensional models, most inversion algorithms cannot obtain a true three-dimensional model with intact unique feature information. TriPlaneNet is currently the best inversion algorithm for EG3D models. Although TriPlaneNet's inversion method can better handle the problem of three-dimensional image inversion and improve the accuracy and stability of the inversion results. However, for non-face images, TfiPlaneNet still has low authenticity and unreliability, and cannot preserve the unique features of the target. Summary of the invention

[0004] The purpose of the present invention is to avoid the shortcomings of existing inversion methods in the problem of unique feature loss, especially the challenging problem of inverting two-dimensional images into three-dimensional models. A generative adversarial network inversion method combined with binary loss constraints is proposed to achieve inversion with unique feature preservation. The technical solution is as follows:

[0005] A Generative Adversarial Network Inversion Method Combined with Bigram Loss:

[0006] (1) Use the TriPlaneNet algorithm to build an inversion network structure: Using the optimization and 3D model fitting capabilities of TfiPlaneNet, an inversion network structure is constructed to accurately invert the features or noise of the original input from the generated image;

[0007] (2) Dataset preprocessing: adding unique identity (ID) information obtained based on the category to the training set label;

[0008] (3) Constructing a binary loss: By constructing a binary loss, the differences between different latent codes obtained by inverting images with the same ID information are constrained to be small, so as to improve the accuracy and stability of the inversion; the differences between different latent codes obtained by inverting images with different ID information are constrained to be large, so as to ensure the unique characteristics of the inversion;

[0009] (4) Introducing binary loss: Introduce binary loss into the loss function of TfiPlaneNet and start training the network;

[0010] (5) Using the trained TriPlaneNet network with binary loss, the untrained images and labels are used as the input of the network to obtain the target prediction 3D model result;

[0011] The beneficial effects of the present invention are:

[0012] (1) By combining TriPlaneNet and binary loss, the inversion method of the present invention can more accurately reconstruct the features or noise of the original input, thereby improving the accuracy and stability of the inversion. Compared with traditional methods, the original input information can be restored more accurately, reducing the error of the inversion result.

[0013] (2) The inversion method of the present invention is applicable to various three-dimensional image processing tasks, including medical image reconstruction, computer-aided design and other fields. By improving the accuracy and stability of inversion, the application field and scope of applicability are expanded, providing new solutions for a wider range of fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is an explanatory diagram of the introduction of binary loss of the present invention; DETAILED DESCRIPTION

[0015] To make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. The present invention is specifically implemented according to the following steps:

[0016] The specific implementation is as follows:

[0017] (1) Data and model preparation: Collect a large number of 3D image datasets and preprocess the data, including standardizing the image scale and the direction angle labels that were previously applicable to EG3D, to ensure data consistency and usability. Use the constructed dataset to train EG3D and obtain a pre-trained network that can generate realistic 3D models:

[0018] (2) TriPlaneNet inversion network construction: Construct the TriPlaneNet inversion network structure, which includes a PSP-based inversion model and a ResNet-based three-plane encoder-decoder network. The main structure of the PSP inversion network includes key components such as the encoder, perceptual similarity loss, cycle consistency loss, and StyleGAN decoder, which are used to map the input image to features or noise suitable for the StyleGAN decoder;

[0019] (3) Definition of binary loss function: Define the binary loss function and use the Euclidean distance error loss to measure the difference between the feature vectors obtained by inverting different images to constrain the difference between features. The following is a specific expression of the binary loss formula for calculating two features using the Euclidean distance:

[0020] 1. Constrain the difference between different potential codes obtained by inverting the same ID information image to be small:

[0021]

[0022] Where N represents the number of samples, represents the features of the i-th generated image, f i Represents the features of the corresponding original input. The goal of this loss function is to make the inversion results of the same ID image as close to the original features as possible, thereby improving the accuracy and stability of the inversion.

[0023] 2. The differences between different potential codes obtained by inverting different ID information images are constrained to be large:

[0024]

[0025] Where M represents the number of samples, represents the features of the jth generated image, f j Represents the features of the corresponding original input. △ is a pre-set threshold value, which is used to ensure that the difference between the inversion results is greater than the threshold. If the difference is less than the threshold, the value of the loss function is 0, indicating that no penalty is required; if the difference is greater than the threshold, the value of the loss function is the square of the difference. The goal of this loss function is to ensure the unique features of the inversion result. In summary, according to different situations, the above two loss functions can be used to constrain the differences between features, thereby improving the effect and stability of the inversion;

[0026] (4) Training the inversion network: Use the image dataset with ID labels to train the TriPlaneNet inversion network. During the training process, batch stochastic gradient descent (SGD) or Adam optimizer is used to minimize the network's loss function. In order to increase the stability of the training, a progressive training strategy is adopted to gradually increase the complexity and resolution of the network;

[0027] (5) Optimization and adjustment: adjust and optimize the hyperparameters of the inversion network, including learning rate, batch size, network depth, etc. Determine the best hyperparameter combination through cross-validation, grid search and other methods;

[0028] (6) Application practice: Apply the trained inversion model to actual scenarios, such as medical image reconstruction, geological image processing, etc. In actual applications, the model is continuously optimized and adjusted to meet the needs of specific application scenarios and improve the quality and stability of the inversion results;

[0029] Through the above specific implementation methods, the EG3D inversion method based on TriPlaneNet combined with binary loss can be effectively implemented, and high-quality inversion results can be obtained, which is suitable for various GAN inversion tasks that need to retain the unique features of the inversion object.

Claims

1. A generative adversarial network inversion method combined with binary loss constraints, characterized in that: The steps to incorporate bigram loss in the GAN inversion process include: (1) Use TriPlaneNet to build an inversion network structure: Use TriPlaneNet's optimization and 3D model fitting capabilities to build an inversion network structure; (2) Dataset preprocessing: add unique ID information obtained according to the category to the training set label; (3) Constructing a binary loss: By constructing a binary loss, the differences between different latent codes obtained by inverting images with the same ID information are constrained to be small, so as to improve the accuracy and stability of the inversion; the differences between different latent codes obtained by inverting images with different ID information are constrained to be large; (4) Introducing binary loss: Introducing binary loss into the loss function of TriPlaneNet and starting to train the network; (5) Using the trained TriPlaneNet network with binary loss, the untrained images and labels are used as the input of the network to obtain the target prediction 3D model result; (6) Apply the trained inversion model to actual scenarios. In the fields of medical image reconstruction and geological image processing, the model is continuously optimized and adjusted in practical applications to meet the needs of specific application scenarios and improve the quality and stability of the inversion results.

2. The generative adversarial network inversion method combined with binary loss constraints as claimed in claim 1, characterized in that: The data set preprocessing method in step (2) is: Each sample label in the training set is processed to add unique information based on the category, and samples of different categories are distinguished by adding new unique identifiers to the existing labels; The identifier for each category in the dataset is made unique by numbering the categories and then calculating the hash value; the label of each sample is associated with unique information obtained based on the category by adding the unique identifier or value to the label of the sample and storing it in the metadata of the dataset.

3. The generative adversarial network inversion method combined with binary loss constraints as claimed in claim 1, characterized in that: The definition and construction method of the bigram loss in step (3) are: Define the binary loss function, and use the Euclidean distance error loss to measure the difference between the feature vectors obtained by inverting different images, so as to constrain the difference between features in different situations; the following is a specific expression of the binary loss formula for calculating two features using the Euclidean distance: First, the difference between different potential codes obtained by inverting the same ID information image is constrained to be small: Where N represents the number of samples, represents the features of the i-th generated image, f i Represents the features of the corresponding original input. The goal of this loss function is to make the inversion results of the same ID image as close to the original features as possible, thereby improving the accuracy and stability of the inversion; Secondly, the difference between different potential codes obtained by inverting different ID information images is constrained to be large: Where M represents the number of samples, represents the features of the jth generated image, f j Represents the features of the corresponding original input. Δ is a pre-set threshold value, which is used to ensure that the difference between the inversion results is greater than the threshold. If the difference is less than the threshold, the value of the loss function is 0, indicating that no penalty is required. If the difference is greater than the threshold, the value of the loss function is the square of the difference. The goal of this loss function is to ensure the unique features of the inversion results. In summary, according to different situations, the above two loss functions can be used to constrain the differences between features, thereby improving the inversion effect and stability.

Images