Semantic level picture decoupling and generation optimization method

Abstract

The application discloses a semantic-level picture decoupling and generation optimization method. The method is based on a self-supervised decoupling representation learning technology, uses contrast learning and a pixel-level Gram matrix as a constraint to update the coefficient weight of an encoder and a decoder of a beta variational autoencoder, thereby obtaining semantic-level decoupling representation and generating clearer pictures through the decoupling representation. The technology can be applied to various picture editing software, including editing of facial expressions in pictures, editing of shapes and positions of things, etc., and has great commercial value.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and pattern recognition, primarily focusing on semantic-level decoupling of image content and decoding the decoupled features to generate new, editable images. This technology can be applied to various image editing software, including editing facial expressions, object shapes, and positions within images, and possesses significant commercial value. Background Technology

[0002] The semantic-level image decoupling and generation optimization method is an image editing technique based on self-supervised decoupling representation learning. With the rapid development of artificial intelligence, the advantages of self-supervised learning are becoming increasingly prominent. Compared to supervised learning, its main advantage lies in the fact that it does not require extensive manual labeling of data. Furthermore, supervised learning algorithms often suffer from low data utilization efficiency, and in particular, lack the generalization and robustness unique to biological intelligence.

[0003] Self-supervised decoupled representation learning effectively solves the problem of feature reusability. Models designed based on this technique can generate new images using the decoupled features. These independent and identically distributed features are also referred to as variation factors in some contexts. In 2011, Courville et al. proposed a pegboard-constrained Boltzmann machine to clean up obvious variation factors in data. Generally speaking, modeling methods used to study variation factors in data are called decoupled representation learning. However, there is no unified definition for decoupled representation learning to date, but the definition used by Bengio et al. in 2013 is commonly adopted: a representation in which a change in one dimension corresponds to a change in a variation factor, while other factors remain relatively unchanged. Specifically, decoupled representation learning essentially models the independent features of data morphology, so that a change in one decoupled feature occurs while other features remain unchanged. Therefore, models designed based on self-supervised decoupled representation learning are interpretable and possess the characteristics of object generation and manipulation. In 2013, Kingma et al. proposed the variational autoencoder model. A major feature of this model is its ability to obtain structural changes in relatively simple images, such as shape changes and tilt angles. In 2016, Higgins et al. modified the objective function of the variational autoencoder and proposed the visual decoupling representation learning model β-VAE. The representations decoupled by this model often contain certain visual meanings. For example, by using the β-VAE model to decouple face images, semantic features such as hair length and eye size can be obtained.

[0004] Decoupled representation learning offers significant advantages in image processing. Compared to traditional image editing techniques, it provides richer functionality, such as directly editing facial expressions in images, resulting in more realistic photos than those based on generative adversarial models. Furthermore, decoupled representation learning exhibits significant advantages in interpretability, broadening its application scenarios. Summary of the Invention

[0005] Purpose of the Invention: To address the problems encountered by conventional image editing methods in facial expression editing, such as inaccurate facial expression capture, insufficient processing of expression attribute details, changes in other facial attributes during expression editing, and distortion and blurriness of the edited image. This invention utilizes deep learning technology, combining a β-variable autoencoder model, contrastive learning techniques, and Gram matrices to solve the aforementioned problems of general image editing software. It provides targeted modeling of expression elements in facial images, resulting in more natural and realistic edited images.

[0006] Technical Solution: This invention proposes a semantic-level image decoupling and generation optimization method, including the following steps:

[0007] Step (1): Input the original image with a pixel size of 64×64 into the encoder, and obtain a low-dimensional decoupled representation through a downsampling convolutional layer and a fully connected layer;

[0008] Step (2): Input the low-dimensional decoupled representation into the decoder, and obtain a reconstructed image with a pixel size of 64×64 through a fully connected layer and an upsampling convolutional layer;

[0009] Step (3): Use the original image, the reconstructed image, and images randomly selected from other batches as anchor samples, positive samples, and negative samples, respectively. The number of anchor samples, positive samples, and negative samples in each batch are 1, 1, and 127, respectively. Input these three types of samples into the contrastive learning module, and use the contrastive loss function in this module to continuously update the coefficient weights in the encoder and decoder.

[0010] Step (4): Input the original image and the reconstructed image into the image generation optimization module based on the pixel-level Gram matrix. First, the length and width of the two 3D images are transformed into a two-dimensional matrix with length multiplied by width and number of channels by the inner product. Then, the two-dimensional matrix is ​​inner-producted with its transpose to obtain the pixel-level Gram matrix of the two 3D images. Finally, the F-norm of the two pixel-level Gram matrices is used as the loss function to continuously update the coefficient weights in the encoder and decoder.

[0011] Furthermore, the encoder in step (1) is specifically implemented as follows: In the encoder, the input image first passes through four downsampled convolutional layers with feature map sizes of 32@32×32, 32@16×16, 32@8×8, and 32@4×4, respectively. Then, the feature map of the fourth convolutional layer is flattened and connected to two fully connected layers. Each layer has 256 units. Next, there is a fully connected layer consisting of 20 units, which is used to parameterize the mean and logarithmic standard deviation of 10 Gaussian random variables. Using the reparameterization technique, a ten-dimensional decoupled representation is finally obtained.

[0012] Furthermore, the specific implementation method of the decoder in step (2) is as follows: In the decoder, the connection layer consisting of 20 units is first connected to two fully connected layers, each with 256 units, followed by four upsampled convolutional layers with feature map sizes of 32@4×4, 32@8×8, 32@16×16, and 32@32×32, respectively, and finally a reconstructed image with a pixel size of 64×64 is generated.

[0013] Furthermore, the specific implementation method of the contrastive learning module in step (3) is divided into two stages:

[0014] ① Original image preprocessing stage: First, data augmentation technology is used on the original images to obtain a series of image sets after operations such as enlargement, rotation, and cropping. Each batch of image sets contains 32 images. These image sets are preprocessed using a recurrent neural network. The recurrent neural network used here is the GRU model. Finally, the semantic association encoding between these 32 image sequences is obtained.

[0015] ② Contrastive learning loss function weight update stage: The semantic association encoding of the original image after preprocessing, the reconstructed image, and images randomly selected from other batches are used as anchor samples, positive samples, and negative samples, respectively, and input into the contrastive loss function, which is:

[0016]

[0017] in, x + and x - These represent the semantic association encoding of the original image after preprocessing, the reconstructed image, and the images randomly selected from other batches, namely anchored samples, positive samples, and negative samples. N-1 represents the number of images in other batches, which is 128 images in this case.

[0018] Furthermore, the specific implementation method of the image generation optimization module based on the pixel-level Gram matrix in step (4) is as follows: Input the original image and the reconstructed image into the pixel-level Gram matrix formula, which is:

[0019]

[0020] Where x represents the input image, (i,j,k) represent the image's width, height, and number of channels, and G... x This represents the pixel-level Gram matrix of the original or reconstructed image. The final image generation optimization module based on the pixel-level Gram matrix uses the F-norm of the pixel-level Gram matrices of the input and reconstructed images as a loss function to continuously update the coefficient weights of the encoder and decoder, as shown in the following formula:

[0021] L Gram =∑ k ∑ k' ||G x -G x' || F Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the core structure of the semantic-level image decoupling and generation optimization method described in this invention;

[0023] Figure 2 This is a schematic diagram of the overall structure of the semantic-level decoupled representation learning model described in this invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, this invention provides a semantic-level image decoupling and generation optimization method, including a β variational autoencoder, contrastive learning, and Gram matrix. The implementation method of this invention will be described in detail below.

[0026] Step 1: The main framework of the model is a β variational autoencoder, such as... Figure 2 As shown, it consists of two parts: a decoder and an encoder. The original image is input into the encoder, and after passing through a downsampling convolutional layer and a fully connected layer, a low-dimensional decoupled representation is obtained. The low-dimensional decoupled representation is then input into the decoder, and after passing through a fully connected layer and an upsampling convolutional layer, the reconstructed image is obtained.

[0027] Step 1.1: Input the original image with a pixel size of 64×64 into the encoder. The input image first passes through four downsampled convolutional layers with feature map sizes of 32@32×32, 32@16×16, 32@8×8, and 32@4×4, respectively. Then, the feature map of the fourth convolutional layer is flattened and connected to two fully connected layers, each with 256 units. Next, there is a fully connected layer with 20 units, which is used to parameterize the mean and log standard deviation of 10 Gaussian random variables. Using the reparameterization technique, a ten-dimensional decoupled representation is finally obtained.

[0028] Step 1.2: In the decoder, a connection layer consisting of 20 units is first connected to two fully connected layers, each with 256 units. Then there are four upsampled convolutional layers with feature map sizes of 32@4×4, 32@8×8, 32@16×16, and 32@32×32, respectively. Finally, a reconstructed image with a pixel size of 64×64 is generated.

[0029] Step 2: The main framework was used to achieve simple decoupling and generation of image content. However, to achieve semantic-level decoupling and higher-quality image generation, it is necessary to optimize the coefficient weights of the encoder and decoder by comparing the constraints of the learning module and the pixel-level Gram matrix.

[0030] Step 2.1: The specific implementation method of the contrastive learning module consists of two stages: the first is the stage of preprocessing the original image to obtain semantic association encoding, and the second is the stage of updating the encoder and decoder weights using the contrastive learning loss function, as described below:

[0031] ① Original image preprocessing stage: First, data augmentation technology is used on the original images to obtain a series of image sets after operations such as enlargement, rotation, and cropping. Each batch of image sets contains 32 images. These image sets are preprocessed using a recurrent neural network. The recurrent neural network used here is the GRU model. Finally, the semantic association encoding between these 32 image sequences is obtained.

[0032] ② Contrastive learning loss function weight update stage: The semantic association encoding of the original image after preprocessing, the reconstructed image, and images randomly selected from other batches are input into the contrastive loss function as anchor samples, positive samples, and negative samples, respectively. The number of anchor samples, positive samples, and negative samples in each batch are 1, 1, and 127, respectively. The contrastive loss function is:

[0033]

[0034] in, x + and x -These represent the semantic association encoding of the original image after preprocessing, the reconstructed image, and images randomly selected from other batches, i.e., anchor samples, positive samples, and negative samples, respectively. N-1 represents the number of images in other batches, which is 128 in this case. The purpose of this loss function is to learn the features of the dataset by having the model learn similar or different data points. These features often possess semantic information. Furthermore, since the features extracted by the β variational autoencoder are relatively independent, semantically independent features are obtained under the constraint of the contrastive loss function.

[0035] Step 2.2: Input the original image and the reconstructed image into the image generation and optimization module based on the pixel-level Gram matrix, such as... Figure 2 As shown, firstly, the length and width of these two 3D images are multiplied by an inner product to form a 2D matrix consisting of length, width, and number of channels. Then, this 2D matrix is ​​multiplied by its transpose to form an inner product. This yields the pixel-level Gram matrix of the two 3D images, as shown in the formula:

[0036]

[0037] Where x represents the input image, (i,j,k) represent the image's width, height, and number of channels, and G... x This represents the pixel-level Gram matrix of the original or reconstructed image. The final image generation optimization module based on the pixel-level Gram matrix uses the F-norm of the pixel-level Gram matrices of the input and reconstructed images as a loss function to continuously update the coefficient weights of the encoder and decoder, as shown in the following formula:

[0038] L Gram =∑ k ∑ k' ||G x -G x '|| F

[0039] This constraint makes the relationships between pixels in the original image and the reconstructed image closer, thus improving the quality of the reconstructed image and consequently enhancing the quality of the new image generated after editing.

[0040] The detailed descriptions listed above are merely specific descriptions of feasible implementations of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent implementations or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A semantic-level image decoupling and generation optimization method, characterized in that, Includes the following steps: Step (1): Input the original image with a pixel size of 64×64 into the encoder. The input image first goes through four downsampled convolutional layers. Each convolutional layer has 32 channels. Here, a 4×4 convolutional kernel is used with a stride of 2. The size of the feature map of each layer is 32×32, 16×16, 8×8, and 4×4, respectively. Then, the feature map of the fourth convolutional layer is flattened and connected to two fully connected layers. Each layer has 256 units. Next, there is a fully connected layer consisting of 20 units, which is used to parameterize the mean and log standard deviation of 10 Gaussian random variables. Here, the reparameterization technique is used to finally obtain a ten-dimensional decoupled representation. Step (2): In the decoder, a fully connected layer consisting of 20 units is first connected to two fully connected layers, each with 256 units. Then there are four upsampled convolutional layers, each with 32 channels. Here, a 4×4 convolutional kernel is used with a stride of 2. The size of the feature maps in each layer is 4×4, 8×8, and 4×4 respectively. 16×16, 32×32, finally generating a reconstructed image with a pixel size of 64×64; Step (3): The original image, the reconstructed image, and the images randomly selected from other batches are used as anchor samples, positive samples, and negative samples, respectively. The number of anchor samples, positive samples, and negative samples in each batch are 1, 1, and 127, respectively. These three types of samples are input into the contrastive learning module, and the contrastive loss function in the module is used to continuously update the coefficient weights in the encoder and decoder. Step (4): Input the original image and the reconstructed image into the image generation optimization module based on the pixel-level Gram matrix to obtain the pixel-level Gram matrix of the two 3D images. Use the F-norm of the pixel-level Gram matrix as the loss function, and continuously update the coefficient weights in the encoder and decoder through the backpropagation algorithm. Under the combined action of the encoder, decoder and pixel-level Gram matrix, the loss function of the image generation optimization module based on the pixel-level Gram matrix is ​​as follows: Where x represents the input image, (i,j,k) represent the image's width, height, and number of channels, and G... x This represents the pixel-level gram matrix of the original or reconstructed image.

2. The semantic-level image decoupling and generation optimization method according to claim 1, characterized in that, The specific implementation method of the contrastive learning module in step (3) is divided into two stages: the first is the original image preprocessing stage to obtain semantic association encoding, and the second is the contrastive learning loss function to update the encoder and decoder weights. In the original image preprocessing stage, the original image is first subjected to data augmentation technology to obtain a series of image sets after operations such as enlargement, rotation, and cropping. Each batch of image sets contains 32 images. These image sets are preprocessed using a recurrent neural network. The recurrent neural network used here is the GRU model. Finally, the semantic association encoding between these 32 image sequences is obtained. In the contrastive learning loss function to update the encoder and decoder weights stage, the semantic association encoding of the original image after preprocessing, the reconstructed image, and the randomly selected images from other batches are used as anchor samples, positive samples, and negative samples, respectively, and input into the contrastive loss function. Under the joint action of the encoder, decoder, and contrastive loss function, the loss function of the β variational autoencoder based on contrastive learning is: in, x + and x - These represent the semantic association encoding of the original image after preprocessing, the reconstructed image, and the images randomly selected from other batches, namely anchored samples, positive samples, and negative samples. N-1 represents the number of images in other batches, which is 128 here.

Images