Method and device for generating high-quality images by training a generative adversarial network through introducing self-supervised composite tasks

Abstract

The present invention discloses a method and device for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task, comprising the following steps: (1) preparing an original image data set and a spliced ​​image data set; (2) designing a composite task for realizing self-supervised learning of a generative adversarial network; (3) building a model, constructing an adversarial training branch and a self-supervised composite task branch; (4) training the model and saving the parameters of the network; (5) using the trained generator network to generate images. The method for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task provided by the present invention utilizes both the information within the image and the information between images at the same time, constructs a composite task including three subtasks, guides the network to learn more stable and more general features in the image, and also constructs a local discriminator to improve the network's ability to extract local information of the image, which can significantly improve the training effect of the network and improve the quality of the image finally generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer image generation, and in particular to a method and device for generating high-quality images by introducing self-supervised composite task training to generate adversarial networks. Background Art

[0002] Real image datasets are essential tools for training networks in the field of computer vision. A large number of real images can help the network learn useful feature representations and perform well in subsequent tasks. However, in the process of making real image datasets, due to the differences in image acquisition equipment, the original images collected need to be resized, aligned, and have uniform resolutions, which usually requires huge manpower costs. This is an important reason for the high cost of network training. In addition, for some samples with low existing volume and high acquisition difficulty, the production cost and difficulty of the dataset are higher. Insufficient sample size will also lead to poor network training results. A generated image is an image that is similar to the real image in the training set and generated by a well-trained generative model. It can be directly mapped from random noise by the generative model. One of the ways to solve the above problems is to expand the existing dataset by using a large number of generated images generated by the generative model. The generative model generates realistic and diverse generated images by learning from the real images in the existing dataset, which can greatly reduce the dataset production cost caused by manual collection and processing of data.

[0003] Generative adversarial networks (GANs) are a popular generative model in recent years. They were proposed by Goodfellow et al. in the article "Generative adversarial networks." (NeuralIPS, 2014). In a GAN, the generator is responsible for receiving random noise and generating images, and the discriminator is responsible for receiving real image samples and samples generated by the generator, and judging whether the received samples are real images. During the training process of the network, the generator and the discriminator are continuously optimized in mutual confrontation. However, GANs have the disadvantages of "catastrophic forgetting" of the discriminator and unstable training process, which may even lead to mode collapse. One of the current solutions is to complete the self-supervised learning of the GAN by introducing additional auxiliary tasks, so that the discriminator can learn more general and stable features to improve the stability of the training process. However, existing auxiliary tasks are usually single tasks, which easily lead to obvious task bias in the features learned by the network. For example, the rotation task proposed by Gidaris et al. in the article "Unsupervised representation learning by predicting image rotations." (ICLR, 2018) randomly rotates the input image and requires the network to determine the rotation angle corresponding to the received image. Although this task can effectively help the network learn the structural features of the image, the color, texture and other features of the image are not very helpful for the task of determining the rotation angle and are easily ignored by the network. The existing self-supervised auxiliary tasks proposed for generative adversarial networks have the problems of single tasks and insufficient coverage of features. They have a strong bias in guiding the process of network learning features, which is not conducive to the network learning general and stable features, and will also affect the quality of the subsequently generated images. Summary of the invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a method and device for generating high-quality images by introducing a self-supervised composite task to train a generative adversarial network. By designing and introducing a reasonable composite task, the network is guided to learn more stable and more general features in the image, thereby improving the training effect of the network and further improving the quality of the final generated image.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In one aspect, the present invention provides a method for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task, comprising the following steps:

[0007] Prepare a training data set, the training data set including original image data and spliced ​​image data; the original image data is used in the training process of the adversarial training branch, and the spliced ​​image data is used in the training process of the self-supervised composite task branch;

[0008] Design three subtasks to form a composite task, the composite task is used to construct a self-supervised composite task branch and provide supervision information for model training, the three subtasks are rotation prediction task, position prediction task and common feature extraction task, the rotation prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, the position prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, and the common feature extraction task is used to first correctly determine which original image each image block belongs to, and then extract the common features between homologous image blocks;

[0009] Building a model, constructing an adversarial training branch and a self-supervised composite task branch respectively, wherein the adversarial training branch includes a local discriminator and a generator, and the self-supervised composite task branch includes a classifier with three output heads;

[0010] The constructed model is trained to obtain a trained generator network; the training specifically includes: taking the original data set as the input of the adversarial training branch, and taking the spliced ​​image data as the input of the self-supervised composite task branch, and training the networks in the two branches. During the training process, the self-supervised composite task branch is responsible for providing supervision information for the local discriminator in the adversarial training branch;

[0011] The image to be processed is input into the trained generator network for image generation.

[0012] As a preferred technical solution, the stitching image data is prepared as follows:

[0013] For the original images in a batch, four image blocks with a certain overlap rate ω are cropped from the upper left, upper right, lower left, and lower right regions of each image. The overlap rate ω is equal to the ratio of the side length of the overlapping part between adjacent image blocks to the side length of the original image.

[0014] The obtained batch of image blocks is randomly shuffled and rotated once, and the rotation angle is randomly selected from the set R = {0°, 90°, 180°, 270°};

[0015] The size of a batch of image blocks is adjusted using bilinear interpolation so that the side length of the image block is equal to half the side length of the original image;

[0016] The obtained image blocks are stitched together in groups of 4 to obtain a batch of stitched images with the same size as the original images, thereby completing the production of stitched image data.

[0017] As a preferred technical solution, the specific designs of the three subtasks are as follows:

[0018] In the rotation prediction task, for a batch of spliced ​​images, each image contains four image blocks, each corresponding to a rotation angle, and a pseudo label l is assigned to each image block through this rotation angle. r , l r ∈{0°, 90°, 180°, 270°};

[0019] In the position prediction task, for a batch of spliced ​​images, each image contains four image blocks that correspond to a fixed area position in the original image to which it belongs. A pseudo label l is assigned to each image block through this position information. l , l l ∈{upper left, upper right, lower left, lower right};

[0020] In the common feature extraction task, for a batch of spliced ​​images obtained, the four image blocks contained in each image correspond to an original image respectively. The four image blocks belonging to the same original image are defined as homologous image blocks, and the features with high similarity between homologous image blocks are defined as common features.

[0021] As a preferred technical solution, the specific steps of building the model are:

[0022] In the adversarial training branch, a local discriminator local-D and a generator G are constructed. The network structure of the local discriminator is divided into two parts, and a feature block module is added between the two. The first part receives the original image as input and extracts image features. The feature block module is responsible for processing the image features output by the first part into image block features. The second part receives the image block features as input and generates the final output of the local discriminator. The task of the local discriminator local-D is to correctly determine whether the block features are from the real image or the generated image. The loss function of this branch is recorded as L adv , which is consistent with the adversarial loss proposed in the original generative adversarial network, and its specific expression is:

[0023]

[0024] Where x is a real image sampled from the original dataset, P data(x) is the distribution of the real data, z is the random noise sampled from the prior distribution, P z(z) is the prior distribution, D is the local discriminator, and G is the generator;

[0025] In the self-supervised composite task branch, the classifier C is built using the same network architecture as the local discriminator in the adversarial training branch. The classifier network is also divided into two parts. The network architecture of the first part is the same as the first part of local-D, and the two share network weights. The second part contains three output heads of the classifier, two of which are composed of a fully connected layer, responsible for outputting the results of the rotation prediction task and the position prediction task respectively. The third head is composed of a multi-layer perceptron with a hidden layer, responsible for outputting the results of the common feature extraction task. The total loss function of this branch is recorded as L CT .

[0026] As a preferred technical solution, multiple loss functions are used for combined optimization of the self-supervised composite task branch. The total loss function L of the self-supervised composite task branch is CT Defined as:

[0027] L CT =L rot +L loc +L cFE

[0028] Among them, L rot , L loc , L CFE They represent the rotation prediction task loss, position prediction task loss, and common feature extraction task loss respectively;

[0029] The true rotation label of each image block in the spliced ​​image is denoted as l r_gt , the true location label is l l_gt , the rotation label predicted by the classifier for the image patch is l r , the predicted position label is l l , a set of vectors output by the multilayer perceptron in the shared feature extraction task is denoted as v 1 ,v 2 ,…,v {k.k＝n×4} , binary cross entropy is used to calculate the loss of rotation prediction and position prediction, and cosine similarity is used to calculate the similarity between different block features. The calculation formulas for the three task losses are as follows:

[0030] L rot =CrossEntropy(l r ,l r_gt )

[0031] L loc =CrossEntropy(l l ,l l_gt )

[0032]

[0033]

[0034] Where τ is the temperature coefficient, n is the training batch size, and C i It represents the set of subscripts corresponding to the homologous image blocks of the i-th image block, I is an indicator function, and the function value is 1 when the judgment condition is met, otherwise it is 0.

[0035] As a preferred technical solution, the specific steps of model training are:

[0036] In the adversarial training branch, the generator G and the local discriminator local-D are trained alternately and iteratively. The input of the local discriminator is a batch of images sampled from the original image dataset. Its training goal is to correctly judge the authenticity of the image in a certain area of ​​the input image. The input of the generator is random noise. Its training goal is to output a generated image that is as realistic as possible and can deceive the local discriminator.

[0037] In the self-supervised composite task branch, the input of the classifier is a batch of spliced ​​images, and the three output heads output the results of the three subtasks respectively. This branch is calculated through the total loss function L CT Train the network;

[0038] The adversarial training branch and the self-supervised composite task branch are trained simultaneously. During the training process, the two branches are connected by sharing the network weights of the first part of the local discriminator local-D and the three-head classifier C. The total loss function of the model is defined as:

[0039]

[0040] Among them, L adv (G, D) is the adversarial training loss, L CT (C) is the self-supervised composite task loss. During the model training process, the local discriminator local-D is updated alternately with the generator G and updated simultaneously with the three-head classifier C.

[0041] As a preferred technical solution, the image to be processed is input into a trained generator network for image generation, specifically:

[0042] By inputting random noise into the trained generator network, high-quality generated images similar to the training set images can be obtained through forward propagation.

[0043] Another aspect of the present invention provides a system for generating high-quality images by training a generative adversarial network through the introduction of a self-supervised composite task, which is applied to the method for generating high-quality images by training a generative adversarial network through the introduction of a self-supervised composite task, and includes a data set module, a composite task module, a model building module, a model training module, and an image generation module;

[0044] The data set module is used to prepare a training data set, and the training data set includes original image data and spliced ​​image data; the original image data is used in the training process of the adversarial training branch, and the spliced ​​image data is used in the training process of the self-supervised composite task branch;

[0045] The composite task module is used to design three subtasks to form a composite task, the composite task is used to construct a self-supervised composite task branch and provide supervision information for model training, the three subtasks are a rotation prediction task, a position prediction task and a common feature extraction task, the rotation prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, the position prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, and the common feature extraction task is used to first correctly determine which original image each image block belongs to, and then extract the common features between homologous image blocks;

[0046] The model building module is used to construct an adversarial training branch and a self-supervised composite task branch respectively, wherein the adversarial training branch includes a local discriminator and a generator, and the self-supervised composite task branch includes a classifier with three output heads;

[0047] The model training module is used to train the constructed model to obtain a trained generator network; the training is specifically: the original data set is used as the input of the adversarial training branch, the spliced ​​image data is used as the input of the self-supervised composite task branch, and the networks in the two branches are trained. During the training process, the self-supervised composite task branch is responsible for providing supervision information for the local discriminator in the adversarial training branch;

[0048] The image generation module is used to input the image to be processed into the trained generator network for image generation.

[0049] Another aspect of the present invention provides an electronic device, the electronic device comprising:

[0050] at least one processor; and,

[0051] a memory communicatively connected to the at least one processor; wherein,

[0052] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the method of generating high-quality images by introducing self-supervised composite task training to generate adversarial networks.

[0053] On the other hand, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the method of generating high-quality images by introducing self-supervised composite task training to generate adversarial networks.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] (1) The present invention proposes a composite auxiliary task based on multi-level information for self-supervised learning of generative adversarial networks. It also uses intra-image information and inter-image information as supervision during network training, which stabilizes the training process, improves the versatility of features extracted by the network, and enhances the quality of generated images.

[0056] (2) The present invention proposes a local discriminator local-D, which enables the network to pay attention to more local feature information during the feature extraction process, and cooperates with the self-supervised composite task branch to improve the effect of self-supervised learning of the entire model.

[0057] (3) The common feature extraction task proposed in the present invention introduces the idea of ​​contrastive learning into the self-supervised learning of generative adversarial networks. While utilizing the advantages of contrastive learning to improve the quality of extracted features, the training batch size is kept at a smaller value, thereby bringing greater network performance gains at a lower training cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 This is an overall flow chart of a method for generating high-quality images by introducing a self-supervised composite task to train a generative adversarial network in an embodiment of the present invention.

[0060] Figure 2 It is a flow chart of the image processing module of an embodiment of the present invention; it includes three states of image data, state 1 is the initial state of image data in a batch, state 2 is the state after each image is cropped into 4 image blocks, and state 3 is the state of a batch of spliced ​​image data obtained after all image blocks are randomly shuffled, randomly rotated, and spliced.

[0061] Figure 3 It is an overall structural diagram of the network model of an embodiment of the present invention.

[0062] Figure 4 This is a structural diagram of a generator network according to an embodiment of the present invention.

[0063] Figure 5 2 is a structural diagram of a deconvolution module in a generator network according to an embodiment of the present invention.

[0064] Figure 6 This is a structural diagram of the local discriminator network proposed in an embodiment of the present invention.

[0065] Figure 7 4 is a structural diagram of a convolution module in a local discriminator according to an embodiment of the present invention.

[0066] Figure 8 Schematic diagram of a feature segmentation module according to an embodiment of the present invention.

[0067] Fig. 9 The present invention is a block diagram of a system for generating high-quality images by introducing a self-supervised composite task to train a generative adversarial network.

[0068] Fig.10 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0070] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0071] See also Figure 1 This embodiment provides a method for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task, including the following steps:

[0072] S1. Prepare training data sets, including original image data sets and spliced ​​image data sets;

[0073] Furthermore, the original image dataset is used as the input of the adversarial training branch, and the public Cifar-10, CelebA, ImageNet32×32, STL-10 and other datasets are obtained through the network search engine as the original image dataset of this method.

[0074] Furthermore, the spliced ​​image dataset is used as the input of the self-supervised composite task branch, which is obtained by processing the original image dataset. For the processing flow, please refer to Figure 2 First, for the original images in a batch, four image blocks with a certain edge length overlap rate ω = 0.4 are cropped from the upper left, upper right, lower left, and lower right regions of each image. Then, they are randomly shuffled and rotated once. The rotation angle is randomly selected from the set R = {0°, 90°, 180°, 270°}. The size of the image block is adjusted using bilinear interpolation to make its side length equal to half of the side length of the original image. Finally, these image blocks are spliced ​​in groups of 4 to obtain a batch of spliced ​​images with the same size as the original image, that is, the required spliced ​​image dataset is obtained.

[0075] S2. Design a composite task, which is used to construct a powerful supervision signal for the network training process;

[0076] Furthermore, in order to make full use of the image information, this embodiment designs a composite task that includes three subtasks, namely a rotation prediction task, a position prediction task, and a common feature extraction task. The first two tasks are designed based on the information within the image, and the third task is designed based on the information between images.

[0077] Furthermore, the rotation prediction task uses the rotation information of the image block to construct a pseudo label l r , l r ∈{0°, 90°, 180°, 270°}, this task requires the network to correctly determine the rotation labels corresponding to the four image blocks contained in the spliced ​​image. The network can only successfully complete this task if it has a good understanding of the structural information in the image blocks.

[0078] Furthermore, the position prediction task uses the position information of the image block in the original image to construct a pseudo label l l , l l∈{upper left, upper right, lower left, lower right}, this task requires the network to correctly determine the original position labels corresponding to the four image blocks contained in the spliced ​​image. Specifically, when making a spliced ​​image dataset, image blocks are obtained by cropping the upper left, upper right, lower left, and lower right regions of the original image. Therefore, each image block corresponds to a position label in the original image. The network can only correctly predict the position label of the image block if it has a good understanding of the image block and the structural features of the original image.

[0079] Furthermore, the common feature extraction task utilizes the information between different images and requires the network to extract the common features between homologous image blocks. The four image blocks cropped from the same original image are defined as homologous image blocks, and the features with high similarity between them are common features. The principle of this task is that the four homologous image blocks together constitute a semantically complete original image, and there is a certain overlap between them, so there are some common features between homologous image blocks. These common features can help the network correctly distinguish homologous image blocks from non-homologous image blocks. Therefore, in the process of completing this task, the network's ability to extract common features will continue to be enhanced, which helps the network learn representative features in the image.

[0080] S3, build a network model;

[0081] See also Figure 3 ,The entire network model building process includes four parts: using the ResNet-50 network to implement the generator, using the ResNet-50 network to implement the local discriminator, building a three-head classifier, designing the loss function of each branch and the total loss function of the network.

[0082] Furthermore, the specific steps for building a network model are as follows:

[0083] S31. Build the generator network:

[0084] See also Figure 4 , the structure of the generator network constructed; for the input random noise, it is first learned through a fully connected layer with an output channel number of 4096, and the size of the obtained vector is adjusted to 4×4×256; then it passes through three consecutive deconvolution modules; after that, a batch normalization, ReLU activation and 3×3 convolution (the number of output channels is 3 and the step size is 1) are performed, and finally after Sigmoid activation, the final output size is 32×32×3.

[0085] The structure of the deconvolution module can be found in Figure 5 , which includes the following operations: batch normalization, ReLU activation, upsampling, 3×3 convolution, ReLU activation, 3×3 convolution, and obtain X 1; Upsample the input X and perform 1×1 convolution to obtain X 2 , the final output of the deconvolution module is X 1 +X 2 The number of output channels of the convolution operations involved in the deconvolution module is 256, and the step size is 1.

[0086] S32. Build a local discriminator network:

[0087] See also Figure 6 The structure of the proposed local discriminator network; the local discriminator network is divided into two parts. The first part consists of multiple convolution modules and ReLU activation operations, which is responsible for outputting the intermediate layer features; the second part consists of feature block modules and fully connected layers, which are responsible for processing the intermediate layer features and generating the final output. The initial size of the input image is 32×32×3. It first passes through the first convolution module, which includes the following steps: 3×3 convolution, ReLU activation, 3×3 convolution, 1×1 convolution (step size 2) on the input X to obtain X 1 , perform 1×1 convolution (step size 2) and 1×1 convolution (step size 1) on the input X to obtain X 2 , the output is X 1 +X 2 ; After that, it passes through three consecutive convolution modules; after a ReLU activation is performed on the output of the convolution module, the obtained intermediate layer features are input into the feature block module for processing, and the block features are output; finally, the block features are input into a fully connected layer with an output channel number of 1 to obtain the final output result.

[0088] Further, the structure of three consecutive convolutional modules can be found in Figure 7 , specifically including the following steps: perform two repeated ReLU activations and 3×3 convolutions on the input X, and one 1×1 convolution (with a step size of 2) to obtain X 1 , perform 1×1 convolution (step size 1), 1×1 convolution (step size 2) on the input X, and get X 2 , the final output of the convolution module is X 1 +X 2 The number of output channels of all convolution operations involved in the convolution module is 128, and the stride of all 3×3 convolutions is 1. It should be noted that in the last two convolution modules, the 1×1 convolution operation with a stride of 2 is discarded, so the size of the intermediate layer features obtained after ReLU activation is 8×8×128.

[0089] The structure of the feature block module is as follows Figure 8 As shown in the figure, for the input intermediate layer feature F, average pooling is first performed (the average pooling kernel size is 2×2 and the step size is 2), and then the obtained feature is divided into 2×2 blocks in the length and width dimensions to obtain the block feature Fp , whose size is 2×2×128, which is four times the number of input intermediate layer features F.

[0090] S33, build a three-head classifier network:

[0091] The structure of the three-head classifier network is also divided into two parts. The structure of the first part is the same as the first part of the local discriminator, and they share weights and update parameters synchronously. This part is mainly responsible for extracting the middle layer features of the input spliced ​​image; the second part includes a feature block module and three output heads. For the structure of the feature block module, please refer to Figure 8 , which is responsible for processing the intermediate layer features into block features and outputting them to three heads. Among the three output heads, two heads are composed of a simple fully connected layer with 4 output channels, which are responsible for predicting the rotation angle and original position label of the image block respectively; the third head is composed of a multi-layer perceptron containing a hidden layer, which is responsible for completing the common feature extraction task. The number of output channels of the hidden layer is consistent with the number of channels of the input block features. The ReLU activation function is used, and the number of output channels of the final output layer is 64. The method of using a multi-layer perceptron to perform an additional nonlinear transformation on the features was first proposed in the article "A simple framework for contrastive learning of visual representations." (ICML, 2020) by Chen et al., which can help the network learn higher quality feature representations.

[0092] S4. Design loss function:

[0093] The total loss function L of the model proposed in this embodiment final It consists of the loss functions of two branches, namely: the loss function L of the adversarial training branch adv And the loss function L of the self-supervised composite task branch CT The expression of the total loss function of the model is as follows:

[0094]

[0095] Among them, L adv The calculation method of L is consistent with the loss function in the classic generative adversarial network, which was first proposed by Goodfellow et al. in the article "Generative adversarial networks." (NeuralIPS, 2014). adv The specific expression is as follows:

[0096]

[0097] Where x is a real image sampled from the original dataset, P data(x) is the distribution of the real data, z is the random noise sampled from the prior distribution, P z(z) is the prior distribution, D is the local discriminator, and G is the generator.

[0098] L CT It consists of three subtask losses, namely: rotation prediction task loss L rot , location prediction task loss L loc , common feature extraction task loss L CFE , which is defined as:

[0099] L CT =L rot +L loc +L CFE

[0100] L rot =CrossEntropy(l r ,l r_gt )

[0101] L loc =CrossEntropy(l l ,l l_gt )

[0102]

[0103]

[0104] Among them, l r_gt With l l_gt They represent the true rotation label and true position label of each image block in the spliced ​​image, respectively. r With l l They represent the rotation label and position label predicted by the classifier for the image patch, CrossEntropy(·) is the binary cross entropy function, and v 1 ,v 2 ,…,v {k.k＝n×4} represents a set of vectors output by a multilayer perceptron, sim(·) is the cosine similarity function, I is the indicator function, the function value is 1 when the judgment condition is met, otherwise it is 0, τ is the temperature coefficient, the default value is 0.3, n is the training batch size, the default value is 64, C i Represents the set of subscripts corresponding to the homologous image blocks of the i-th image block.

[0105] S4, training the constructed model;

[0106] The original image dataset is input into the generator and local discriminator for alternating training. At the same time, the spliced ​​image dataset is input into the three-head classifier for training. The Adam algorithm with adaptive learning rate is used to optimize the loss. The training batch size is 64 and a total of 300,000 training times are performed.

[0107] Furthermore, during the training process, for the adversarial training branch, random noise is input into the generator to obtain a generated image. When the generated image is input into the discriminator, the adversarial training loss is calculated and the gradient is back-propagated. The generator adjusts parameters for optimization so that it tends to generate generated images that are closer to the real image. When a batch of real images sampled from the original data set is input into the local discriminator, the adversarial training loss is calculated and the gradient is back-propagated. The discriminator adjusts parameters for optimization so that it tends to improve the ability to distinguish between generated images and real images. For the self-supervised composite task branch, when a batch of spliced ​​images is input into the three-head classifier, the classifier will perform the three subtasks in the composite task at the same time, calculate the total loss of the composite task and back-propagate the gradient. The classifier adjusts parameters for optimization so that it tends to extract more comprehensive and general image features to better complete the composite task. The classifier shares parameters with the local discriminator, so it is updated simultaneously with the local discriminator during the training process, and the local discriminator also obtains the feature extraction capability brought by the composite task. During the training process of the model, the generator and the local discriminator form a mutually adversarial relationship, that is, the generator is committed to generating realistic images that are more able to deceive the discriminator, while the discriminator is committed to continuously improving its ability to distinguish between real images and generated images. When the confrontation between the two reaches a balance, the image generation ability of the generator and the feature extraction ability of the discriminator both reach a high level. When the training is completed, the parameters of the entire model are saved.

[0108] S5, image generation;

[0109] By removing the local discriminator in the adversarial training branch and the entire self-supervised composite task branch, image generation can be completed using only the generator network: random noise is input into the generator, and a high-quality image similar to the original dataset can be obtained through forward propagation.

[0110] The method provided in this embodiment generates high-quality images by introducing a self-supervised composite task to train a generative adversarial network, utilizes both intra-image information and inter-image information simultaneously, constructs a composite task consisting of three subtasks, guides the network to learn more stable and general features in the image, and also constructs a local discriminator to improve the network's ability to extract local information from the image, which can significantly improve the training effect of the network and the quality of the final generated image.

[0111] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0112] Based on the same idea as the method for generating high-quality images by introducing self-supervised compound task training to generate adversarial networks in the above-mentioned embodiment, the present invention also provides a system for generating high-quality images by introducing self-supervised compound task training to generate adversarial networks, which can be used to execute the above-mentioned method for generating high-quality images by introducing self-supervised compound task training to generate adversarial networks. For ease of explanation, in the structural schematic diagram of the embodiment of the system for generating high-quality images by introducing self-supervised compound task training to generate adversarial networks, only the parts related to the embodiment of the present invention are shown, and those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0113] See also Fig. 9 In another embodiment of the present application, a system 100 for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task is provided, the system comprising a data set module 101, a composite task module 102, a model building module 103, a model training module 104 and an image generation module 105;

[0114] The data set module 101 is used to prepare a training data set, and the training data set includes original image data and spliced ​​image data; the original image data is used in the training process of the adversarial training branch, and the spliced ​​image data is used in the training process of the self-supervised composite task branch;

[0115] The composite task module 102 is used to design three subtasks to form a composite task, and the composite task is used to construct a self-supervised composite task branch and provide supervision information for model training. The three subtasks are a rotation prediction task, a position prediction task, and a common feature extraction task. The rotation prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, the position prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, and the common feature extraction task is used to first correctly determine which original image each image block belongs to, and then extract the common features between the homologous image blocks;

[0116] The model building module 103 is used to construct an adversarial training branch and a self-supervised composite task branch respectively, wherein the adversarial training branch includes a local discriminator and a generator, and the self-supervised composite task branch includes a classifier with three output heads;

[0117] The model training module 104 is used to train the constructed model to obtain a trained generator network; the training specifically includes: taking the original data set as the input of the adversarial training branch, and taking the spliced ​​image data as the input of the self-supervised composite task branch, and training the networks in the two branches. During the training process, the self-supervised composite task branch is responsible for providing supervision information for the local discriminator in the adversarial training branch;

[0118] The image generation module 105 is used to input the image to be processed into the trained generator network for image generation.

[0119] It should be noted that the system of the present invention for generating high-quality images by introducing self-supervised compound task training to generate adversarial networks corresponds one to one with the method of the present invention for generating high-quality images by introducing self-supervised compound task training to generate adversarial networks. The technical features and beneficial effects described in the above-mentioned embodiment of the method for generating high-quality images by introducing self-supervised compound task training to generate adversarial networks are all applicable to the embodiment of generating high-quality images by introducing self-supervised compound task training to generate adversarial networks. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0120] In addition, in the implementation of the system for generating high-quality images by introducing self-supervised composite task training to generate adversarial networks in the above-mentioned embodiments, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the system for generating high-quality images by introducing self-supervised composite task training to generate adversarial networks is divided into different program modules to complete all or part of the functions described above.

[0121] See also Fig.10 In one embodiment, an electronic device is provided for implementing a method for generating high-quality images by introducing a self-supervised composite task to train a generative adversarial network. The electronic device 200 may include a first processor 201, a first memory 202, and a bus. It may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a self-supervised composite task to train a generative adversarial network to generate high-quality images program 203.

[0122] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the self-supervised composite task training generation adversarial network to generate high-quality image program 203, but also can be used to temporarily store data that has been output or is to be output.

[0123] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules (such as federated learning defense programs, etc.) stored in the first memory 202, and calls data stored in the first memory 202 to execute various functions of the electronic device 200 and process data.

[0124] Fig.10 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0125] The self-supervised composite task training generative adversarial network generation high-quality image generation program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0126] Prepare a training data set, the training data set including original image data and spliced ​​image data; the original image data is used in the training process of the adversarial training branch, and the spliced ​​image data is used in the training process of the self-supervised composite task branch;

[0127] Design three subtasks to form a composite task, the composite task is used to construct a self-supervised composite task branch and provide supervision information for model training, the three subtasks are rotation prediction task, position prediction task and common feature extraction task, the rotation prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, the position prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, and the common feature extraction task is used to first correctly determine which original image each image block belongs to, and then extract the common features between homologous image blocks;

[0128] Building a model, constructing an adversarial training branch and a self-supervised composite task branch respectively, wherein the adversarial training branch includes a local discriminator and a generator, and the self-supervised composite task branch includes a classifier with three output heads;

[0129] The constructed model is trained to obtain a trained generator network; the training specifically includes: taking the original data set as the input of the adversarial training branch, and taking the spliced ​​image data as the input of the self-supervised composite task branch, and training the networks in the two branches. During the training process, the self-supervised composite task branch is responsible for providing supervision information for the local discriminator in the adversarial training branch;

[0130] The image to be processed is input into the trained generator network for image generation.

[0131] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task, characterized in that: The steps include: Prepare a training data set, the training data set including original image data and spliced ​​image data; the original image data is used in the training process of the adversarial training branch, and the spliced ​​image data is used in the training process of the self-supervised composite task branch; Design three subtasks to form a composite task, the composite task is used to construct a self-supervised composite task branch and provide supervision information for model training. The three subtasks are rotation prediction task, position prediction task and common feature extraction task. The rotation prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, the position prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, and the common feature extraction task is used to first correctly determine which original image each image block belongs to, and then extract the common features between homologous image blocks; Building a model, constructing an adversarial training branch and a self-supervised composite task branch respectively, wherein the adversarial training branch includes a local discriminator and a generator, and the self-supervised composite task branch includes a classifier with three output heads; The constructed model is trained to obtain a trained generator network; the training specifically includes: taking the original data set as the input of the adversarial training branch, and taking the spliced ​​image data as the input of the self-supervised composite task branch, and training the networks in the two branches. During the training process, the self-supervised composite task branch is responsible for providing supervision information for the local discriminator in the adversarial training branch; Input the image to be processed into the trained generator network for image generation; The specific designs of the three subtasks are as follows: In the rotation prediction task, for a batch of spliced ​​images, each image contains four image blocks, each corresponding to a rotation angle, and a pseudo label l is assigned to each image block through this rotation angle. r , l r ∈{0°, 90°, 180°, 270°}; In the position prediction task, for a batch of spliced ​​images, each image contains four image blocks that correspond to a fixed area position in the original image to which it belongs. A pseudo label l is assigned to each image block through this position information. l , l l ∈{upper left, upper right, lower left, lower right}; In the common feature extraction task, for a batch of spliced ​​images, each image contains four image blocks corresponding to an original image. The four image blocks belonging to the same original image are defined as homologous image blocks, and the features with high similarity between homologous image blocks are defined as common features. The image to be processed is input into the trained generator network for image generation, specifically: By inputting random noise into the trained generator network, high-quality generated images similar to the training set images can be obtained through forward propagation.

2. The method for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task according to claim 1, characterized in that: The stitching image data is prepared as follows: For the original images in a batch, four image blocks with a certain overlap rate ω are cropped from the upper left, upper right, lower left, and lower right regions of each image. The overlap rate ω is equal to the ratio of the side length of the overlapping part between adjacent image blocks to the side length of the original image. The obtained batch of image blocks is randomly shuffled and rotated once, and the rotation angle is randomly selected from the set R = {0°, 90°, 180°, 270°}; The size of a batch of image blocks is adjusted using bilinear interpolation so that the side length of the image block is equal to half the side length of the original image; The obtained image blocks are stitched together in groups of 4 to obtain a batch of stitched images with the same size as the original images, thereby completing the production of stitched image data.

3. The method for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task according to claim 1, characterized in that: The specific steps to build the model are: In the adversarial training branch, a local discriminator local-D and a generator G are constructed. The network structure of the local discriminator is divided into two parts, and a feature block module is added between the two. The first part receives the original image as input and extracts image features. The feature block module is responsible for processing the image features output by the first part into image block features. The second part receives the image block features as input and generates the final output of the local discriminator. The task of the local discriminator local-D is to correctly determine whether the block features are from the real image or the generated image. The loss function of this branch is recorded as L adv , which is consistent with the adversarial loss proposed in the original generative adversarial network, and its specific expression is: Where x is a real image sampled from the original dataset, P data(x) is the distribution of the real data, z is the random noise sampled from the prior distribution, P z(z) is the prior distribution, D is the local discriminator, and G is the generator; In the self-supervised composite task branch, the classifier C is built using the same network architecture as the local discriminator in the adversarial training branch. The classifier network is also divided into two parts. The network architecture of the first part is the same as the first part of local-D, and the two share network weights. The second part contains three output heads of the classifier, two of which are composed of a fully connected layer, responsible for outputting the results of the rotation prediction task and the position prediction task respectively. The third head is composed of a multi-layer perceptron with a hidden layer, responsible for outputting the results of the common feature extraction task. The total loss function of this branch is recorded as L CT .

4. The method for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task according to claim 3, characterized in that: For the self-supervised composite task branch, multiple loss functions are used for combined optimization. The total loss function L CT Defined as: L CT =L rot +L loc +L CFE ; Among them, L rot , L loc , L CFE They represent the rotation prediction task loss, position prediction task loss, and common feature extraction task loss respectively; The true rotation label of each image block in the spliced ​​image is denoted as l r_gt , the true location label is l l_gt , the rotation label predicted by the classifier for the image patch is l r , the predicted position label is l l , a set of vectors output by the multilayer perceptron in the shared feature extraction task is denoted as v1,v2,…,v {k.k＝n×4} , binary cross entropy is used to calculate the loss of rotation prediction and position prediction, and cosine similarity is used to calculate the similarity between different block features. The calculation formulas for the three task losses are as follows: L rot =CrossEntropy(l r ,l r_gt ); L loc =CorssEntropy(l l ,l l_gt ); Where τ is the temperature coefficient, n is the training batch size, C i It represents the set of subscripts corresponding to the homologous image blocks of the i-th image block, I is an indicator function, and the function value is 1 when the judgment condition is met, otherwise it is 0.

5. The method for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task according to claim 1, characterized in that: The specific steps of model training are: In the adversarial training branch, the generator G and the local discriminator local-D are trained alternately and iteratively. The input of the local discriminator is a batch of images sampled from the original image dataset. Its training goal is to correctly judge the authenticity of the image in a certain area of ​​the input image. The input of the generator is random noise. Its training goal is to output a generated image that is as realistic as possible and can deceive the local discriminator. In the self-supervised composite task branch, the input of the classifier is a batch of spliced ​​images, and the three output heads output the results of the three subtasks respectively. This branch is calculated through the total loss function L CT Train the network; The adversarial training branch and the self-supervised composite task branch are trained simultaneously. During the training process, the two branches are connected by sharing the network weights of the first part of the local discriminator local-D and the three-head classifier C. The total loss function of the model is defined as: Among them, L adv (G, D) is the adversarial training loss, L CT (C) is the self-supervised composite task loss. During the model training process, the local discriminator local-D is updated alternately with the generator G and updated simultaneously with the three-head classifier C.

6. A system for generating high-quality images by training a generative adversarial network through the introduction of a self-supervised composite task, characterized in that: A method for generating high-quality images by training a generative adversarial network through the introduction of a self-supervised composite task, as applied to any one of claims 1-5, comprising a data set module, a composite task module, a model building module, a model training module, and an image generation module; The data set module is used to prepare a training data set, and the training data set includes original image data and spliced ​​image data; the original image data is used in the training process of the adversarial training branch, and the spliced ​​image data is used in the training process of the self-supervised composite task branch; The composite task module is used to design three subtasks to form a composite task, the composite task is used to construct a self-supervised composite task branch and provide supervision information for model training, the three subtasks are a rotation prediction task, a position prediction task and a common feature extraction task, the rotation prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, the position prediction task is used to correctly determine the label corresponding to the image block contained in each spliced ​​image, and the common feature extraction task is used to first correctly determine which original image each image block belongs to, and then extract the common features between homologous image blocks; The model building module is used to construct an adversarial training branch and a self-supervised composite task branch respectively, wherein the adversarial training branch includes a local discriminator and a generator, and the self-supervised composite task branch includes a classifier with three output heads; The model training module is used to train the constructed model to obtain a trained generator network; the training is specifically: the original data set is used as the input of the adversarial training branch, the spliced ​​image data is used as the input of the self-supervised composite task branch, and the networks in the two branches are trained. During the training process, the self-supervised composite task branch is responsible for providing supervision information for the local discriminator in the adversarial training branch; The image generation module is used to input the image to be processed into the trained generator network for image generation.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the method of generating high-quality images by introducing self-supervised composite task training to generate adversarial networks as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the method for generating high-quality images by training a generative adversarial network through introducing a self-supervised composite task as described in any one of claims 1 to 5 is implemented.

Images