Depth generation model for enhancing image diversity

Abstract

The invention relates to a depth generation model for enhancing image diversity in the field of artificial intelligence. The depth generation model comprises a style feature extractor Ex, a style specific multi-mechanism generator, an image style encoder Es and a domain specific discriminator D. The Ex samples noise vectors from Gaussian distribution, and outputs a plurality of style feature vectors through multi-layer full-connection mapping. Comprising a shared coding layer and a style-specific decoding layer, the shared coding layer extracts image content through a down-sampling network, the style-specific decoding layer adopts a multi-branch parallel up-sampling design, and each branch is fused with a style feature vector to output a corresponding style generated image. And Es, extracting the style features of the generated image, comparing the style features with the reference style feature vector, extracting the original image features, fusing the original image features with the generated sample, and comparing with the original sample. And D adopts a multi-branch output design and is used for outputting the probability that the D input image belongs to the corresponding image domain and realizing that the generated image is uniformly distributed in a plurality of image domains. The model can cross multiple image domains, and the diversity of images generated in each domain is enhanced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically to a deep generative model that enhances image diversity. Background Technology

[0002] Virtual Reality (VR) and Augmented Reality (AR) technologies provide users with immersive and interactive experiences, revolutionizing fields such as entertainment, education, and business design. High-quality image data is central to the demonstration and dissemination of VR / AR technologies. High-quality image data not only requires high-definition images but also diverse content. Cross-domain image generation technology is an important image generation technique in the VA / AR field, covering many fundamental computer vision tasks such as image inpainting, style transfer, image colorization, and super-resolution. Its main goal is to map training images from one image domain to another, thereby generating new images.

[0003] Current image generation models are mainly based on, for example Figure 1 The diagram shows a Generative Adversarial Network (GAN). A GAN model primarily consists of two deep neural network modules: a generator G and a discriminator D. First, the generator G maps a randomly sampled noise vector z to the image space Pg, outputting a fake sample G(z). Both fake and real samples are input into the discriminator D, which identifies the authenticity of the input sample D(x). Image generation models based on GANs have achieved significant progress in image quality, but the mode collapse problem inherent in GAN models makes it difficult to meet the demands of VR / AR applications for image diversity and realism. Therefore, further improvements are needed to enhance the diversity of generated images.

[0004] To address the aforementioned technical issues, current approaches primarily enhance the diversity (i.e., multimodal) of images generated by the model in two ways: one is to directly add label information to the model based on the Conditional GAN ​​(CGAN) model to assist it in generating images of a specific style. The label information is usually image labels or related paired image data, which are not only limited in quantity or difficult to obtain due to their high cost, but also make the model highly dependent on the training data, thus limiting the model's generalization ability.

[0005] Another is to use an encoder to extract image feature encoding to overcome the dependence of the model on training data. They use an encoder to extract style encoding from real images instead of fixed conditional information, effectively overcoming the dependence of the model on training data. These models decompose images into content encoding and feature encoding, and achieve diversified image generation by exchanging image style encoding, and then generate diversified images.

[0006] The above two methods mainly focus on the mutual generation between two image domains, as shown in Figure 2 (a), which is simply referred to as a two-domain image generation model. For a data set with image domains, it is necessary to train generator G to achieve mutual conversion between different image domains, as shown in Figure 2 (b), which is the problem of collaborative difficulty in cross-multi-domain generation. To solve this problem, the most direct method is to add multiple image domain-specific label information. For example, StarGAN model realizes generation tasks on multiple image domains by introducing multiple target domain labels. The target domain label information used in this method is usually fixed, and the fixed label information limits the diversity of generated images. Therefore, researchers introduce an encoder to extract domain-specific feature information from images instead of fixed label information. For example, the DRIT++ model is based on a Disentangled Representation for Image-to-Image Translation (DRIT) model for image-to-image generation, which is extended to cross-multi-domain image generation using multiple domain-specific encoders. es

[0007] In summary, in existing cross-domain image generation, enhancing image diversity generation mainly develops in two dimensions: one is to capture the multi-mode characteristics of mapping; the other is to solve the problem of collaborative difficulty in cross-multi-domain. The former refers to enhancing the performance style of generated images, and the latter refers to realizing the mutual conversion between multiple image categories. However, in existing research work, few researchers consider both dimensions. In addition, the current cross-multi-domain image generation research work considering the mapping multi-mode characteristics does not strictly distinguish between the two dimensions, and there is a problem of confusing multi-mode and multi-domain concepts. They use multiple style combinations as image domains, and the essence is still to consider the multi-mode characteristics in the two dimensions. SUMMARY

[0008] ​The present application aims to provide a deep generation model for enhancing image diversity, namely a multi-modal multi-domain image-to-image generation model (MM-I2I), to achieve simultaneous cross multiple image domains and enhance the diversity of generated images within each image domain.

[0009] To achieve the above purpose, the present application adopts the following technical scheme: a deep generation model for enhancing image diversity, comprising a style feature extractor Ex, a style-specific multi-mechanism generator , an image style encoder Es and a domain-specific discriminator D; The style feature extractor Ex is used to randomly sample a noise vector from a Gaussian distribution, and output a plurality of style feature vectors through a multi-layer fully connected mapping; The style-specific multi-mechanism generator includes a shared encoding layer and a style-specific decoding layer, the shared encoding layer extracts image content through a down-sampling network, and the style-specific decoding layer adopts a multi-branch parallel up-sampling design, each branch corresponds to fusing one of the style feature vectors to output a generated image of corresponding style; The image style encoder Es is used to extract the style of the generated image sample to obtain the generated image style feature, and compare it with the reference style feature vector, and is used to extract the features of the original image, and after fusing it with the generated sample, it is compared with the original sample; The domain-specific discriminator D adopts a multi-branch output design, each branch corresponds to an image domain, and is used to output the probability that the input image of the domain-specific discriminator D belongs to the corresponding image domain; and the domain-specific discriminator D and the style-specific multi-mechanism generator between them are supervised by the maximum information entropy confrontation training to realize the uniform distribution of the generated images in multiple image domains.

[0010] The beneficial effects of the present scheme: The present scheme realizes efficient cooperation through the architecture of "shared encoding layer + style-specific decoding layer": the shared encoding layer reuses content feature extraction capability, and the style decoding layer adapts to different domain styles with a multi-branch design, without the need to train a generator for each domain, greatly reducing the parameter size and training cost, realizing efficient conversion of N domains, and significantly improving the practicality of multi-domain scenarios. At the same time, the model strictly distinguishes the conceptual boundaries between "multi-domain conversion" and "single-domain diversity", the shared encoding layer focuses on content feature extraction to support cross-domain conversion (multi-domain dimension), and the style-specific decoding layer realizes style variant generation within the same domain by fusing diversified style vectors through multi-branch, completely solving the problem of concept confusion in the prior art, and simultaneously meeting the dual needs of cross-domain conversion and single-domain diversity enhancement. In terms of diversity improvement and data dependence breakthrough, the model generates rich style feature vectors from Gaussian noise through the style feature extractor, providing the generator with diversified style sources that are independent of training data, and the multi-branch decoding layer of the generator can fuse different style vectors to output multiple style variants of the same content within the same domain, greatly improving the diversity of generated images and the generalization ability of the model. In addition, the image style encoder ensures accurate style transfer and content fidelity through double contrast (consistency of generated image style and reference style, and content preservation degree of original image and generated image), and the domain-specific discriminator optimizes the detail realism through adversarial training, which, in combination, significantly improves the clarity, style consistency and content integrity of the generated image, meeting the requirements of high-quality images in VR / AR scenarios. Finally, the modular design of the model has good universality and can be directly applied to various cross-domain image generation tasks such as image inpainting, style transfer, image coloring, super-resolution, etc., without the need for significant adjustments for specific tasks, reducing the technical landing threshold and having a wide range of applications.

[0011] Preferably, as an improvement, the input noise vector dimension of the style feature extractor Ex is 16, and the output style feature vector dimension is 64.

[0012] Preferably, as an improvement, the shared encoding layer is constructed with a down-sampling residual block with a convolution kernel of 3x3 and a step of 1, each residual block uses an Instance Normalization normalization function and a LeakyReLU function activation, the shared encoding layer includes 6 layers of networks, and the output of each layer is 32, 64, 128, 256, 512, and 512 respectively; the style-specific decoding layer is constructed with an up-sampling residual block with a convolution kernel of 3x3 and a step of 1, each residual block uses a LeakyReLU function activation, and different style feature vectors are fused through an AdaIN normalization function.

[0013] Preferably, as an improvement, the style-specific multi-mechanism generator The style-specific decoding layer includes a 7-layer network, and each layer outputs a final output corresponding to the generated image of the style after fusing style feature vectors.

[0014] Preferably, as an improvement, the style-specific multi-mechanism generator and the domain-specific discriminator D in the mutual adversarial learning process, the style-specific multi-mechanism generator generates realistic images, and the adversarial training between the two is through the adversarial loss L adv Implementation: ; Wherein, E x,yj , E x,yj,si respectively represent the mathematical expectation about the independent variable x,y j and the mathematical expectation about x,y j ,s i , s i is the feature extracted by the style feature extractor Ex, D yj (*) indicates the output of the domain-specific discriminator D corresponding to the arbitrary input sample, and the output value indicates the probability that the input sample comes from the image domain y j ; the style-specific multi-mechanism generator generates realistic images to confuse the domain-specific discriminator D with the purpose of minimizing the adversarial loss ; the domain-specific discriminator D distinguishes between real samples and generated fake samples by maximizing the adversarial loss.

[0015] Preferably, as an improvement, the style-specific multi-mechanism generator maximizes the information entropy L H to encourage the generated samples I i to be uniformly distributed in multiple image domains, and the information entropy L H is maximized as follows:

[0016] Wherein, N is the number of image domains, I i= G sm ( x , s i ) represents the generated sample, x and represents the input sample. D yj (I i )= D yj ( G sm ( x , s i ) indicates that the generated samples come from the image domain. y j The probability of.

[0017] Preferably, as an improvement, a diversity loss function is used to encourage the model to generate diverse images. The diversity loss function is as follows: ; in, Represents two images in image space and The distance between them Represents two eigenvectors in the feature space S i and S j The distance between them; This loss increases image diversity by maximizing the ratio of distance between images to distance between features, or by directly maximizing the distance between images, thereby capturing diverse image patterns.

[0018] Preferably, as an improvement, style reconstruction loss L sty-rec Ensure that the generated samples accurately represent the random style features extracted by the style feature extractor Ex. s i Generate samples I i Will be by image style encoder Es Extracting style features E s ( I i ), and its original characteristics s i Comparison; Style Reconstruction Loss L sty-rec : ; in, Indicates the input image Post-image style encoder Es Output results Indicates the input and generated image Post-image style encoder Es The output result.

[0019] Preferably, as an improvement, a cycle consistency loss L is adopted. cycSupervised style-specific multi-mechanism generator of training, cycle consistency loss L cyc As follows: .

[0020] Preferably, as an improvement, the style feature extractor Ex, style-specific multi-mechanism generator , image style encoder Es and domain-specific discriminator D are coordinated with each other through the loss function L adv , L H , L sd , L sty-rec and L cyc The total objective function is as follows: ; Where, λ H , λ Sd , λ sty-rec and λ cyc is a non-negative hyperparameter that controls the weight of the corresponding term in the total objective function. During model training, it is adjusted according to experience.

[0021] Advantages: In the deep generative model for enhancing image diversity of the present application, the style-specific multi-mechanism generator can simultaneously fuse multiple features to generate multiple images; the domain-specific discriminator D increases the classification function to identify the category of the input image of the module. Through the domain-specific discriminator D, multiple image domains can be converted through the training of one model. The multi-mechanism style-specific multi-mechanism generator can simultaneously fuse multiple image features to generate diversified image data. The diversity of generated images is enhanced from two dimensions of multi-domain and multi-modal.

[0022] Application field: The deep generative model for enhancing image diversity can be used in medical and agricultural fields. For example, in the synthesis of medical images, corresponding CT images can be generated from MRI images, thereby alleviating the problem of insufficient number of high-tech and high-cost labeled data.

[0023] Economic benefits: The deep generative model for enhancing image diversity can be used in the e-commerce field, such as virtual fitting of clothes, which can reduce the cost of manual retouching. According to industry case estimates, the cost of generating a single commodity image is reduced from 50 yuan per image to 5 yuan per image, with a cost savings of 90%.

[0024] Technical effects: The deep generative model for enhancing image diversity can train a style-specific multi-mechanism generator The transformation between multiple image domains is realized, and the diversity within the image domain is enhanced. Experiments prove that the method greatly improves the diversity of the existing SG-I2IT model on the AFHQ multi-domain image dataset. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The existing technology is a generative adversarial network (GAN) model.

[0026] Figure 2 The existing technology is a cross-domain image generation model framework; wherein (a) is a dual-domain image generation model, and (b) is a multi-domain image generation model.

[0027] Figure 3 The model framework of the deep generative model for enhancing image diversity in the embodiment of the application.

[0028] Figure 4 The generated cat domain image sample comparison chart; wherein (a) is a dog domain to cat domain generated image sample, and (b) is a wild animal domain to cat domain generated image sample.

[0029] Figure 5 The generated wild animal domain image sample comparison chart; wherein (a) is a cat domain to wild animal domain generated image sample, and (b) is a dog domain to wild animal domain generated image sample.

[0030] Figure 6 The network structure diagram of the style feature extractor Ex.

[0031] Figure 7 The network structure diagram of the domain-specific discriminator D. DETAILED DESCRIPTION

[0032] The following is further described in detail through specific embodiments: Embodiment: a deep generative model (MM-I2I model) for enhancing image diversity 1. Model framework The MM-I2I model structure block diagram is shown in the accompanying Figure 3 The style feature extractor Ex (feature extractor in the figure), the style-specific multi-mechanism generator (style-specific multi-mechanism generator in the figure), the image style encoder Es (image style encoder in the figure), and the domain-specific discriminator D (discriminator in the figure) are included. The innovation of the patent lies in the style-specific multi-mechanism generator and the domain-specific discriminator D. The following is an introduction to the four modules: (1) Style feature extractor Ex To enhance the diversity of image style, the MM-I2I model (deep generative model for enhancing image diversity) proposed in this paper uses a fully connected style feature extractor Ex to extract image features. The model randomly samples a noise vector of dimension 16 from a Gaussian distribution z i As input, through multi-layer fully connected mapping, a 64-dimensional style feature vector is output s . The style feature extractor Ex uses a multi-branch output design to output multiple style feature vectors s i to enhance the diversity of generated sample styles. Its network structure is shown in Figure 6 .

[0033] (2) Style-specific multi-mechanism generator

[0034] To enhance the diversity of model-generated image samples, the MM-I2I model designs a style-specific multi-mechanism generator , which uses a multi-branch output design specific to style features, with the number of branches determined according to the number of reference styles. The style-specific multi-mechanism generator G sm includes two parts: a shared encoding layer and a style-specific decoding layer. The former extracts image content through a 6-layer downsampling network, and the latter uses a multi-branch parallel upsampling design. During decoding, each branch fuses a style feature s i . The network structure of the style-specific multi-mechanism generator is shown in Table 1. In the encoding process, a shared network layer is constructed using downsampling residual blocks with a convolution kernel of 3x3 and a stride of 1. Each residual block uses an Instance Normalization (IN) normalization function and a LeakyReLU function activation. The output of each layer is 32, 64, 128, 256, 512, and 512, respectively. In the decoding process, multiple decoding branches are constructed using upsampling residual blocks with a convolution kernel of 3x3 and a stride of 1. Each residual block uses a LeakyReLU function activation and fuses different style features s 1 and s 2 through an AdaIN normalization function, and then outputs generated images s 1 and s 2 corresponding to style features I 1 and I 2, respectively.

[0035] Table 1 Network structure of style-specific multi-mechanism generator

[0036] ​(3) Image style encoder Es Image style encoder in MM-I2I model Es There are two main purposes: first, to ensure that the generated image samples effectively integrate the feature styles extracted by the style feature extractor Ex s i , the MM-I2I model uses an image style encoder Es to extract the style of the generated image samples E s I i , and compare it with the feature style it refers to s i Second, to ensure that the generated samples retain some features of the original image, the model uses an image style encoder Es to extract features from the original image E s x and then compare them with the generated samples after fusion, expecting the generated samples to retain some features of the original image.

[0037] (4) Domain-specific discriminator D The domain-specific discriminator D of the MM-I2I model uses a multi-branch output design, with each branch corresponding to an image domain. The first j branch outputs the probability y j of the input image belonging to the image domain D yj (*) of the domain-specific discriminator D. In addition, to ensure that the generated image can cover all image domains, the model uses maximum information entropy to supervise the adversarial training between the domain-specific discriminator D and the style-specific multi-mechanism generator , encouraging the model to generate images that are uniformly distributed in multiple image domains. Its network structure is shown in Figure 7 .

[0038] 2. Mathematical reasoning This patent uses five loss functions to form the training objective function of the model, connecting the above four modules to realize mutual cooperation between the modules, and thus achieving the purpose of enhancing the diversity of generated images.

[0039] (1) Adversarial training loss The training of the MM-I2I model is based on the adversarial training idea of GAN. In the mutual adversarial learning process of the style-specific multi-mechanism generator and the domain-specific discriminator D, the style-specific multi-mechanism generator is trained to generate realistic images. The adversarial training between the two is achieved through the adversarial loss L adv : ​​ ; Among them, E x,yj E x,yj,si They represent the independent variables respectively. x,y j Mathematical expectation and about x,y j ,s i The mathematical expectation, s i Features extracted by the style feature extractor Ex D yj (*) refers to the output of the domain-specific discriminator D corresponding to any input sample*, whose output value indicates that the input sample comes from the image domain. y j The probability of; style-specific multi-mechanism generator To minimize the damage to the enemy The goal is to generate realistic images to confuse the domain-specific discriminator D; the domain-specific discriminator D then distinguishes between real samples and generated fake samples by maximizing the adversarial loss.

[0040] (2) Maximize information entropy Style-specific multi-mechanism generator The goal is to generate realistic images G sm ( x , s i The deceptive domain-specific discriminator D is misled into believing that a sample has been generated. G sm ( x , s i (From the image domain) y j Simultaneously, the model aims to generate image samples corresponding to multiple image domains, meaning that the generated samples have an equal probability of corresponding to each image domain. Treating the identification of an input sample as originating from a certain image domain by the domain-specific discriminator D as a random event, then the image... x (Referring to any input image) from the image domain y j The probability of this happening is: ; in, N This represents the number of image domains. The information content of this event is: ; So, the information entropy of this random event L H for: ; Input image of the domain-specific discriminator D x Training image that can be real x Model-generated sample that can also be I i When the input sample is a generated sample I i = G sm ( x , s i ), the maximum information entropy function LH is:

[0041] wherein, N is the number of image domains, I i= G sm ( x , s i ) represents a generated sample, x and the input sample; D yj ( I i )= D yj ( G sm ( x , s i )) represents the probability that the generated sample comes from the image domain y j .

[0042] Style-specific multi-mechanism generator By maximizing the information entropy H ( D yj ( Ii )), the generated sample I i is encouraged to be uniformly distributed in multiple image domains, thereby consolidating the model to achieve the goal of mapping across multiple image domains.

[0043] (3) Diversity loss function To encourage the model to generate diverse images, the MM-I2I model adopts a diversity loss function: ; wherein, represents the distance between two images and in the image space, and represents the distance between two feature vectors Si and S j the distance between the images; The loss increases the diversity of the images by maximizing the ratio of the inter-image distance to the inter-feature distance, or directly maximizing the distance between the images, thereby capturing the image pattern of image diversity.

[0044] (4) Style reconstruction loss The generated sample accurately expresses the random style feature s i It can be ensured that the model captures the random and diverse image pattern. To ensure that the generated sample accurately expresses the random style feature extracted by the style feature extractor Ex s i , the generated sample I i will be encoded by the image style encoder Es to extract the style feature E s ( I i ), and compared with its original feature s i . This goal is achieved through the style reconstruction loss L sty-rec : .

[0045] wherein, denotes the input image the output result of the image style encoder Es after denotes the output result of the image style encoder after inputting the generated image Es .

[0046] (5) Cycle consistency loss To ensure that the generated sample still retains the identity content features of the original image, the MM-I2I model adopts the cycle consistency loss L cyc which is widely used in existing image translation models to supervise the training of the style-specific multi-mechanism generator: .

[0047] The cycle consistency loss L cyc encourages the model to retain the identity content attributes of the original image by minimizing the distance between the original image style guided generated sample and the original sample.

[0048] (6) Total objective of the model In summary, the MM-I2I model consists of four modules: the style feature extractor Ex, the style-specific multi-mechanism generator​ , image style encoder Es and domain-specific discriminator D. These four modules are coordinated with each other through five loss functions: adversarial loss , maximizing information entropy , diversity loss function , style reconstruction loss and cycle consistency loss . Therefore, the total objective function of the MM-I2I model is: ; wherein, λ H , λ Sd , λ sty-rec and λ cyc is a non-negative hyperparameter that controls the weight of the corresponding term in the total objective function. In the effect verification process of the present application, the experiment was carried out on a server configured with 1 NVIDIA GeForce RTX 2080Ti GPU, and the total objective function parameter was set to λ H =1 , λ Sd =1 , λ sty-rec =0.3, λ cyc =0.2.

[0049] 3. Effect verification 3.1 Dataset In order to study the intra-domain diversity of cross-multi-domain image generation, the experiment was verified on an AFHQ dataset with 3 image domains. It contains three image domains of cats, dogs and wild animals. Each image domain has 5000 images, of which each domain has 4500 images in the training set and 500 images in the test set.

[0050] 3.2 Comparative method The MM-I2I model proposed in this paper further studies the intra-domain diversity of generated samples on the basis of cross-multi-domain image translation. Therefore, the proposed model is mainly compared with the StarGAN-v2 model and the SG-I2IT model, which are the latest cross-multi-domain image translation models.

[0051] 3.3 Evaluation index The evaluation of the generated samples by the model in this paper is carried out from two aspects: image diversity and image quality. For the evaluation of image diversity, this paper mainly uses the LPIPS evaluation index. This index mainly measures the diversity of samples by calculating the distance between the features of generated samples. It is the only quantitative evaluation index that is widely used to measure sample diversity. For the evaluation of the quality of generated images, this paper mainly uses the FID evaluation index. This index has been widely used in various image translations to measure the quality of generated samples and is the most recognized image quality evaluation index in image translation.

[0052] 3.4 Experimental results (1) Visual experimental results Figure 4 Cat domain images generated by the StarGAN-v2 model, the SG-I2IT model and the MM-I2I model proposed in this paper according to the input images are given. Figure 4 (a) Generated images from the dog domain to the cat domain by the three models are given, Figure 4 (b) Cat domain images generated by the three models according to the input wild animals are given. By comparing the differences between the cat domain images generated by each model, it can be seen that the cat domain images generated by the StarGAN-v2 model are greatly affected by the color of the input image, resulting in a strong similarity in color among the generated cat domain images. The color of the cat domain image samples generated by the SG-I2IT model is relatively rich, while the MM-I2I model generates cat domain images with not only large color differences but also large facial line differences. Therefore, the MM-I2I model generates the most diverse cat domain images, followed by the SG-I2IT model, and the StarGAN-v2 model has relatively low diversity in the cat domain.

[0053] Figure 5 Wild animal domain image samples generated by the three models are shown. Figure 5 (a) Wild animal domain image samples generated by the three models according to the cat domain images are given, Figure 5 (b) Wild animal domain image samples generated by the three models according to the input dog domain images are given. By comparing the quality of the wild animals generated by the three models, it can be seen that the SG-I2IT model generates the least artifacts in the wild animal images, and its quality is the best; the quality of the images generated by the MM-I2I model is only second to that of the SG-I2IT model; the StarGAN-v2 model has the most artifacts and the lowest quality. By comparing the diversity of the generated samples by the three models, the SG-I2IT model generates only lions, which is relatively single; the StarGAN-I2I model generates tigers and leopards, which has higher diversity than the SG-I2IT model; the MM-I2I model generates lions, tigers and wolves, which has the highest diversity.

[0054] (2) Quantitative data analysis Table 2 shows the quantitative quality evaluation and diversity evaluation results of the three models in generating images on the AFHQ dataset. Among them, the lower the FID value, the higher the quality of the generated samples, and the higher the LPIPS value, the higher the diversity of the generated samples. The table shows the measurement results of the three methods in each image domain and the results on the AFHQ dataset. In terms of the quality of the generated samples, the MM-I2I model generates the lowest FID value in the cat and dog domains, and the quality of the generated samples is the highest. In the wildlife domain, the SG-I2IT model generates the lowest FID value, and the quality of the generated images is the best, followed by the MM-I2I model. On the whole dataset, the MM-I2I model is slightly better than the SG-I2IT model. In terms of the diversity of the generated samples, in the three image domains, the MM-I2I model has the highest LPIPS value, indicating that it has the best diversity performance.

[0055] Table 2 Quantitative comparison of generated samples in each image domain on the AFHQ dataset

[0056] The above is only an embodiment of the present application, and the specific technical solutions and / or common knowledge of the scheme are not described in detail. It should be noted that for those skilled in the art, without departing from the technical solutions of the present application, some modifications and improvements can be made, which should also be considered as the protection scope of the present application, and these will not affect the effect and practicality of the patent. The protection scope of the present application should be subject to the content of its claims, and the specific implementation mode and the like in the specification can be used to explain the content of the claims.

Claims

1. A deep generative model for enhancing image diversity, characterized in that: Includes style feature extractor Ex and style-specific multi-mechanism generator. Image style encoder Es Domain-specific discriminant D; The style feature extractor Ex is used to randomly sample noise vectors from a Gaussian distribution and output multiple style feature vectors through a multi-layer fully connected mapping. The style-specific multi-mechanism generator It includes a shared encoding layer and a style-specific decoding layer. The shared encoding layer extracts image content through a downsampling network. The style-specific decoding layer adopts a multi-branch parallel upsampling design, with each branch corresponding to a fused style feature vector to output a generated image of the corresponding style. The image style encoder Es It is used to extract style from the generated image samples, obtain the style features of the generated image, and compare them with the reference style feature vector. It is also used to extract features from the original image, fuse them with the generated sample, and then compare them with the original sample. The domain-specific discriminator D employs a multi-branch output design, with each branch corresponding to an image domain, used to output the probability that the input image belongs to the corresponding image domain; and the domain-specific discriminator D and the style-specific multi-mechanism generator... Adversarial training with maximum information entropy supervision is used to achieve uniform distribution of generated images across multiple image domains.

2. The deep generative model for enhancing image diversity according to claim 1, characterized in that: The style feature extractor Ex has an input noise vector dimension of 16 and an output style feature vector dimension of 64.

3. A deep generative model for enhancing image diversity according to claim 2, characterized in that: The shared coding layer is constructed using downsampled residual blocks with a 3×3 kernel and a stride of 1. Each residual block is activated by the InstanceNormalization normalization function and the LeakyReLU function. The shared coding layer consists of 6 layers with outputs of 32, 64, 128, 256, 512, and 512, respectively. The style-specific decoding layer is constructed using upsampled residual blocks with a 3×3 kernel and a stride of 1. Each residual block is activated by the LeakyReLU function and different style feature vectors are fused using the AdaIN normalization function.

4. The deep generative model for enhancing image diversity according to claim 3, characterized in that: The style-specific multi-mechanism generator The style-specific decoding layer consists of 7 network layers. The outputs of each layer are fused with style feature vectors, and the final output is a generated image of the corresponding style.

5. A deep generative model for enhancing image diversity according to any one of claims 1 to 4, characterized in that: Style-specific multi-mechanism generator During the adversarial learning process with the domain-specific discriminator D, a style-specific multi-mechanism generator is trained. Generate realistic images, and the adversarial training between the two uses adversarial loss L. adv accomplish: ； Among them, E x,yj E x,yj,si They represent the independent variables respectively. x,y j Mathematical expectation and about x,y j ,s i The mathematical expectation, s i Features extracted by the style feature extractor Ex D yj (*) refers to the output of the domain-specific discriminator D corresponding to any input sample*, whose output value indicates that the input sample comes from the image domain. y j The probability of; style-specific multi-mechanism generator To minimize the damage to the enemy The goal is to generate realistic images to confuse the domain-specific discriminator D; the domain-specific discriminator D then distinguishes between real samples and generated fake samples by maximizing the adversarial loss.

6. A deep generative model for enhancing image diversity according to claim 5, characterized in that: Style-specific multi-mechanism generator By maximizing information entropy L H Encourage the generation of samples I i Evenly distributed across multiple image domains to maximize information entropy. L H The function is as follows ： in, N The number of image domains, I i= G sm ( x , s i () indicates the generation of samples. x Indicates the input sample; D yj ( I i )= D yj ( G sm ( x , s i ) indicates that the generated samples come from the image domain. y j The probability of.

7. A deep generative model for enhancing image diversity according to claim 6, characterized in that: A diversity loss function is used to encourage the model to generate diverse images. The diversity loss function is shown below: ； in, Represents two images in image space and The distance between them Represents two eigenvectors in the feature space S i and S j The distance between them; This loss increases image diversity by maximizing the ratio of distance between images to distance between features, or by directly maximizing the distance between images, thereby capturing diverse image patterns.

8. A deep generative model for enhancing image diversity according to claim 7, characterized in that: Loss L from style reconstruction sty-rec Ensure that the generated samples accurately represent the random style features extracted by the style feature extractor Ex. s i Generate samples I i Will be by image style encoder Es Extracting style features E s ( I i ), and its original characteristics s i Comparison; Style Reconstruction Loss L sty-rec : ； in, Indicates the input image Post-image style encoder Es Output results Indicates the input and generated image Post-image style encoder Es The output result.

9. A deep generative model for enhancing image diversity according to claim 8, characterized in that: Using cycle consistency loss L cyc Supervised style-specific multi-mechanism generator Training, cycle consistency loss L cyc as follows: 。 10. A deep generative model for enhancing image diversity according to claim 9, characterized in that: The style feature extractor Ex and the style-specific multi-mechanism generator Image style encoder Es The domain-specific discriminator D passes through the loss function. L adv 、L H 、L sd 、L sty-rec and L cyc With mutual coordination, the overall objective function is as follows: ； in, λ H , λ Sd , λ sty-rec and λ cyc is a non-negative hyperparameter that controls the weight of the corresponding term in the overall objective function.

Images