Adversarial generated multi-task image synthesis method, electronic device, and medium

Abstract

The application provides a kind of based on the multi-task image synthesis method of adversarial generation, electronic equipment and medium, the original image, pose variable, category variable, background variable are regarded as input, training deep learning network model, obtain image synthesis model;Deep learning network model includes generator, discriminator and Unet network;Generator includes encoder, first generation network, second generation network, third generation network.The application realizes the effective separation and semantic segmentation of foreground and background by category-background matching, so that the model generates high-semantic mask and accurate target segmentation without relying on supervision information;Through clustering analysis and mutual information quantification, the independence and diversity of features are guaranteed.The application uses a single generator, discriminator and segmentation network, reduces the computational cost, so that the method is more suitable for application in resource-constrained scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning technology, and in particular relates to a multi-task image synthesis method based on adversarial generation, an electronic device, and a medium. Background Art

[0002] With the rapid development of generative adversarial networks (GANs) in image generation, segmentation, and synthesis tasks, related technologies have achieved significant progress in foreground-background separation and semantic mask generation. These achievements are primarily due to GAN's ability to generate highly realistic images and semantic masks, which automates foreground-background distinction. However, these methods generally rely on supervised data, using pre-trained models to generate semantic masks to ensure effective foreground-background separation under supervised or weakly supervised conditions. However, this reliance significantly increases data labeling costs and lacks flexibility in complex scenes. Furthermore, due to the limitations of the GAN architecture, the generation process often suffers from "trivial solutions"—generated masks lacking meaningful semantics, often appearing either completely full or completely empty. This directly results in the model failing to effectively segment foreground and background. Traditional methods typically mitigate this problem by adding regularization, hyperparameter tuning, or incorporating supervisory signals. However, these strategies are complex and unstable, and are also extremely sensitive to the dataset's class, size, and scale. Consequently, they often fail to generalize well to unsupervised tasks.

[0003] Current image combination generation methods have made significant progress, but still have the following shortcomings:

[0004] 1. Strong Supervision Dependence: Most current methods rely on explicitly labeled training data to generate semantic masks and perform supervised learning during training. This requirement for labeled data is not only costly but also limits the model's potential for widespread application in unsupervised settings. This is especially true in scenarios where data labeling is difficult or target domain data is unavailable, significantly limiting the applicability of existing methods.

[0005] 2. Complex model structure and high computational cost: Many existing technologies, such as OneGAN, typically rely on a combination of multiple encoders, generators, and discriminators for foreground-background separation, unsupervised segmentation, and clustering, increasing computational resource requirements. This complex structure not only increases implementation difficulty but also limits the model's real-time performance and applicability in resource-constrained scenarios. Summary of the Invention

[0006] The purpose of the present invention is to address the shortcomings of the existing technology and provide a multi-task image synthesis method, electronic device and medium based on adversarial generation, which can effectively separate the foreground and background without supervised data.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A multi-task image synthesis method based on adversarial generation includes the following steps:

[0009] S1: Obtain an original image, a posture variable, a category variable, and a background variable, wherein the posture variable is sampled from a standard Gaussian distribution, and the category variable and the background variable are randomly extracted encoding vectors;

[0010] S2: using the original image, the posture variable, the category variable, and the background variable as inputs of a deep learning network model to train the deep learning network model;

[0011] The deep learning network model includes a generator, a discriminator and a Unet network; the generator is connected to the discriminator and the Unet network respectively;

[0012] The generator includes an encoder, a first generation network, a second generation network, and a third generation network, and the encoder is connected to the first generation network, the second generation network, and the third generation network respectively;

[0013] The training process of the deep learning network model includes:

[0014] The posture variables are processed by the encoder to obtain posture features;

[0015] The posture feature and the category variable are input into the first generation network to obtain a foreground image;

[0016] The posture feature and the category variable are input into the second generation network to obtain a mask image;

[0017] The posture features and the background variables are input into the third generation network to obtain a background image;

[0018] combining the foreground image, the mask image, and the background image to obtain a composite image;

[0019] Inputting the synthesized image and the original image into a discriminator to obtain a feature image;

[0020] Inputting the mask image and the original image into the Unet network to obtain a segmented image;

[0021] Constructing the loss function of the discriminator through cluster analysis and mutual information quantification;

[0022] The objective function is constructed through the loss function of the generator, the loss function of the discriminator, the loss function of the Unet network, and the adversarial loss function. The deep learning network model is trained by iterative optimization of the objective function to obtain an image synthesis model.

[0023] The present invention achieves effective separation and semantic segmentation of foreground and background through category-background matching, allowing the model to generate highly semantic masks and accurate target segmentation without relying on supervised information. The present invention ensures the independence and diversity of features through cluster analysis and mutual information quantification. The network modules achieve synergy through feature complementarity and information transfer, achieving effective extraction and differentiation of image features and improving the robustness of the model in complex scenarios. The present invention uses a single generator, discriminator, and segmentation network, reducing computational costs and making the method more suitable for resource-constrained scenarios.

[0024] Furthermore, the category variable and the background variable are the same. By setting an identity mapping between the category variable and the background variable to associate the category and the background, the generated image is made more natural and realistic.

[0025] Furthermore, the category variable and the background variable are different. By mismatching the category and background images, the model can focus on the semantic features (such as texture and shape) of the foreground object, avoiding feature confusion and over-reliance on supervisory information.

[0026] Furthermore, the expression of the composite image x is as follows:

[0027] x=F(φ,c)⊙M(φ,c)+B(φ,b)⊙(1-M(φ,c))

[0028] Among them, ⊙ represents element-by-element multiplication, c is the category variable, b is the background variable, φ is the posture feature, F(φ,c) is the foreground image, M(φ,c) is the mask image, and B(φ,b) is the background image.

[0029] Furthermore, the loss function L of the generator is bin The expression is as follows:

[0030]

[0031] in, represents the expectation of the corresponding distribution, x m is the mask image.

[0032] Furthermore, the loss function L of the discriminator is info The expression is as follows:

[0033]

[0034] Where S(·,·) is the cosine similarity, exp(·) is the exponential function used to normalize the similarity; τ is the smoothing parameter that controls the smoothness of the distribution; κ i =Ψ(c i ), κ j=Ψ(c j ), c i 、c j is a categorical variable, n is the number of categorical variables, Ψ(·) is the output of the fully connected layer, ψ κ (·) is the first output of the discriminator, is the expectation of z, c, and b, and z is the posture variable.

[0035] Furthermore, the loss function L of the Unet network is seg The expression is as follows:

[0036]

[0037] in, is the original image, and S′(·) is the output of the Unet network.

[0038] Furthermore, the adversarial loss function L adv The expression is as follows:

[0039]

[0040] in, is the expectation of x, ψ h (·) is the second output of the discriminator.

[0041] Furthermore, the first generation network includes multiple layers of cascaded convolutional layers, the second generation network includes multiple layers of cascaded convolutional layers, and the third generation network includes multiple layers of cascaded convolutional layers.

[0042] Based on the same inventive concept, the present invention further provides an electronic device, comprising:

[0043] one or more processors;

[0044] A memory having one or more programs stored thereon, which, when executed by the one or more processors, enables the one or more processors to implement the steps of a multi-task image synthesis method based on adversarial generation.

[0045] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing a computer program, which implements the steps of the multi-task image synthesis method based on adversarial generation when the computer program is executed by a processor.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention achieves effective separation and semantic segmentation of foreground and background through category-background matching, allowing the model to generate highly semantic masks and accurate target segmentation without relying on supervised information. The present invention ensures the independence and diversity of features through cluster analysis and mutual information quantification. The network modules achieve synergy through feature complementarity and information transfer, achieving effective extraction and differentiation of image features and improving the robustness of the model in complex scenarios. The present invention uses a single generator, discriminator, and segmentation network, reducing computational costs and making the method more suitable for resource-constrained scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the structure of the image synthesis model of the present invention;

[0049] Figure 2 Schematic diagram of the generator structure of the present invention;

[0050] Figure 3 Schematic diagram of the discriminator structure of the present invention;

[0051] Figure 4 Schematic diagram of the segmentation network structure of the present invention;

[0052] Figure 5 Generate images for different data sets using the method of the present invention;

[0053] Figure 6 Visualization of features extracted by various methods trained on the CUB dataset;

[0054] Figure 7 Generate images with changing category variables, fixed background variables and pose variables;

[0055] Figure 8 Generate images with varying background variables, fixed category variables, and fixed pose variables;

[0056] Figure 9 Generate images for changing background variables, category variables, and posture variables;

[0057] Figure 10 The segmentation results of different data sets. DETAILED DESCRIPTION

[0058] The present invention will be described in detail below with reference to the following embodiments. It should be noted that the embodiments and features of the embodiments may be combined unless they conflict. For ease of description, the words "upper," "lower," "left," and "right" appearing below merely indicate the directions of upper, lower, left, and right relative to the accompanying drawings and do not limit the structure.

[0059] Example 1

[0060] As Figure 1 The unsupervised image synthesis model based on the generative adversarial network of the embodiment comprises three main modules: an image decoupling module, a fine clustering module and a target segmentation module. Through the collaborative design and mutual support of the modules, efficient separation and automatic segmentation of the image foreground and background are realized, and high semantic masks and accurate target segmentation are generated. The modules realize collaborative action through feature complementation and information transmission, realize effective extraction and differentiation of image features, and improve the robustness of the model in complex scenes.

[0061] The design principles and execution steps of the method are described below from the technical solutions and work flows of each module.

[0062] 1. Image decoupling module

[0063] The main task of the image decoupling module is to generate images with clear foreground and background separation, and to provide strong semantic foreground masks in an unsupervised environment. By introducing the pose variable z, the category variable c and the background variable b, the module can control the image content and layout in an unsupervised manner.

[0064] The image decoupling module is mainly a generator, such as Figure 2 The generator includes an encoder, a foreground generation network (first generation network), a mask generation network (second generation network) and a background generation network (third generation network), and the encoder is connected with the foreground generation network, the mask generation network and the background generation network respectively.

[0065] The foreground generation network comprises a plurality of cascaded convolution layers, the mask generation network comprises a plurality of cascaded convolution layers, and the background generation network comprises a plurality of cascaded convolution layers.

[0066] 1.1 Pose, category and background variables

[0067] The pose variable z is sampled from a standard Gaussian distribution, which is used to control the angle, position or shape change of the object in the image, to ensure that the model can generate objects with different poses, thereby enhancing the diversity and authenticity of the generated images.

[0068] The category variable c is a randomly extracted encoding vector, of which only the element corresponding to the category identifier is 1, and all other elements are 0. It is used to represent the object category in the image and control the semantic features of the object such as texture and shape through one-hot encoding. This design supports the subsequent clustering task and ensures that the generated image has clear category distinction.

[0069] The background variable b is a pre-defined encoding vector used to represent the background type of the image, also in the form of one-hot encoding.

[0070] In some implementations of this embodiment, to ensure a reasonable match between the background and the category, the category and background are associated by setting an identity mapping of b = c. This mapping makes the generated image more natural and realistic, for example, generating a "duck in water" instead of a "duck in a tree."

[0071] In some other implementations of this embodiment, the inherent association between the foreground and background is broken, and a mismatch mechanism is designed. The background variable is a randomly extracted coding variable, which is different from the category variable. By mismatching the category and background images, the model's attention to the semantic features of the foreground object (such as texture and shape) is strengthened to avoid easy confusion of features and excessive reliance on supervisory information. This type of mismatch mechanism can produce unreasonable image combinations (such as "ducks in a tree"), and then use a discriminator to help the model identify which features are inherent to the foreground object and which are background features.

[0072] 1.2 Conditional Generation of Foreground, Background, and Mask

[0073] To decouple foreground and background, this embodiment uses a decoder and three conditional networks:

[0074] Decoder Φ(·): Generates shared pose features φ based on pose variables, which is formulated as: φ = Φ(z).

[0075] Foreground generation network F(·,·): Generates the corresponding foreground image x based on the category variable f , the formula is: f =F(φ,c).

[0076] Background generation network B(·,·): Generates different backgrounds x based on background variables b , the formula is: b =B(φ,b).

[0077] Mask generation network M(·,·): Generates a binary mask x m , the mask determines the synthesis area of ​​the foreground and background, and its formula is: m =M(φ,c).

[0078] 1.3 Image synthesis formula

[0079] The generated foreground, background, and mask are combined into the complete image x, which is formulated as:

[0080] x=F(φ,c)⊙M(φ,c)+B(φ,b)⊙(1-M(φ,c))

[0081] where denotes element-wise multiplication. This operation ensures that the foreground and background regions do not interfere with each other, with the foreground region generated by F(·,·) and overlaid onto the region indicated by the mask, while the background generated by B(·,·) fills the other regions.

[0082] 1.4 Binary regularization

[0083] To generate a clear mask, a binary loss function L bin is introduced to make each pixel close to 0 or 1.

[0084]

[0085] where, denotes the expectation of the corresponding distribution, and log is the natural logarithm used to calculate the cross-entropy loss.

[0086] 1.5 Workflow details

[0087] (1) The pose variable z, the category variable c, and the background variable b are used to control the image generation process.

[0088] (2) The conditional network generates foreground, background, and mask, and synthesizes a complete image.

[0089] (3) Binary regularization ensures that the mask has a clear foreground-background segmentation effect.

[0090] 2. Fine-grained clustering module

[0091] The fine-grained distance module is mainly for the discriminator, such as Figure 3 . The fine-grained clustering module performs clustering analysis on image features to achieve class division under unsupervised conditions. This module uses clustering prototype embedding, mutual information constraint, and contrastive loss to ensure that the class features have obvious distinguishability.

[0092] 2.1 Clustering prototype embedding calculation

[0093] The set of category variables is defined as I c = {c1, …, c n}, which contains all category variables c i , and n represents the total number of category variables in the data set, indicating all possible categories in the data set. I c is mapped to the clustering prototype space κ = {κ1, …, κ n}, which is calculated by the mapping function Ψ(·) composed of fully connected layers, and its formula is: κ = Ψ(I c ).

[0094] 2.2 Contrastive loss function based on mutual information

[0095] Mutual information I(A; B) represents the degree of association between two variables A and B. Mutual information I(·; ·) is used to associate visual areas and category features to enhance the clustering effect of the model. The formula is: I(x f ,x m ;c)+I(x b ; b). According to existing literature, the contrast loss L info Optimize the lower bound of mutual information to distinguish different categories of features from each other. The formula is:

[0096] I(x f ,x m ;c)+I(x b ; b) ≥ log(n)-L info ,

[0097]

[0098]

[0099] Where S(·,·) is the cosine similarity, exp(·) is the exponential function used to normalize the similarity; τ is the smoothing parameter that controls the smoothness of the distribution, and κ i , κ j represents the cluster prototype, κ i =Ψ(c i ), κ j =Ψ(c j ), c i 、c j is a categorical variable, n is the number of categorical variables, Ψ(·) represents the output of the fully connected layer, ψ κ (·) is the first output of the discriminator, which is used for image feature extraction. is the expectation of z, c, and b.

[0100] 2.3 Synthetic image x and original image Input discriminator to classify the image as real / fake and generate realistic synthetic images. The loss function of adversarial training is L adv for:

[0101]

[0102] in, is the expectation of x, ψ h (·) is the second output of the discriminator, which is used to classify the image as real or fake.

[0103] 2.4 Detailed Workflow

[0104] (1) Categorical variables are embedded in the cluster prototype space to generate cluster prototypes.

[0105] (2) Mutual information constraint ensures that the generated features and category features are closely related, and contrast loss ensures the independence of category features.

[0106] 3. Unsupervised Object Segmentation Module

[0107] The unsupervised object segmentation module provides a supervisory signal to the segmentation network through the generated mask, achieving accurate segmentation of foreground objects. When the training is close to the Nash equilibrium, the segmentation network can effectively generalize to real images.

[0108] 3.1 Segmentation Network and Regularization

[0109] like Figure 4 In this embodiment, the commonly used Unet model is used as the segmentation network S′(·). By using the generated mask x m As a supervisory signal, used to predict the original image Segmentation mask, segmentation loss function L seg The formula used to optimize the performance of S′(·) is:

[0110]

[0111] 3.2 Partition Generalization under Nash Equilibrium

[0112] When the adversarial generative model training reaches Nash equilibrium, the generated synthetic image x and the original image The distribution of is consistent. At this time, the segmentation network S′(·) trained by the generated mask can be generalized to the real image, realizing unsupervised segmentation of the real image.

[0113] 3.3 Detailed Workflow

[0114] (1) The generated masks and segmentation loss function are used to train the segmentation network.

[0115] (2) When the generative adversarial network is close to Nash equilibrium, the segmentation network can generalize to real images and improve the segmentation effect.

[0116] This embodiment forms a network to perform image synthesis, fine-grained clustering and object segmentation tasks. Given G = {Φ(·), F(·,·), B(·,·), M(·,·)}, we get the objective function as follows:

[0117]

[0118] Among them, L adv and L bin For image synthesis tasks, L info For fine-grained clustering tasks, L seg For target segmentation tasks, min · 、max. Indicates the corresponding network to optimize the loss function.

[0119] The deep learning network model is trained through iterative optimization of the objective function to obtain the image synthesis model.

[0120] This embodiment solves the common problems of strong label dependence and model complexity in traditional methods through the synergy between image decoupling, fine clustering and object segmentation modules. The image decoupling module generates reasonable foreground and background masks, and guides the discriminator to focus on the texture and shape features of the object through class-background matching and mismatching strategies. The fine-grained clustering module further utilizes mutual information and contrast loss to ensure the distinguishability between different categories, providing stable category feature information for the segmentation module, and ensuring the independence and diversity of category features. The segmentation module uses the generated mask as an unsupervised signal to achieve high-quality segmentation of real images. The three modules work together to greatly improve the accuracy and applicability of the model.

[0121] This embodiment overcomes the model's difficulty distinguishing between categories on unlabeled datasets by embedding categorical variables into cluster prototypes and utilizing cosine similarity. This embodiment streamlines the model structure, employing a single generator, discriminator, and segmentation network. Compared to traditional methods, this reduces computational cost, making it more suitable for resource-constrained scenarios.

[0122] Example 2

[0123] 1. Dataset Details

[0124] To verify the effectiveness of this method in image generation and semantic mask generation, experiments were conducted on five fine-grained datasets and one multi-object dataset. The specific datasets are described as follows:

[0125] CUB (Caltech-UCSD Birds): Contains 11,788 images of 200 bird species and provides manually annotated ground-truth segmentation masks.

[0126] Stanford Dogs: Contains 20,580 images of dogs from 120 categories, annotated with bounding boxes; 5,024 of these images have supervised ground-truth segmentation masks obtained through manual annotation methods.

[0127] Stanford Cars: Contains 16,185 images of 196 car categories, and the ground-truth segmentation masks for 5,024 images are generated using the same approximation strategy as for the Stanford Dogs dataset.

[0128] Oxford Flowers: Contains about 8,000 flower images of 102 categories, with masks generated by automated methods.

[0129] CLEVR6: A subset of the CLEVR dataset, containing objects of 6 different shapes and color combinations. 8,000 images are sampled from the dataset and scaled to 128×128 pixels.

[0130] 2. Training details

[0131] The model is trained using the Adam optimizer with a learning rate of 0.0002 and a weight decay of 0.00001.

[0132] During training, the model uses mutual information and contrastive loss to enhance category discrimination when generating foreground and background.

[0133] Each dataset is trained for 20,000 steps to verify the performance of DCS-ComGAN in generating realistic images and high semantic masks.

[0134] 3. Experimental Results

[0135] This method generates high-quality real images and clear semantic masks on multiple datasets, and can effectively separate foreground and background. Figure 5 Figure 1 shows, from top to bottom, the synthesized image, mask image, foreground image, and background image. On the CUB, Stanford Dogs, Stanford Cars, Oxford Flowers, and CLEVR6 datasets, our method generates realistic images, and the generated semantic mask accurately segments the foreground of objects. The model is able to distinguish the boundaries and texture features of objects across different categories and generate images similar to real data.

[0136] Example 3

[0137] The method of the present invention overcomes the label dependency and performance limitations of existing methods through unsupervised multivariate controlled image generation, fine-grained clustering, and complex background segmentation modules. Traditional methods often rely on supervised labels. Specifically, FineGAN generates images in layers, achieving controllable image generation based on the separation of background, target shape, and appearance, and relying on annotated bounding boxes to assist in separating the target from the background. MixNMatch further refines this approach, decoupling and encoding four key factors: background, target pose, shape, and texture. OneGAN is a complex GAN-based generative model that emphasizes the importance of background in the feature decoupling process. Labels4Free uses the difference between foreground and background to segment objects, emphasizing that accurate coupling is crucial for unsupervised segmentation. DRC proposes mutual information regularization and explores the synergy between image generation and segmentation. C3-GAN, on the other hand, uses a scene decomposition-based approach to improve the clustering effect by improving the distribution of cluster centroids. The method of the present invention does not require any external labels and directly uses independent variables to generate highly semantic samples and clearly separated class features, verifying its superiority in multi-scene tasks. The method of the present invention has significant advantages in the following aspects:

[0138] 1. Improvement of fine-grained clustering capabilities

[0139] This method improves the performance of fine-grained clustering through semantic separation and unsupervised generation strategies. Compared with methods such as C3-GAN, this method significantly improves common clustering metrics such as accuracy (Acc) and normalized mutual information (NMI) on multiple fine-grained datasets (such as CUB, Dogs, Cars, and Flowers), clearly separating features from different categories. Experimental data is shown in Table 1.

[0140] Table 1 Comparison of clustering indicators of various methods

[0141]

[0142] As shown in Table 1, this method achieved significant improvements on the CUB and Flowers datasets, with an average accuracy increase of approximately 11 percentage points compared to C3-GAN. In particular, in the detail-rich flower and bird datasets, this method demonstrated significant advantages in semantic separation and class feature clustering, demonstrating its ability to achieve high-precision clustering even under unsupervised conditions. Figure 6 Figure 1 is a visualization of the extracted features of the three methods trained on the CUB dataset, using test images from 10 randomly sampled classes, with the same color being applied to features with the same semantics. Figure 6 It can be seen that compared with other methods, this method forms a tighter category feature distribution during clustering, significantly reduces confusion between categories, and enhances clustering performance.

[0143] 2. Significant reduction in data dependency

[0144] This method relies on diverse samples generated by independent control variables (c, b, z), rather than relying on label annotation or clean background input, effectively reducing data dependence. In existing methods, a certain proportion of labeled data (such as label information for FineGAN and clean background for OneGAN) is usually required to improve the decoupling ability of the model; however, this method can separate categories, backgrounds, and postures on multiple fine-grained datasets without relying on labels, and achieve precise control. Figure 7 、 Figure 8 、 Figure 9 The generation results show that this method can achieve controllable generation of unsupervised images by adjusting the independent variables c, b, and z.

[0145] (1) The variable z controls the posture of the object. When z changes, the posture of the object changes.

[0146] (2) The variable c controls the category. When c changes, the shape and texture of the object also change, reflecting the semantic information of the object category.

[0147] (3) Variable b controls the background. Changing the background will not affect the category and posture of the object.

[0148] The images generated by this method have strong controllability, which gives it an important advantage in application scenarios that require diverse image generation.

[0149] 3. Improved unsupervised segmentation performance

[0150] This method demonstrates strong unsupervised segmentation capabilities, significantly improving common segmentation metrics such as Intersection Over Union (Iou) and Dice coefficient (Dice) in complex backgrounds and multi-object scenes. Comparative experiments demonstrate that this method outperforms other supervised and weakly supervised segmentation methods on multiple datasets, as shown in Table 2.

[0151] Table 2 Comparison of segmentation indicators of various methods

[0152]

[0153] As shown in Table 2, our method outperforms the comparison methods on various datasets, particularly when no external background annotation is required, achieving leading segmentation performance across all metrics. This demonstrates that our method can effectively separate foreground and background using the generated highly semantic masks. Figure 10 The segmentation results of different data sets are shown in the figure, from left to right: original image, real mask, and predicted mask. Figure 10From the qualitative segmentation results, we can observe that our method is superior in detail segmentation, such as the legs of birds and the rearview mirror of cars, and even exceeds the accuracy of partial truth masks.

[0154] Example 4

[0155] This embodiment provides an electronic device, including:

[0156] one or more processors;

[0157] A memory having one or more programs stored thereon, which, when executed by one or more processors, enables the one or more processors to implement the steps of a multi-task image synthesis method based on adversarial generation.

[0158] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage.

[0159] In other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors, which are not limited herein.

[0160] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of the multi-task image synthesis method based on adversarial generation are implemented.

[0161] The contents illustrated in the above embodiments should be understood as these embodiments are only used to more clearly illustrate the present invention, and are not used to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art shall fall within the scope defined by the claims attached to this application.

Claims

1. A multi-task image synthesis method based on adversarial generation, characterized in that: The following steps are involved: S1: Obtain an original image, a posture variable, a category variable, and a background variable, wherein the posture variable is sampled from a standard Gaussian distribution, and the category variable and the background variable are randomly extracted encoding vectors; S2: using the original image, the posture variable, the category variable, and the background variable as inputs of a deep learning network model to train the deep learning network model; The deep learning network model includes a generator, a discriminator and a Unet network; the generator is connected to the discriminator and the Unet network respectively; The generator includes an encoder, a first generation network, a second generation network, and a third generation network, and the encoder is connected to the first generation network, the second generation network, and the third generation network respectively; The training process of the deep learning network model includes: The posture variables are processed by the encoder to obtain posture features; The posture feature and the category variable are input into the first generation network to obtain a foreground image; The posture feature and the category variable are input into the second generation network to obtain a mask image; The posture features and the background variables are input into the third generation network to obtain a background image; combining the foreground image, the mask image, and the background image to obtain a composite image; Inputting the synthesized image and the original image into the discriminator to obtain a feature image; Inputting the mask image and the original image into the Unet network to obtain a segmented image; Construct the discriminator's loss function through cluster analysis and mutual information quantification; The objective function is constructed through the loss function of the generator, the loss function of the discriminator, the loss function of the Unet network, and the adversarial loss function. The deep learning network model is trained by iterative optimization of the objective function to obtain an image synthesis model.

2. The multi-task image synthesis method based on adversarial generation according to claim 1, characterized in that: The categorical variable and the background variable are the same, or the categorical variable and the background variable are different.

3. The multi-task image synthesis method based on adversarial generation according to claim 2, characterized in that: The expression of the composite image x is as follows: x=F(φ,c)⊙M(φ,c)+B(φ,b)⊙(1-M(φ,c)) Among them, ⊙ represents element-by-element multiplication, c is the category variable, b is the background variable, φ is the posture feature, F(φ,c) is the foreground image, M(φ,c) is the mask image, and B(φ,b) is the background image.

4. The multi-task image synthesis method based on adversarial generation according to claim 1, characterized in that: The loss function L of the generator bin The expression is as follows: in, represents the expectation of the corresponding distribution, x m is the mask image.

5. The multi-task image synthesis method based on adversarial generation according to claim 3, characterized in that: The loss function L of the discriminator info The expression is as follows: Among them, S(·,·) is the cosine similarity, exp(·) is the exponential function, τ is the smoothing parameter, κ i =Ψ(c i ), κ j =Ψ(c j ), c i 、c j is a categorical variable, n is the number of categorical variables, Ψ(·) is the output of the fully connected layer, ψ κ (·) is the first output of the discriminator, is the expectation of z, c, and b, and z is the posture variable.

6. The multi-task image synthesis method based on adversarial generation according to claim 1, characterized in that: The loss function L of the Unet network seg The expression is as follows: in, is the original image, and S′(·) is the output of the Unet network.

7. The multi-task image synthesis method based on adversarial generation according to claim 3, characterized in that: The adversarial loss function L adv The expression is as follows: in, is the expectation of x, ψ h (·) is the second output of the discriminator.

8. The multi-task image synthesis method based on adversarial generation according to claim 1, characterized in that: The first generation network includes multiple layers of cascaded convolutional layers, the second generation network includes multiple layers of cascaded convolutional layers, and the third generation network includes multiple layers of cascaded convolutional layers.

9. An electronic device, characterized in that: include: one or more processors; A memory having one or more programs stored thereon, which, when executed by the one or more processors, enables the one or more processors to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer program is stored therein, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images