Multitask image synthesis method based on adversarial generation, electronic equipment and medium

Abstract

The invention provides a multitask image synthesis method based on adversarial generation, electronic equipment and a medium, and the method comprises the steps: taking an original image, an attitude variable, a category variable and a background variable as input, training a deep learning network model, and obtaining an image synthesis model; the deep learning network model comprises a generator, a discriminator and a Unet network; the generator comprises an encoder, a first generation network, a second generation network and a third generation network. According to the method, effective separation and semantic segmentation of the foreground and the background are realized through category-background matching, so that the model generates a high-semantic mask and accurate target segmentation without depending on supervision information; and through clustering analysis and mutual information quantification, the independence and diversity of the features are ensured. According to the method, a single generator, a discriminator and a segmentation network are adopted, so that the calculation cost is reduced, and the method is more suitable for being applied to a resource-limited scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a multi-task image synthesis method based on adversarial generation, an electronic device and a medium. Background Art

[0002] With the rapid development of generative adversarial networks (GANs) in image generation, segmentation, and synthesis tasks, related technologies have made significant progress in foreground-background separation and semantic mask generation. These achievements are mainly due to the ability of GANs to generate highly realistic images and semantic masks, which makes the distinction between foreground and background automatic. However, these methods generally rely on supervised data, and use pre-trained models to generate semantic masks to ensure effective separation of foreground and background under supervised or weakly supervised conditions. However, this reliance significantly increases the cost of data labeling and cannot be flexibly responded to complex scenes. In addition, due to the limitations of the GAN structure, the generation process often has the problem of "trivial solution", that is, the generated mask lacks actual semantics, usually in the form of full or empty, which directly causes the model to lose effective segmentation of the foreground and background. Traditional methods usually alleviate this problem by adding regularization, hyperparameter tuning, or adding supervisory signals, but these strategies are not only complex and unstable, but also extremely sensitive to the category, size, and scale of the dataset. Therefore, when generalized to unsupervised tasks, the effect is often not ideal.

[0003] The current image combination generation methods have made important progress, but still have the following shortcomings:

[0004] 1. Strong supervision dependency: Most current methods rely on explicitly labeled training data to generate semantic masks and perform supervised learning during training. This requirement for labeled data is not only costly, but also limits the potential for widespread application of the model in unsupervised environments, especially in scenarios where data labeling is difficult or target domain data is unavailable, which significantly limits the applicability of existing methods.

[0005] 2. Complex model structure and high computational cost: Many existing technologies, such as OneGAN, usually rely on a combination of multiple encoders, generators, and discriminators for foreground-background separation, unsupervised segmentation, and clustering, which increases the demand for computing resources. This complex structure not only increases the difficulty of implementation, but also limits the real-time performance of the model and its applicability in resource-constrained scenarios. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies in the prior art and provide a multi-task image synthesis method, electronic device and medium based on adversarial generation, which can effectively separate the foreground and background without supervised data.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A multi-task image synthesis method based on adversarial generation includes the following steps:

[0009] S1: obtaining an original image, a posture variable, a category variable, and a background variable, wherein the posture variable is sampled from a standard Gaussian distribution, and the category variable and the background variable are randomly extracted encoding vectors;

[0010] S2: using the original image, the posture variable, the category variable, and the background variable as inputs of a deep learning network model to train the deep learning network model;

[0011] The deep learning network model includes a generator, a discriminator and a Unet network; the generator is connected to the discriminator and the Unet network respectively;

[0012] The generator includes an encoder, a first generation network, a second generation network, and a third generation network, and the encoder is connected to the first generation network, the second generation network, and the third generation network respectively;

[0013] The training process of the deep learning network model includes:

[0014] The posture variable is processed by the encoder to obtain a posture feature;

[0015] The posture feature and the category variable are input into the first generation network to obtain a foreground image;

[0016] The posture feature and the category variable are input into the second generation network to obtain a mask image;

[0017] The posture features and the background variables are input into the third generation network to obtain a background image;

[0018] Combining the foreground image, the mask image and the background image to obtain a composite image;

[0019] Inputting the synthesized image and the original image into a discriminator to obtain a feature image;

[0020] Inputting the mask image and the original image into the Unet network to obtain a segmented image;

[0021] Constructing the loss function of the discriminator through cluster analysis and mutual information quantification;

[0022] The objective function is constructed through the loss function of the generator, the loss function of the discriminator, the loss function of the Unet network, and the adversarial loss function. The deep learning network model is trained by iterative optimization of the objective function to obtain an image synthesis model.

[0023] The present invention realizes effective separation and semantic segmentation of foreground and background through category-background matching, so that the model can generate highly semantic masks and accurate target segmentation without relying on supervised information; the present invention ensures the independence and diversity of features through cluster analysis and mutual information quantification. The network modules achieve synergy through feature complementarity and information transmission, realize effective extraction and distinction of image features, and improve the robustness of the model in complex scenes; the present invention adopts a single generator, discriminator and segmentation network, which reduces the computational cost, making the method more suitable for application in resource-constrained scenarios.

[0024] Furthermore, the category variable and the background variable are the same. By setting the identity mapping between the category variable and the background variable to associate the category and the background, the generated image is made more natural and realistic.

[0025] Furthermore, the category variable and the background variable are different. The category and background mismatch images are used to strengthen the model's attention to the semantic features (such as texture and shape) of the foreground object, avoiding easy feature confusion and excessive reliance on supervisory information.

[0026] Furthermore, the expression of the composite image x is as follows:

[0027] x=F(φ,c)⊙M(φ,c)+B(φ,b)⊙(1-M(φ,c))

[0028] Among them, ⊙ represents element-by-element multiplication, c is the category variable, b is the background variable, φ is the posture feature, F(φ,c) is the foreground image, M(φ,c) is the mask image, and B(φ,b) is the background image.

[0029] Furthermore, the loss function L of the generator is bin The expression is as follows:

[0030]

[0031] in, represents the expectation of the corresponding distribution, x m is the mask image.

[0032] Furthermore, the loss function L of the discriminator is info The expression is as follows:

[0033]

[0034] Among them, S(·,·) is the cosine similarity, exp(·) is the exponential function used to normalize the similarity; τ is the smoothing parameter that controls the smoothness of the distribution; κ i =Ψ(c i ), κ j=Ψ(c j ), c i 、c j is a categorical variable, n is the number of categorical variables, Ψ(·) is the output of the fully connected layer, ψ κ (·) is the first output of the discriminator, is the expectation of z, c, b, and z is the posture variable.

[0035] Furthermore, the loss function L of the Unet network is seg The expression is as follows:

[0036]

[0037] in, is the original image, and S′(·) is the output of the Unet network.

[0038] Furthermore, the adversarial loss function L adv The expression is as follows:

[0039]

[0040] in, is the expectation of x, ψ h (·) is the second output of the discriminator.

[0041] Furthermore, the first generation network includes multiple layers of cascaded convolutional layers, the second generation network includes multiple layers of cascaded convolutional layers, and the third generation network includes multiple layers of cascaded convolutional layers.

[0042] Based on the same inventive concept, the present invention also provides an electronic device, including:

[0043] one or more processors;

[0044] A memory having one or more programs stored thereon, which, when executed by the one or more processors, enables the one or more processors to implement the steps of a multi-task image synthesis method based on adversarial generation.

[0045] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing a computer program, which implements the steps of a multi-task image synthesis method based on adversarial generation when executed by a processor.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention realizes effective separation and semantic segmentation of foreground and background through category-background matching, so that the model can generate highly semantic masks and accurate target segmentation without relying on supervised information; the present invention ensures the independence and diversity of features through cluster analysis and mutual information quantification. The network modules achieve synergy through feature complementarity and information transmission, realize effective extraction and distinction of image features, and improve the robustness of the model in complex scenes; the present invention adopts a single generator, discriminator and segmentation network, which reduces the computational cost, making the method more suitable for application in resource-constrained scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a structural schematic diagram of the image synthesis model of the present invention;

[0049] Figure 2 It is a schematic diagram of the generator structure of the present invention;

[0050] Figure 3 Schematic diagram of the discriminator structure of the present invention;

[0051] Figure 4 It is a schematic diagram of the segmentation network structure of the present invention;

[0052] Figure 5 Generate images of the method of the present invention on different data sets;

[0053] Figure 6 Visualization of features extracted by various methods trained on the CUB dataset;

[0054] Figure 7 Generate images for changing the category variable, fixing the background variable and the pose variable;

[0055] Figure 8 Generate images with changing background variables, fixed category variables and pose variables;

[0056] Fig. 9 Generate images for changing background variables, category variables and posture variables;

[0057] Fig.10 The segmentation results of different data sets. DETAILED DESCRIPTION

[0058] The present invention will be described in detail below in conjunction with the embodiments. It should be noted that the embodiments and features of the embodiments of the present invention can be combined with each other without conflict. For the convenience of description, if the words "upper", "lower", "left" and "right" appear in the following, they only indicate that the upper, lower, left and right directions are consistent with the drawings themselves, and do not limit the structure.

[0059] Example 1

[0060] like Figure 1 The unsupervised image synthesis model based on the generative adversarial network of this embodiment consists of three main modules: image decoupling module, fine clustering module and target segmentation module. Through the collaborative design and mutual support of each module, efficient separation and automatic segmentation of the image foreground and background are achieved, and highly semantic masks and accurate target segmentation are generated. The modules achieve synergy through feature complementarity and information transmission, which realizes the effective extraction and distinction of image features and improves the robustness of the model in complex scenes.

[0061] The following describes the design principles and implementation steps of this method from the technical solution and workflow of each module.

[0062] 1. Image decoupling module

[0063] The main task of the image decoupling module is to generate images with clear foreground and background separation and provide a semantically strong foreground mask in an unsupervised environment. By introducing the pose variable z, the category variable c and the background variable b, the module can control the image content and layout in an unsupervised manner.

[0064] The image decoupling module is mainly a generator, such as Figure 2 The generator includes an encoder, a foreground generation network (a first generation network), a mask generation network (a second generation network), and a background generation network (a third generation network). The encoder is connected to the foreground generation network, the mask generation network, and the background generation network respectively.

[0065] The foreground generation network includes multiple layers of cascaded convolutional layers, the mask generation network includes multiple layers of cascaded convolutional layers, and the background generation network includes multiple layers of cascaded convolutional layers.

[0066] 1.1 Posture, Category, and Background Variables

[0067] Pose variable z: sampled from a standard Gaussian distribution, used to control the angle, position or shape changes of objects in the image, ensuring that the model can generate objects in different poses, thereby enhancing the diversity and authenticity of the generated images.

[0068] Categorical variable c: A randomly extracted encoding vector, in which only the element corresponding to the category identification bit is 1, and all other elements are 0. It is used to represent the object category in the image and control the semantic features of the object, such as texture and shape, through one-hot encoding. This design supports subsequent clustering tasks and ensures that the generated images have clear category distinction.

[0069] Background variable b: A predefined encoding vector used to represent the background type of the image, also in the form of one-hot encoding.

[0070] In some implementations of this embodiment, in order to ensure that the background and the category have a reasonable matching relationship, the category and the background are associated by setting an identity mapping of b = c. This mapping makes the generated image more natural and realistic, for example, generating a "duck in the water" instead of a "duck in the tree".

[0071] In some other implementations of this embodiment, the inherent association between the foreground and the background is broken, and a mismatch mechanism is designed. The background variable is a randomly extracted coded variable, which is different from the category variable. The model's attention to the semantic features of the foreground object (such as texture and shape) is strengthened by mismatching images of categories and backgrounds, avoiding easy confusion of features and excessive reliance on supervisory information. This type of mismatch mechanism can produce unreasonable image combinations (such as "ducks in a tree"), and then through the discriminator, it helps the model identify which features are inherent to the foreground object and which are background features.

[0072] 1.2 Conditional Generation of Foreground, Background, and Mask

[0073] In order to decouple the foreground from the background, this embodiment uses a decoder and three conditional networks:

[0074] Decoder Φ(·): Generates a shared posture feature φ based on the posture variables, with the formula: φ=Φ(z).

[0075] Foreground generation network F(·,·): Generates the corresponding foreground image x based on the category variable f , the formula is: f =F(φ,c).

[0076] Background generation network B(·,·): Generates different backgrounds x based on background variables b , the formula is: b =B(φ,b).

[0077] Mask generation network M(·,·): Generates a binary mask x m , the mask determines the synthesis area of ​​the foreground and background, and its formula is: m =M(φ,c).

[0078] 1.3 Image synthesis formula

[0079] The generated foreground, background, and mask are combined into the complete image x, which is formulated as:

[0080] x=F(φ,c)⊙M(φ,c)+B(φ,b)⊙(1-M(φ,c))

[0081] Among them, ⊙ represents element-by-element multiplication. This operation ensures that the foreground and background regions do not interfere with each other. The foreground area is generated by F(·,·) and covers the area indicated by the mask, while the background generated by B(·,·) fills the other areas.

[0082] 1.4 Binary Regularization

[0083] In order to generate a clear mask, a binarization loss function L is introduced bin , so that each pixel is close to 0 or 1.

[0084]

[0085] in, Represents the expectation of the corresponding distribution, log is the natural logarithm, which is used to calculate the cross entropy loss.

[0086] 1.5 Detailed explanation of the workflow

[0087] (1) Use posture variables z, category variables c, and background variables b to control the image generation process.

[0088] (2) The conditional network generates the foreground, background, and mask separately and synthesizes the complete image.

[0089] (3) Binary regularization ensures that the mask has a clear foreground-background segmentation effect.

[0090] 2. Fine-grained clustering module

[0091] The fine-grained distance module is mainly a discriminator, such as Figure 3 The fine-grained clustering module performs cluster analysis on image features to achieve unsupervised classification. This module uses cluster prototype embedding, mutual information constraint and contrast loss to ensure that the category features are clearly distinguishable.

[0092] 2.1 Clustering Prototype Embedding Calculation

[0093] The set of categorical variables is defined as I c ={c 1 ,…,c n}, including all categorical variables c i , n represents the total number of categorical variables in the data set, which is used to represent all possible categories of the data set. c is mapped to the cluster prototype space κ = {κ 1 ,…,κ n}, which is calculated by the mapping function Ψ(·) composed of the fully connected layer, and its formula is: κ=Ψ(I c ).

[0094] 2.2 Contrastive loss function based on mutual information

[0095] Mutual information I(A; B) represents the degree of association between two variables A and B. Mutual information I(·; ·) is used to associate visual areas with category features to enhance the clustering effect of the model. The formula is: I(x f ,x m ; c)+I(x b ; b). According to existing literature, using contrast loss L info Optimize the lower bound of mutual information to distinguish different categories of features from each other. The formula is:

[0096] I(x f ,x m ; c)+I(x b ; b) ≥ log(n)-L info ,

[0097]

[0098]

[0099] Among them, S(·,·) is the cosine similarity, exp(·) is the exponential function used to normalize the similarity; τ is the smoothing parameter that controls the smoothness of the distribution, and κ i , κ j represents the cluster prototype, κ i =Ψ(c i ), κ j =Ψ(c j ), c i 、c j is a categorical variable, n is the number of categorical variables, Ψ(·) represents the output of the fully connected layer, ψ κ (·) is the first output of the discriminator, which is used for image feature extraction. is the expectation for z, c, and b.

[0100] 2.3 The synthetic image x and the original image x are input into the discriminator to classify the image as real or fake, which is used to generate realistic synthetic images. The loss function of the adversarial training is L adv for:

[0101]

[0102] in, is the expectation of x, ψ h (·) is the second output of the discriminator, which is used for real / fake image classification.

[0103] 2.4 Detailed explanation of the workflow

[0104] (1) Categorical variables are embedded in the cluster prototype space to generate cluster prototypes.

[0105] (2) The mutual information constraint ensures that the generated features and the category features are closely related, and the contrast loss ensures the independence of the category features.

[0106] 3. Unsupervised Object Segmentation Module

[0107] The unsupervised object segmentation module provides supervision signals to the segmentation network through the generated masks, achieving accurate segmentation of foreground objects. When the training is close to the Nash equilibrium, the segmentation network can effectively generalize to real images.

[0108] 3.1 Segmentation Network and Regularization

[0109] like Figure 4 In this embodiment, the commonly used Unet model is used as the segmentation network S′(·). By using the generated mask x m As a supervisory signal, used to predict the original image Segmentation mask, segmentation loss function L seg The formula used to optimize the performance of S′(·) is:

[0110]

[0111] 3.2 Partition Generalization under Nash Equilibrium

[0112] When the adversarial generative model training reaches Nash equilibrium, the generated synthetic image x and the original image The distribution of is consistent. At this time, the segmentation network S′(·) trained by the generated mask can be generalized to the real image, realizing the unsupervised segmentation of the real image.

[0113] 3.3 Detailed explanation of the workflow

[0114] (1) The generated masks and segmentation loss function are used to train the segmentation network.

[0115] (2) When the generative adversarial network is close to Nash equilibrium, the segmentation network can generalize to real images and improve the segmentation effect.

[0116] The network structure of this embodiment performs image synthesis, fine-grained clustering and object segmentation tasks. Given G = {Φ(·), F(·,·), B(·,·), M(·,·)}, we get the objective function as follows:

[0117]

[0118] Among them, L adv and L bin For image synthesis tasks, L info For fine-grained clustering tasks, L seg For target segmentation tasks, min · 、max· Represents the corresponding network to optimize the loss function.

[0119] The deep learning network model is trained by iterative optimization of the objective function to obtain the image synthesis model.

[0120] This embodiment solves the common problems of strong label dependence and model complexity in traditional methods through the synergy between image decoupling, fine clustering and object segmentation modules. The image decoupling module generates reasonable foreground and background masks, and guides the discriminator to focus on the texture and shape features of the object through class-background matching and mismatching strategies. The fine-grained clustering module further utilizes mutual information and contrast loss to ensure the distinguishability between different categories, provides stable category feature information for the segmentation module, and ensures the independence and diversity of category features. The segmentation module uses the generated mask as an unsupervised signal to achieve high-quality segmentation of real images. The three modules work together to greatly improve the accuracy and applicability of the model.

[0121] This embodiment overcomes the difficulty of the model in generating category distinctions on unlabeled data sets by embedding category variables into cluster prototypes and using cosine similarity calculation. This embodiment uses a single generator, discriminator, and segmentation network by streamlining the model structure. Compared with traditional methods, the computational cost is reduced, making this method more suitable for application in resource-constrained scenarios.

[0122] Example 2

[0123] 1. Dataset Details

[0124] In order to verify the effectiveness of this method in image generation and semantic mask generation, experiments were conducted on five fine-grained datasets and one multi-object dataset. The specific datasets are described as follows:

[0125] CUB (Caltech-UCSD Birds): Contains 11,788 images of 200 bird species and provides manually annotated ground-truth segmentation masks.

[0126] Stanford Dogs: Contains 20,580 images of 120 dog categories, annotated with bounding boxes; 5,024 of these images were manually annotated to obtain supervised ground-truth segmentation masks.

[0127] Stanford Cars: Contains 16,185 images of 196 car categories. The ground-truth segmentation masks for 5,024 images were generated using the same approximation strategy as for the Stanford Dogs dataset.

[0128] Oxford Flowers: Contains about 8,000 flower images with 102 categories, and masks are generated by automated methods. FS-100: Contains face data of 100 categories, with a total of about 15,000 images, from the FaceScrub dataset, and each image is scaled to 64×64 pixels.

[0129] CLEVR6: A subset of the CLEVR dataset, containing objects of 6 different shapes and color combinations. 8,000 images are sampled from the dataset and scaled to 128×128 pixels.

[0130] 2. Training details

[0131] The model is trained using the Adam optimizer with a learning rate of 0.0002 and a weight decay of 0.00001.

[0132] During training, the model uses mutual information and contrastive loss to enhance category discrimination when generating foreground and background.

[0133] Each dataset is trained for 20,000 steps to verify the performance of DCS-ComGAN in generating realistic images and high semantic masks.

[0134] 3. Experimental results

[0135] This method generates high-quality real images and clear semantic masks on multiple datasets, and can effectively separate foreground and background. Figure 5 As shown in the figure, from top to bottom are the synthetic image, mask image, foreground image, and background image. On the CUB, Stanford Dogs, Stanford Cars, Oxford Flowers, FS-100, and CLEVR6 datasets, the images generated by this method are realistic, and the generated semantic mask can accurately segment the foreground of the object. The model can distinguish the boundaries and texture features of objects of different categories and generate images similar to real data.

[0136] Example 3

[0137] The method of the present invention overcomes the limitations of existing methods in label dependence and performance through unsupervised multivariate control image generation, fine-grained clustering and complex background segmentation modules. Traditional methods often rely on supervised labels. Specifically, FineGAN generates images by layers, realizes controllable image generation based on the separation control of background, target shape and appearance, and relies on annotated bounding boxes to assist in separating targets from backgrounds. MixNMatch further refines on this basis, decoupling and encoding four key factors: background, target posture, shape and texture. OneGAN is a complex GAN-based generative model that emphasizes the importance of background in the feature decoupling process. Labels4Free uses the difference between foreground and background to segment objects, emphasizing that accurate coupling is essential for unsupervised segmentation. DRC proposes mutual information regularization and explores the synergy between image generation and segmentation. C3-GAN adopts a scene decomposition-based approach to improve the clustering effect by improving the distribution of cluster centroids. The method of the present invention does not require any external labels, directly uses independent variables to generate highly semantic samples and clearly separated class features, and verifies its superiority in multi-scene tasks. The method of the present invention has significant advantages in the following aspects:

[0138] 1. Improvement of fine-grained clustering capabilities

[0139] This method improves the performance of fine-grained clustering through semantic separation and unsupervised generation strategies. Compared with methods such as C3-GAN, this method significantly improves the common clustering indicators of accuracy (Acc) and normalized mutual information (NMI) on multiple fine-grained datasets (such as CUB, Dogs, Cars, Flowers), and can clearly separate the features of different categories. The experimental data is shown in Table 1.

[0140] Table 1 Comparison of clustering indicators of various methods

[0141]

[0142] As shown in Table 1, this method has achieved significant improvements on the CUB and Flowers datasets, with an average accuracy improvement of about 11 percentage points compared to C3-GAN. Especially in the flower and bird datasets with rich details, the semantic separation and class feature clustering performance of this method are obvious, proving that it can still achieve high-precision clustering under unsupervised conditions. Figure 6 is a visualization of the extracted features of the three methods trained on the CUB dataset, using test images from 10 randomly sampled classes, with the same color being applied to features with the same semantics. Figure 6 It can be seen that compared with other methods, this method forms a tighter category feature distribution during clustering, significantly reduces confusion between categories, and enhances clustering performance.

[0143] 2. Significant reduction in data dependency

[0144] This method relies on diversified samples generated by independent control variables (c, b, z) instead of label annotation or clean background input, which effectively reduces data dependence. In existing methods, a certain proportion of labeled data (such as label information of FineGAN and clean background of OneGAN) is usually required to improve the decoupling ability of the model; while this method can separate categories, backgrounds and postures on multiple fine-grained datasets without relying on labels, and achieve precise control. Figure 7 , Figure 8 , Fig. 9 The generation results show that this method can achieve controllable generation of unsupervised images by adjusting the independent variables c, b, and z.

[0145] (1) The variable z controls the posture of the object. When z changes, the posture of the object changes.

[0146] (2) The variable c controls the category. When c changes, the shape and texture of the object also change, reflecting the semantic information of the object category.

[0147] (3) Variable b controls the background. Changing the background will not affect the category and posture of the object.

[0148] The images generated by this method are highly controllable, which has important advantages in application scenarios that require diversified image generation.

[0149] 3. Improved unsupervised segmentation performance

[0150] This method shows strong unsupervised segmentation capabilities, and significantly improves the commonly used segmentation indicators of various complex backgrounds and multi-object scenes: intersection over union (Iou) and Dice coefficient (Dice). Through comparative experiments, this method outperforms other supervised or weakly supervised segmentation methods on multiple data sets, as shown in Table 2.

[0151] Table 2 Comparison of segmentation indicators of various methods

[0152]

[0153] As shown in Table 2, the proposed method outperforms the comparison method on different data sets, especially under the condition that no external background annotation is required, its segmentation performance is leading in all indicators. This shows that the proposed method can effectively separate the foreground and background by generating a highly semantic mask. Fig.10 The segmentation results of different data sets are shown in Figure 1. From left to right, the original image, the real mask, and the predicted mask. Fig.10From the qualitative segmentation results, it can be observed that the proposed method is superior in detail segmentation, such as the legs of birds and the rearview mirrors of cars, and even exceeds the accuracy of partial truth masks.

[0154] Example 4

[0155] This embodiment provides an electronic device, including:

[0156] one or more processors;

[0157] A memory having one or more programs stored thereon, which, when executed by one or more processors, enables the one or more processors to implement the steps of a multi-task image synthesis method based on adversarial generation.

[0158] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0159] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0160] This embodiment provides a computer-readable storage medium storing a computer program, which implements the steps of a multi-task image synthesis method based on adversarial generation when the computer program is executed by a processor.

[0161] The contents explained in the above embodiments should be understood as these embodiments are only used to more clearly illustrate the present invention, and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

Claims

1. A multi-task image synthesis method based on adversarial generation, characterized in that: The following steps are involved: S1: obtaining an original image, a posture variable, a category variable, and a background variable, wherein the posture variable is sampled from a standard Gaussian distribution, and the category variable and the background variable are randomly extracted encoding vectors; S2: using the original image, the posture variable, the category variable, and the background variable as inputs of a deep learning network model to train the deep learning network model; The deep learning network model includes a generator, a discriminator and a Unet network; the generator is connected to the discriminator and the Unet network respectively; The generator includes an encoder, a first generation network, a second generation network, and a third generation network, and the encoder is connected to the first generation network, the second generation network, and the third generation network respectively; The training process of the deep learning network model includes: The posture variable is processed by the encoder to obtain a posture feature; The posture feature and the category variable are input into the first generation network to obtain a foreground image; The posture feature and the category variable are input into the second generation network to obtain a mask image; The posture feature and the background variable are input into the third generation network to obtain a background image; Combining the foreground image, the mask image and the background image to obtain a composite image; Inputting the synthesized image and the original image into the discriminator to obtain a feature image; Inputting the mask image and the original image into the Unet network to obtain a segmented image; Construct the loss function of the discriminator through cluster analysis and mutual information quantification; The objective function is constructed through the loss function of the generator, the loss function of the discriminator, the loss function of the Unet network, and the adversarial loss function. The deep learning network model is trained by iterative optimization of the objective function to obtain an image synthesis model.

2. The multi-task image synthesis method based on adversarial generation according to claim 1, characterized in that: The categorical variable and the background variable are the same, or the categorical variable and the background variable are different.

3. The multi-task image synthesis method based on adversarial generation according to claim 2, characterized in that: The expression of the composite image x is as follows: x=F(φ,c)⊙M(φ,c)+B(φ,b)⊙(1-M(φ,c)) Among them, ⊙ represents element-by-element multiplication, c is the category variable, b is the background variable, φ is the posture feature, F(φ,c) is the foreground image, M(φ,c) is the mask image, and B(φ,b) is the background image.

4. The multi-task image synthesis method based on adversarial generation according to claim 1, characterized in that: The loss function L of the generator bin The expression is as follows: in, represents the expectation of the corresponding distribution, x m is the mask image.

5. The multi-task image synthesis method based on adversarial generation according to claim 3, characterized in that: The loss function L of the discriminator info The expression is as follows: Among them, S(·,·) is the cosine similarity, exp(·) is the exponential function, τ is the smoothing parameter, and κ i =Ψ(c i ), κ j =Ψ(c j ), c i 、c j is a categorical variable, n is the number of categorical variables, Ψ(·) is the output of the fully connected layer, ψ κ (·) is the first output of the discriminator, is the expectation of z, c, b, and z is the posture variable.

6. The multi-task image synthesis method based on adversarial generation according to claim 1, characterized in that: The loss function L of the Unet network seg The expression is as follows: in, is the original image, and S′(·) is the output of the Unet network.

7. The multi-task image synthesis method based on adversarial generation according to claim 3, characterized in that: The adversarial loss function L adv The expression is as follows: in, is the expectation of x, ψ h (·) is the second output of the discriminator.

8. The multi-task image synthesis method based on adversarial generation according to claim 1, characterized in that: The first generation network includes multiple layers of cascaded convolutional layers, the second generation network includes multiple layers of cascaded convolutional layers, and the third generation network includes multiple layers of cascaded convolutional layers.

9. An electronic device, characterized in that: include: one or more processors; A memory having one or more programs stored thereon, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer program is stored therein, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images