Image generation method and system based on triple control text graph model

Abstract

The invention discloses an image generation method and system based on a triple control text graph model, and the method comprises the steps: obtaining original sample information which comprises a prompt image, a prompt sample and a control condition; based on the stable diffusion model, introducing a plurality of adapter networks, and constructing a triple control text graph model; and performing semantic feature generation processing on the original sample information based on the triple control text graph model to obtain a target text synthesis image. According to the embodiment of the invention, the local and global feature information of the text image can be extracted by fully utilizing the control information, and the complexity of the text image model is reduced, so that the generation precision of the text image is improved. The method can be widely applied to the technical field of image generation.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to an image generation method and system based on a triple control Vincent graph model. Background Art

[0002] Users can guide the text-based graph model to generate images by inputting prompt words. However, the related technologies that only use text to describe the images in mind are often inefficient, and short descriptions are difficult to accurately capture the images in mind. This often requires inputting more prompt words and multiple attempts to generate satisfactory images. For related technologies, some research is dedicated to developing controllable text-based graph models. For example, the Composer model explores methods to integrate multiple control signals with text descriptions, but its demand for GPU resources is very high, which makes it difficult for ordinary researchers to replicate its research results, and it also does not effectively utilize existing pre-trained models. In addition, although studies such as ControlNet and T2I Adapter allow different control conditions to control the generation process, they also have defects. First, when there are multiple control conditions, they need to train multiple control adapter models, which causes the resources required for model inference to increase linearly with the increase in control conditions. Second, the image encoders of these studies fail to fully understand the visual features of the images used for control. In addition, the controllability of these models is insufficient and cannot achieve full control. For example, ControlNet only incorporates control information into the decoder part of the Stable Diffusion denoising model, while T2I Adapter only adds control information to the encoder part. Furthermore, although the control information of ControlNet is added to the decoder, it is generated by using a learnable structure copied from the encoder of the StableDiffusion denoising model, which is similar to the semantic features of the generator and has certain feature conflicts with the decoder with strong generation capabilities.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an image generation method and system based on a triple control Vincent image model, which can fully utilize control information to extract local and global feature information of the Vincent image, reduce the complexity of the Vincent image model, and thus improve the generation accuracy of the Vincent image.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application proposes an image generation method based on a triple control Vincent graph model, the method comprising:

[0006] Acquiring original sample information, wherein the original sample information includes a prompt image, a prompt sample, and a control condition;

[0007] Based on the stable diffusion model, several adapter networks are introduced to construct a triple control Vincent graph model;

[0008] The original sample information is processed for semantic feature generation based on the triple-control Vincent graph model to obtain a target Vincent composite image.

[0009] In some embodiments, the triple-controlled Vincent graph model includes a stable diffusion model, a first adapter network, a second adapter network, a third adapter network, a text encoder and a cross-attention module, wherein the output end of the first adapter network is connected to the first input end of the stable diffusion model, the output end of the second adapter network is connected to the second input end of the stable diffusion model, the output end of the text encoder is connected to the first input end of the cross-attention module, the output end of the third adapter network is connected to the second input end of the cross-attention module, and the output end of the cross-attention module is connected to the third input end of the stable diffusion model.

[0010] In some embodiments, the first adapter network includes a convolutional layer, a downsampling layer, a first residual connection block, a second residual connection block, a third residual connection block, a fourth residual connection block and a feature pyramid network module, wherein the output end of the convolutional layer is connected to the input end of the downsampling layer, the output end of the downsampling layer is connected to the input end of the first residual connection block, the first residual connection block, the second residual connection block, the third residual connection block and the fourth residual connection block are connected in sequence, and the output end of the first residual connection block, the output end of the second residual connection block, the output end of the third residual connection block and the output end of the fourth residual connection block are all connected to the input end of the feature pyramid network module.

[0011] In some embodiments, the second adapter network includes a control condition preprocessing module, a first SD encoder module, a second SD encoder module, a third SD encoder module, a fourth SD encoder module and an SD intermediate module, wherein the output end of the control condition preprocessing module, the output end of the first SD encoder module, the output end of the second SD encoder module, the output end of the third SD encoder module, the output end of the fourth SD encoder module and the output end of the SD intermediate module are all connected with a zero convolution layer, and the control condition preprocessing module, the first SD encoder module, the second SD encoder module, the third SD encoder module, the fourth SD encoder module and the SD intermediate module are connected in sequence.

[0012] In some embodiments, the third adapter network includes an image encoder module, a fully connected layer, a normalization layer and an activation layer, wherein the image encoder module, the fully connected layer, the normalization layer and the activation layer are connected in sequence, and the stable diffusion model includes a first SD encoder module, an SD intermediate module and a second SD encoder module, wherein the output end of the second SD encoder module is connected to the first input end of the SD intermediate module, the output end of the SD intermediate module is connected to the first input end of the first SD encoder module, the output end of the cross attention module is connected to the second input end of the first SD encoder module, the second input end of the SD intermediate module and the first input end of the second SD encoder module, the output end of the first adapter network is connected to the third input end of the first SD encoder module, and the output end of the second adapter network is connected to the second input end of the second SD encoder module.

[0013] In some embodiments, the processing of generating semantic features on the original sample information based on the triple-control Vincent graph model to obtain a target Vincent composite image includes:

[0014] Inputting the original sample information into the triple control Vincent graph model;

[0015] Based on the first adapter network of the triple control Vincent graph model, feature extraction processing is performed on the control condition to obtain a first multi-scale control feature;

[0016] Based on the second adapter network of the triple control Vincent graph model, feature extraction processing is performed on the control condition to obtain a second multi-scale control feature;

[0017] Based on the third adapter network of the triple control Vincent graph model, performing feature extraction processing on the prompt image to obtain control image features;

[0018] Based on the text encoder of the triple control text graph model, feature extraction processing is performed on the prompt sample to obtain prompt sample features;

[0019] Based on the cross attention module of the triple control Wensheng graph model, the control image features and the prompt sample features are respectively subjected to attention feature extraction processing to obtain the control image attention features and the prompt sample attention features;

[0020] Based on the stable diffusion model of the triple control Vincent graph model, the prompt sample attention feature, the first multi-scale control feature, the second multi-scale control feature and the control image attention feature are input in sequence, and feature fusion and noise reduction processing are performed to obtain the target Vincent synthetic image.

[0021] In some embodiments, the first adapter network based on the triple control Vincent graph model performs feature extraction processing on the control condition to obtain a first multi-scale control feature, including:

[0022] inputting the control condition into a first adapter network of the triple control Vincent graph model;

[0023] Based on the convolution layer and the downsampling layer of the first adapter network, the control condition is subjected to convolution and bilinear interpolation downsampling processing to obtain a downsampled control feature;

[0024] Based on the first residual connection block, the second residual connection block, the third residual connection block and the fourth residual connection block of the first adapter network, sequentially performing feature extraction processing on the downsampled control features to obtain first residual control features, second residual control features, third residual control features and fourth residual control features;

[0025] Based on the feature pyramid network module of the first adapter network, feature fusion processing is performed on the first residual control feature, the second residual control feature, the third residual control feature and the fourth residual control feature to obtain the first multi-scale control feature.

[0026] In some embodiments, the second adapter network based on the triple control Vincent graph model performs feature extraction processing on the control condition to obtain a second multi-scale control feature, including:

[0027] inputting the control condition into a second adapter network of the triple control Vincent graph model;

[0028] Based on the control condition preprocessing module of the second adapter network, the control condition is preprocessed to obtain the control condition feature after dimension reduction;

[0029] Based on the first SD encoder module, the second SD encoder module, the third SD encoder module, the fourth SD encoder module and the SD intermediate module of the second adapter network, sequentially performing feature extraction processing on the control condition features after dimension reduction to obtain first coding features, second coding features, third coding features, fourth coding features and intermediate coding features;

[0030] The first coding feature, the second coding feature, the third coding feature, the fourth coding feature and the intermediate coding feature are integrated to obtain the second multi-scale control feature.

[0031] In some embodiments, the third adapter network based on the triple control Vincent graph model performs feature extraction processing on the prompt image to obtain control image features, including:

[0032] inputting the prompt image into a third adapter network of the triple control Vincent graph model;

[0033] Based on the encoder module of the third adapter network, encoding the prompt image to obtain encoded prompt image features;

[0034] Based on the fully connected layer of the third adapter network, performing feature classification processing on the encoded prompt image features to obtain classified prompt image features;

[0035] Based on the normalization layer of the third adapter network, normalizing the classified prompt image features to obtain normalized prompt image features;

[0036] Based on the activation layer of the third adapter network, the normalized prompt image features are activated to obtain the control image features.

[0037] To achieve the above purpose, another aspect of the embodiment of the present application provides an image generation system based on a triple control Vincent graph model, the system comprising:

[0038] An acquisition module, used for acquiring original sample information, wherein the original sample information includes a prompt image, a prompt sample and a control condition;

[0039] A building module is used to introduce several adapter networks based on the stable diffusion model to construct a triple control Vincent graph model;

[0040] A generation module is used to perform semantic feature generation processing on the original sample information based on the triple control Vincent graph model to obtain a target Vincent composite image.

[0041] The embodiments of the present application include at least the following beneficial effects: The present application provides an image generation method and system based on a triple-control Vincent graph model. The scheme introduces an adapter network based on a stable diffusion model to construct a triple-control Vincent graph model, extracts local detail texture and other features of the Vincent original image, and then controls the global feature information of the Vincent original image. The constructed triple-control Vincent graph model can process a variety of control conditions in any number at the same time, and allows each control condition to be combined, thereby avoiding feature conflicts with the decoder, further reducing the design complexity of the model, reducing the cost of data training and the efficiency of data training, and thus improving the generation accuracy of the Vincent image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of an image generation method based on a triple control Vincent graph model provided in an embodiment of the present application;

[0043] Figure 2 It is a structural schematic diagram of an image generation system based on a triple control Vincent graph model provided in an embodiment of the present application;

[0044] Figure 3 It is a structural schematic diagram of a triple control Vincent graph model provided in an embodiment of the present application;

[0045] Figure 4 is a first adapter network schematic diagram provided in an embodiment of the present application;

[0046] Figure 5 is a schematic diagram of a second adapter network provided in an embodiment of the present application;

[0047] Figure 6 This is a schematic diagram of a third adapter network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0049] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0050] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] Reference Figure 1 , Figure 1 A flowchart of an image generation method based on a triple control Vincent graph model provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0053] S100, obtaining original sample information, where the original sample information includes prompt images, prompt samples, and control conditions;

[0054] S200, based on the stable diffusion model, introduce several adapter networks to construct a triple control Vincent graph model;

[0055] It should be noted that in some embodiments, Figure 3 As shown, the triple control text graph model includes a stable diffusion model, a first adapter network, a second adapter network, a third adapter network, a text encoder and a cross-attention module, wherein the output end of the first adapter network is connected to the first input end of the stable diffusion model, the output end of the second adapter network is connected to the second input end of the stable diffusion model, the output end of the text encoder is connected to the first input end of the cross-attention module, the output end of the third adapter network is connected to the second input end of the cross-attention module, and the output end of the cross-attention module is connected to the third input end of the stable diffusion model.

[0056] In some specific embodiments, Figure 4 As shown, the first adapter network includes a convolutional layer, a downsampling layer, a first residual connection block, a second residual connection block, a third residual connection block, a fourth residual connection block and a feature pyramid network module, wherein the output end of the convolutional layer is connected to the input end of the downsampling layer, the output end of the downsampling layer is connected to the input end of the first residual connection block, the first residual connection block, the second residual connection block, the third residual connection block and the fourth residual connection block are connected in sequence, and the output end of the first residual connection block, the output end of the second residual connection block, the output end of the third residual connection block and the output end of the fourth residual connection block are all connected to the input end of the feature pyramid network module.

[0057] In some specific embodiments, Figure 5As shown, the second adapter network includes a control condition preprocessing module, a first SD encoder module, a second SD encoder module, a third SD encoder module, a fourth SD encoder module and an SD intermediate module, wherein the output end of the control condition preprocessing module, the output end of the first SD encoder module, the output end of the second SD encoder module, the output end of the third SD encoder module, the output end of the fourth SD encoder module and the output end of the SD intermediate module are all connected with a zero convolution layer, and the control condition preprocessing module, the first SD encoder module, the second SD encoder module, the third SD encoder module, the fourth SD encoder module and the SD intermediate module are connected in sequence.

[0058] In some specific embodiments, Figure 6 As shown, the third adapter network includes an image encoder module, a fully connected layer, a normalization layer and an activation layer, wherein the image encoder module, the fully connected layer, the normalization layer and the activation layer are connected in sequence, and the stable diffusion model includes a first SD encoder module, an SD intermediate module and a second SD encoder module, wherein the output end of the second SD encoder module is connected to the first input end of the SD intermediate module, the output end of the SD intermediate module is connected to the first input end of the first SD encoder module, the output end of the cross attention module is connected to the second input end of the first SD encoder module, the second input end of the SD intermediate module and the first input end of the second SD encoder module, the output end of the first adapter network is connected to the third input end of the first SD encoder module, and the output end of the second adapter network is connected to the second input end of the second SD encoder module.

[0059] S300, performing semantic feature generation processing on the original sample information based on the triple control Wensheng graph model to obtain a target Wensheng synthetic image;

[0060] It should be noted that, in some embodiments, step S300 may include: S310, inputting original sample information into a triple control Vincent graph model;

[0061] In the embodiment of the present invention, in addition to the SD main model U-Net (stable diffusion model) in the network, there are three adapters for controlling image generation. The first and second adapters are mainly used to control the generated content, texture, details, etc., and their inputs are several kinds of various control conditions, such as image edge texture, sketch, sketch map, depth map, full body key points, semantic segmentation map, etc. The third adapter is mainly used for global control and style control of the generated image, and its image control features are extracted from the reference image using image CLIP. TCSDM (Triple Control Vincent Graph Model) can provide comprehensive control over the generation process by utilizing global and local detail control conditions.

[0062] S320, performing feature extraction processing on the control condition based on the first adapter network of the triple control Vincent graph model to obtain a first multi-scale control feature;

[0063] In step S320 of some embodiments, the control condition is input into the first adapter network of the triple control Vincent graph model; based on the convolution layer and the downsampling layer of the first adapter network, the control condition is convolved and bilinear interpolation downsampling is performed to obtain the downsampled control feature; based on the first residual connection block, the second residual connection block, the third residual connection block and the fourth residual connection block of the first adapter network, the downsampled control feature is sequentially subjected to feature extraction processing to obtain the first residual control feature, the second residual control feature, the third residual control feature and the fourth residual control feature; based on the feature pyramid network module of the first adapter network, the first residual control feature, the second residual control feature, the third residual control feature and the fourth residual control feature are subjected to feature fusion processing to obtain the first multi-scale control feature.

[0064] In the embodiment of the present invention, the network structure of the first adapter mainly includes 4 residual connection blocks in the ResNet18 network and the Feature Pyramid Networks (FPN) structure. The resolution of the initial control conditions C is 512×512, C={C1,C2,...,C n}, n is the maximum number of total control conditions. i After a layer of step size 2×2 and bilinear interpolation downsampling, the feature f with a dimension of 128×128 is obtained. i 1 , i∈[1,n]. i 1 Then, after passing through the 4-layer residual blocks in ResNet18, the features f are obtained in turn. i 2 、f i 3 、f i 4 、f i 5 , with resolutions of 64×64, 32×32, 16×16, and 8×8, respectively. All control conditions are obtained after passing through this shared ResNet18 Then f 2 、f 3 、f 4 、f 5 Send it to the feature pyramid network FPN for multi-scale control feature fusion α i =w i / (w1+w2+...+w n ), i∈[1,n], w i is 0 or 1. In each different batch of network training, w1 to w n are all random numbers between 0 and 1. The advantage of this design is that only one adapter network can be used to process multiple different control conditions at the same time, and the data of the control conditions is variable, whether it is one or more conditions. During inference, no matter how many control conditions there are, the generation operation can be controlled. The channel dimension of the feature is consistent with the channel dimension of each resolution dimension in the decoder of U-Net in SD. Then the two sets of features are added element by element to complete the control feature injection of the first adapter.

[0065] S330, performing feature extraction processing on the control condition based on the second adapter network of the triple control Vincent graph model to obtain a second multi-scale control feature;

[0066] In step S330 of some embodiments, the control conditions are input into the second adapter network of the triple control Vincent graph model; the control conditions are preprocessed by the control condition preprocessing module based on the second adapter network to obtain the control condition features after dimensionality reduction; the first SD encoder module, the second SD encoder module, the third SD encoder module, the fourth SD encoder module and the SD intermediate module based on the second adapter network sequentially perform feature extraction processing on the control condition features after dimensionality reduction to obtain the first coding feature, the second coding feature, the third coding feature, the fourth coding feature and the intermediate coding feature; the first coding feature, the second coding feature, the third coding feature, the fourth coding feature and the intermediate coding feature are integrated to obtain the second multi-scale control feature.

[0067] In the embodiment of the present invention, the second adapter also uses local information such as image edge texture, sketch, sketch map, depth map, whole body key points, semantic segmentation map, etc. to control the generated content. At the same time, the idea of ​​ControlNet is borrowed, and there are two improvements compared with it. First, the second adapter uses a network to process any multiple control conditions at the same time, while when ControlNet processes multiple control conditions, multiple control networks are required to extract multiple different control features respectively. Several control conditions require several control networks, which leads to a linear increase in computing requirements and time consumption. Secondly, our second adapter is easier to learn than ControlNet. The control network structure of ControlNet is copied from the U-Net encoder in SD, so the output of ControlNet has more similarities with the output of the encoder than the output of the decoder, and the output of ControlNet is added to the U-Net decoder, which requires the control network of ControlNet to learn not only the understanding of the control conditions but also the mapping from the encoder space to the decoder space, which increases the difficulty of learning. The role of the encoder is to extract features or compress information from the input data to form a compact representation in the latent space. This stage mainly understands and encodes the input content. Therefore, our second adapter adds control information to the encoder, which helps SD better understand the content of the control information and reduces the learning difficulty of the network.

[0068] The second adapter network structure includes a preprocessing model for controlling conditional resolution and five adapter modules. The first half of each module is a copy training of the encoder part and the middle module of the SD denoising model, where X t During training, an image in the training set is subjected to data augmentation and VAE image encoder, and then diffused through the SD model diffusion process for t steps (which can be considered as noise that is infinitely close to a standard Gaussian distribution). t The noise in the test is subject to standard Gaussian distribution, and the latter part is the 1×1 zero convolution in ControlNet. The 1×1 zero convolution is a 1×1 convolution with parameters initialized with 0 and the number of output channels unchanged. The resolution of each module output feature is 64×64, 32×32, 16×16 and 8×8 respectively. Control condition C i First, the feature size of 64×64 is obtained by the control condition preprocessing module and 1×1 zero convolution. After passing through five adapter modules in turn, the characteristics are obtained in turn. The resolutions are 64×64, 32×32, 16×16, 8×8 and 8×8. All control conditions are obtained after passing through this shared second adapter. Then h 1 、h 2 、h 3 、h 4 、h 5 Weighted addition is performed to control feature fusion. α i The processing method is the same as that of the second adapter, which is also to ensure that during inference, no matter how many control conditions there are, the generation operation can be controlled.

[0069] S340, performing feature extraction processing on the prompt image based on the third adapter network of the triple control Vincent graph model to obtain control image features;

[0070] In step S340 of some embodiments, the prompt image is input into the third adapter network of the triple control Vincent graph model; based on the encoder module of the third adapter network, the prompt image is encoded to obtain the encoded prompt image features; based on the fully connected layer of the third adapter network, the encoded prompt image features are feature classified to obtain the classified prompt image features; based on the normalization layer of the third adapter network, the classified prompt image features are normalized to obtain the normalized prompt image features; based on the activation layer of the third adapter network, the normalized prompt image features are activated to obtain the control image features.

[0071] In an embodiment of the present invention, the third adapter uses the image as a prompt condition for image generation. The prompt image first passes through the image CLIP encoder to extract image features, and then uses the fully connected layer, BatchNorm normalization and ReLU activation function to project the control image features to the same dimension as the text prompt word features.

[0072] S350, performing feature extraction processing on the prompt sample based on the text encoder of the triple control text graph model to obtain prompt sample features;

[0073] S360, a cross attention module based on the triple control text graph model, performing attention feature extraction processing on the control image features and the prompt sample features respectively, to obtain the control image attention features and the prompt sample attention features;

[0074] S370, based on the stable diffusion model of the triple control Vincent graph model, the prompt sample attention feature, the first multi-scale control feature, the second multi-scale control feature and the control image attention feature are sequentially input, and feature fusion and noise reduction processing are performed to obtain the target Vincent composite image.

[0075] In an embodiment of the present invention, when a user uses a text-based graph model, an image can be generated by inputting prompt words into the text-based graph model. However, it is inefficient to use text to fully describe the information of an image in the mind. It is difficult to accurately describe the picture in the mind with just a few words, which often requires inputting more prompt words and multiple attempts to generate. Therefore, more information other than text is needed to control the generation of the image to be closer to the picture in the mind. Therefore, the way of injecting image features into U-Net in the third adapter is the same as the way of injecting text features, but there is a sequence. Text features are injected first, and image features are injected later. This is to ensure the original generation ability of the SD model and try not to destroy the original structure of the SD model. The feature that performs the cross-attention operation with the text feature T in the SD model U-Net is represented by Z, and the image feature is represented by I. The output of the first cross-attention operation is Output of the second cross attention Among them, Z∈R hw×c , Z′∈R hw×c , Z″∈R hw×c are the features before cross-attention, the first cross-attention, and the second cross-attention, respectively. h, w, and c represent the height, width, and number of channels of the feature map Z, respectively.

[0076] ″′

[0077] W q , W k , W v , W q , W k , W v are all feature transformation matrices with dimensions of c×c.

[0078] ″′

[0079] The parameters of the SD model are kept, so the above parameters are only W q , W k , W v It is trainable.

[0080] In addition, it should be noted that all parameters of the SD model are frozen during training, and only the parameters of the three adapters are trained. Training is done on the LAION-2B dataset containing image-text pairs. The third adapter is trained separately from the first and second adapters during training, because this allows both to be fully trained. All data in LAION-2B are image and text paired data.

[0081] In summary, the embodiment of the present invention proposes a Wensheng graph model that can effectively control the Wensheng graph generation effect, and it is named the Triple Control Stable Diffusion Model (TCSDM). TCSDM designs three adapters to accurately and effectively control the generation content of the Wensheng graph diffusion model. The main part of TCSDM is a stable diffusion model (SD) with frozen parameters, and the three adapters with learnable parameters play a control role. Two of the adapters are used for local control, and the third adapter is used for global control. TCSDM can handle multiple control conditions of any number at the same time, and allows each control condition to be combined. No matter how many control conditions there are, only three adapters are used. This design greatly reduces the cost of fine-tuning and model size under multiple controls. The first adapter is used to adapt the decoder part of the SD denoising model, and the multi-scale features obtained after the control conditions pass through the first adapter are directly added to the decoder features to complete the feature injection. The second adapter is used to adapt the encoder and intermediate bottleneck structure part of the SD denoising model, and the multi-scale features obtained after the control conditions pass through the second adapter are directly added to the decoder and intermediate bottleneck structure features to complete the feature injection. The third adapter is used for global control. After the control conditions extract semantic features through CLIP, they are injected into all layers of the SD denoising model through cross-attention to complete the global control information injection.

[0082] See also Figure 2 The embodiment of the present application further provides an image generation system based on the triple control Vincent graph model, which can implement the above-mentioned image generation method based on the triple control Vincent graph model, and the system includes:

[0083] An acquisition module 201 is used to acquire original sample information, where the original sample information includes a prompt image, a prompt sample, and a control condition;

[0084] A construction module 202 is used to introduce a plurality of adapter networks based on a stable diffusion model to construct a triple control Vincent graph model;

[0085] The generation module 203 is used to perform semantic feature generation processing on the original sample information based on the triple control Vincent graph model to obtain a target Vincent composite image.

[0086] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0087] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image generation method based on a triple control Vincent graph model, characterized in that: The method comprises the following steps: Acquiring original sample information, wherein the original sample information includes a prompt image, a prompt sample, and a control condition; Based on the stable diffusion model, several adapter networks are introduced to construct a triple control Vincent graph model; The original sample information is processed for semantic feature generation based on the triple-control Vincent graph model to obtain a target Vincent composite image.

2. The method according to claim 1, characterized in that The triple-controlled text graph model includes a stable diffusion model, a first adapter network, a second adapter network, a third adapter network, a text encoder and a cross-attention module, wherein the output end of the first adapter network is connected to the first input end of the stable diffusion model, the output end of the second adapter network is connected to the second input end of the stable diffusion model, the output end of the text encoder is connected to the first input end of the cross-attention module, the output end of the third adapter network is connected to the second input end of the cross-attention module, and the output end of the cross-attention module is connected to the third input end of the stable diffusion model.

3. The method according to claim 2, characterized in that The first adapter network includes a convolution layer, a downsampling layer, a first residual connection block, a second residual connection block, a third residual connection block, a fourth residual connection block and a feature pyramid network module, wherein the output end of the convolution layer is connected to the input end of the downsampling layer, the output end of the downsampling layer is connected to the input end of the first residual connection block, the first residual connection block, the second residual connection block, the third residual connection block and the fourth residual connection block are connected in sequence, and the output end of the first residual connection block, the output end of the second residual connection block, the output end of the third residual connection block and the output end of the fourth residual connection block are all connected to the input end of the feature pyramid network module.

4. The method according to claim 2, characterized in that: The second adapter network includes a control condition preprocessing module, a first SD encoder module, a second SD encoder module, a third SD encoder module, a fourth SD encoder module and an SD intermediate module, wherein the output end of the control condition preprocessing module, the output end of the first SD encoder module, the output end of the second SD encoder module, the output end of the third SD encoder module, the output end of the fourth SD encoder module and the output end of the SD intermediate module are all connected with a zero convolution layer, and the control condition preprocessing module, the first SD encoder module, the second SD encoder module, the third SD encoder module, the fourth SD encoder module and the SD intermediate module are connected in sequence.

5. The method according to claim 2, characterized in that: The third adapter network includes an image encoder module, a fully connected layer, a normalization layer and an activation layer, wherein the image encoder module, the fully connected layer, the normalization layer and the activation layer are connected in sequence, and the stable diffusion model includes a first SD encoder module, an SD intermediate module and a second SD encoder module, wherein the output end of the second SD encoder module is connected to the first input end of the SD intermediate module, the output end of the SD intermediate module is connected to the first input end of the first SD encoder module, the output end of the cross attention module is connected to the second input end of the first SD encoder module, the second input end of the SD intermediate module and the first input end of the second SD encoder module, the output end of the first adapter network is connected to the third input end of the first SD encoder module, and the output end of the second adapter network is connected to the second input end of the second SD encoder module.

6. The method according to claim 2, characterized in that The step of performing semantic feature generation processing on the original sample information based on the triple-control Vincent graph model to obtain a target Vincent composite image includes: Inputting the original sample information into the triple control Vincent graph model; Based on the first adapter network of the triple control Vincent graph model, feature extraction processing is performed on the control condition to obtain a first multi-scale control feature; Based on the second adapter network of the triple control Vincent graph model, feature extraction processing is performed on the control condition to obtain a second multi-scale control feature; Based on the third adapter network of the triple control Vincent graph model, performing feature extraction processing on the prompt image to obtain control image features; Based on the text encoder of the triple control text graph model, feature extraction processing is performed on the prompt sample to obtain prompt sample features; Based on the cross attention module of the triple control Wensheng graph model, the control image features and the prompt sample features are respectively subjected to attention feature extraction processing to obtain the control image attention features and the prompt sample attention features; Based on the stable diffusion model of the triple control Vincent graph model, the prompt sample attention feature, the first multi-scale control feature, the second multi-scale control feature and the control image attention feature are input in sequence, and feature fusion and noise reduction processing are performed to obtain the target Vincent synthetic image.

7. The method according to claim 6, characterized in that The first adapter network based on the triple control Vincent graph model performs feature extraction processing on the control condition to obtain a first multi-scale control feature, including: inputting the control condition into a first adapter network of the triple control Vincent graph model; Based on the convolution layer and the downsampling layer of the first adapter network, the control condition is subjected to convolution and bilinear interpolation downsampling processing to obtain a downsampled control feature; Based on the first residual connection block, the second residual connection block, the third residual connection block and the fourth residual connection block of the first adapter network, sequentially performing feature extraction processing on the downsampled control features to obtain first residual control features, second residual control features, third residual control features and fourth residual control features; Based on the feature pyramid network module of the first adapter network, feature fusion processing is performed on the first residual control feature, the second residual control feature, the third residual control feature and the fourth residual control feature to obtain the first multi-scale control feature.

8. The method according to claim 6, characterized in that The second adapter network based on the triple control Vincent graph model performs feature extraction processing on the control condition to obtain a second multi-scale control feature, including: inputting the control condition into a second adapter network of the triple control Vincent graph model; Based on the control condition preprocessing module of the second adapter network, the control condition is preprocessed to obtain the control condition feature after dimension reduction; Based on the first SD encoder module, the second SD encoder module, the third SD encoder module, the fourth SD encoder module and the SD intermediate module of the second adapter network, sequentially performing feature extraction processing on the control condition features after dimension reduction to obtain first coding features, second coding features, third coding features, fourth coding features and intermediate coding features; The first coding feature, the second coding feature, the third coding feature, the fourth coding feature and the intermediate coding feature are integrated to obtain the second multi-scale control feature.

9. The method according to claim 6, characterized in that The third adapter network based on the triple control Vincent graph model performs feature extraction processing on the prompt image to obtain control image features, including: inputting the prompt image into a third adapter network of the triple control Vincent graph model; Based on the encoder module of the third adapter network, encoding the prompt image to obtain encoded prompt image features; Based on the fully connected layer of the third adapter network, performing feature classification processing on the encoded prompt image features to obtain classified prompt image features; Based on the normalization layer of the third adapter network, normalizing the classified prompt image features to obtain normalized prompt image features; Based on the activation layer of the third adapter network, the normalized prompt image features are activated to obtain the control image features.

10. An image generation system based on a triple control Vincent graph model, characterized in that: The system comprises: An acquisition module, used for acquiring original sample information, wherein the original sample information includes a prompt image, a prompt sample and a control condition; A building module is used to introduce several adapter networks based on the stable diffusion model to construct a triple control Vincent graph model; A generation module is used to perform semantic feature generation processing on the original sample information based on the triple control Vincent graph model to obtain a target Vincent composite image.

Images