AI generative model training method, image generation method and electronic equipment

Abstract

The embodiment of the invention discloses an AI generative model training method, an image generation method and electronic equipment, and the method comprises the steps: obtaining a plurality of training data sets used for training an artificial intelligence AI generative model, and the plurality of training data sets comprise a plurality of condition images corresponding to a plurality of different control types; respectively carrying out feature extraction on the condition image and the text content of the control instruction through an adapter network of an AI generative model so as to obtain control information; and injecting the control information into a backbone network of the AI generative model in a cross attention mode, introducing the spatial coding information of the condition image into the backbone network, and completing training of the AI generative model. Therefore, the AI generative model can be uniformly controlled through the condition images of various different control types. According to the embodiment of the invention, unified control on the model by various different types of control signals can be realized.

Description

Technical Field

[0001] The present application relates to the field of AI generation technology, and in particular to the training of AI generative models, image generation methods and electronic devices. Background Art

[0002] With the development of AI (Artificial Intelligence) generation technology, various forms of AI generative models have emerged in the industry. Among them, in terms of prompting methods and the types of generated content, they can be divided into AI generative models such as "text generates text", "text generates pictures", and "picture generates pictures". Among them, "AI text generates pictures" uses AI generation technology to prompt the AI ​​generative model to draw images based on the input text. In other words, as long as a paragraph is manually input, AI can understand the meaning of the paragraph and then draw the corresponding image.

[0003] However, in some cases, it may be difficult to achieve more precise control by relying solely on text descriptions to prompt the model. In order to achieve more precise control, some solutions can introduce control signals such as depth maps and style maps to jointly guide the image generation of AI generative models with text descriptions. For example, you can control the spatial information of the generated image by introducing depth maps, edge maps, etc., or if you want to generate an image related to a specific object, you can introduce an image about a certain object to control the generated image, and so on.

[0004] Introducing image-based control signals is conducive to achieving more precise control over generated images. However, in the prior art, it is necessary to train AI generative models separately for different types of control signals. In other words, one model can be trained for control needs in terms of spatial information, and another model needs to be trained for control needs to generate a specific object. The two models are completely independent and are not reused in terms of parameters. This method will make the training process more complicated and require more resources. In addition, since the same model can only input the same type of control signals, it is difficult to meet the image generation needs in some complex scenes. Summary of the invention

[0005] The present application provides training of an AI generative model, an image generation method and an electronic device, which can achieve unified control of the AI ​​generative model by a variety of different types of control signals.

[0006] This application provides the following solutions:

[0007] An AI generative model training method, comprising:

[0008] Acquire multiple training data groups for training an artificial intelligence (AI) generative model, wherein the training data groups include: a description text content describing a target image used to supervise a training process, at least one conditional image, and a control instruction text content corresponding to the conditional image; the multiple training data groups include multiple conditional images corresponding to multiple different control types;

[0009] Through the adapter network of the AI ​​generative model, feature extraction is performed on the condition image and the control instruction text content respectively to obtain control information;

[0010] The control information is injected into the backbone network of the AI ​​generative model by means of cross-attention, and the spatial encoding information of the conditional image is introduced into the backbone network, and the training of the AI ​​generative model is completed, so that the AI ​​generative model can be uniformly controlled through conditional images of multiple different control types.

[0011] Among them, it also includes:

[0012] After feature extraction is performed on the conditional image and the control instruction text content respectively, multimodal interaction processing is performed on the first feature vector corresponding to the conditional image and the second feature vector corresponding to the control instruction text content, so that the third feature vector corresponding to the conditional image and the fourth feature vector corresponding to the control instruction text content obtained by the multimodal interaction processing can be determined as the control information.

[0013] The control instruction text content corresponding to the conditional image is used to describe the control type and / or target control dimension of the conditional image.

[0014] Among them, the training data group also includes the target image, which is used to supervise the training process of the AI ​​generative model.

[0015] The adapter network and the backbone network are both multi-layer networks with the same number of layers.

[0016] An AI generative model training method, comprising:

[0017] Acquire multiple training data groups for training an artificial intelligence (AI) generative model, wherein the training data groups include: a description text content describing a target image used to supervise a training process, at least one conditional image, and a control instruction text content corresponding to the conditional image; the multiple training data groups include multiple conditional images corresponding to multiple different control types;

[0018] Through the adapter network of the AI ​​generative model, feature extraction is performed on the condition image and the control instruction text content respectively, and multimodal interactive processing is performed on the first feature vector corresponding to the condition image and the second feature vector corresponding to the control instruction text content to obtain control information;

[0019] The control information is injected into the backbone network of the AI ​​generative model so as to combine the control information with the description text content, and complete the training of the AI ​​generative model so as to achieve unified control of the AI ​​generative model.

[0020] A method for generating an image using an AI generative model, comprising:

[0021] Receive a description text content input by a user to describe an image to be generated, a plurality of condition images, and a control instruction text content corresponding to the condition images, wherein the plurality of condition images correspond to a plurality of different control types;

[0022] Through the adapter network of the AI ​​generative model, feature extraction is performed on the condition image and the control instruction text content respectively to obtain control information;

[0023] The control information is injected into the backbone network of the AI ​​generative model by means of cross-attention, and the spatial encoding information of the conditional image is introduced into the backbone network, so that the AI ​​generative model can be jointly controlled by conditional images of multiple different control types, and the target image can be generated by the AI ​​generative model.

[0024] Among them, it also includes:

[0025] After feature extraction is performed on the conditional image and the control instruction text content respectively, multimodal interaction processing is performed on the first feature vector corresponding to the conditional image and the second feature vector corresponding to the control instruction text content, so that the third feature vector corresponding to the conditional image and the fourth feature vector corresponding to the control instruction text content obtained by the multimodal interaction processing can be determined as the control information.

[0026] A method for generating an image using an AI generative model, comprising:

[0027] Receive a description text content input by a user to describe an image to be generated, a plurality of condition images, and a control instruction text content corresponding to the condition images, wherein the plurality of condition images correspond to a plurality of different control types;

[0028] Through the adapter network of the AI ​​generative model, feature extraction is performed on the condition image and the control instruction text content respectively, and multimodal interactive processing is performed on the first feature vector corresponding to the condition image and the second feature vector corresponding to the control instruction text content to obtain control information;

[0029] The control information is injected into the backbone network of the AI ​​generative model so that the AI ​​generative model is jointly controlled through conditional images of multiple different control types, and the target image is generated by the AI ​​generative model.

[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described above.

[0031] An electronic device, comprising:

[0032] one or more processors; and

[0033] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the methods described above.

[0034] A computer program product comprises a computer program / computer executable instructions, wherein the computer program / computer executable instructions implement the steps of any of the aforementioned methods when executed by a processor in an electronic device.

[0035] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0036] Through the embodiments of the present application, an adapter network can be added on the basis of the backbone network of the AI ​​generative model. When collecting specific training data, it can exist in the form of a quadruple, that is, it can include a target image for supervising the training process, a descriptive text content describing the target image, at least one conditional image, and a control instruction text content corresponding to the conditional image. Among them, multiple training data groups can include multiple conditional images corresponding to multiple different control types, so that the training data can cover multiple different types of control signals. Specifically, during training, the adapter network of the AI ​​generative model can first be used to extract features of the conditional image and the control instruction text content to obtain control information; then, the control information can be injected into the backbone network of the AI ​​generative model by cross-attention, and the spatial coding information of the conditional image can be introduced into the backbone network, so as to combine the control information, spatial coding information and the descriptive text content to achieve control of the AI ​​generative model. Among them, since multiple training data groups include multiple conditional images corresponding to multiple different control types, after the AI ​​generative model is supervised and trained in the above manner, it is possible to achieve unified control of the AI ​​generative model through conditional images of multiple different control types, that is, unified control can be achieved by inputting multiple different types of control signals into the same AI generative model, so as to meet the image generation control requirements in multiple different scenarios through the same AI generative model. Among them, when injecting control information into the backbone network, a cross-attention mechanism is adopted, and the spatial encoding information of the conditional image can also be introduced into the backbone network to compensate for the loss caused by the spatial attention mechanism in the correlation between pixels.

[0037] In a preferred implementation, after the conditional image and the control instruction text content are respectively feature extracted, the first feature vector corresponding to the conditional image and the second feature vector corresponding to the control instruction text content can also be subjected to multimodal interactive processing, so that the third feature vector corresponding to the conditional image and the fourth feature vector corresponding to the control instruction text content obtained by the multimodal interactive processing are determined as the control information. In this way, each can obtain richer information from the other, which is conducive to improving the final image generation quality.

[0038] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0041] Figure 2 is a flow chart of the first method provided in an embodiment of the present application;

[0042] Figure 3 is a flow chart of the second method provided in an embodiment of the present application;

[0043] Figure 4 is a flowchart of the third method provided in an embodiment of the present application;

[0044] Figure 5 is a flowchart of the fourth method provided in an embodiment of the present application;

[0045] Figure 6 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0047] In order to facilitate understanding of the solution provided by the embodiments of the present application, it is first necessary to explain that in order to achieve unified control of multiple different types of control signals, an AI generative model that can unify multiple different types of control signals can be trained, and then multiple different types of control signals can be input into the model to achieve a unified control method by combining multiple different types of control signals with text description content. For example, you can input text description content (describing the image to be generated) and multiple conditional images, which can include a depth map (and specifying reference to the depth information in the image), a contour map (and specifying reference to the contour in the image), an object map (and specifying the object to be generated in the image), a style map (and specifying the style in the image), and so on. In this way, you can use a variety of different conditional images to achieve unified control of the AI ​​generative model in multiple different aspects such as spatial structure, generated objects, and style.

[0048] Among them, how to train the above-mentioned AI generative model that can unify multiple different types of control signals is a problem that needs to be considered. If the control signal is simply added to the backbone network of the AI ​​generative model, the final generation effect may not be ideal.

[0049] In view of the above problems, the embodiments of the present application provide corresponding solutions. First, in terms of model structure, the basic Wenshengtu model has only one backbone network. When training, only the target image and the text description content used to describe the target image are usually required. During the training process, the above text description content can be used as input, and the model generates the image, and the training process is supervised by the target image. In the prediction stage, specific text description content can be input into the model, and the model generates the corresponding image. In the embodiment of the present application, first, an "adapter network" can be added to the backbone network of the basic "Wenshengtu" model. The training data can exist in the form of a quadruple. In addition to the target image and the text description content describing the target image, the quadruple can also include a conditional image and the control instruction text content corresponding to the conditional image (which can be used to describe the type of conditional image, or what dimension needs to be controlled, etc., for example, "reference the style in the image to generate an image"). Among them, each quadruple can include one or more conditional images, and the multiple conditional images can be images corresponding to multiple different control types. For example, the same quadruple may include a depth map, a contour map, a style map, and the like.

[0050] Among them, the conditional image and the corresponding control instruction description text are not directly input into the above-mentioned backbone network, but are first input into the aforementioned adapter network, and the adapter network performs feature extraction on the conditional image and the corresponding control instruction description text respectively to obtain the first feature vector and the second feature vector. Such feature vectors can be directly introduced into the backbone network as control information for the model. Alternatively, in a preferred implementation, after feature extraction of the conditional image and the control instruction text content is completed, multimodal interaction processing can be performed between the first feature vector and the second feature vector, so that the third feature vector corresponding to the conditional image and the fourth feature vector corresponding to the control instruction text content obtained by the multimodal interaction processing are determined as control information, and then injected into the backbone network. In other words, through multimodal interaction processing, the output is still two feature vectors, but due to multimodal interaction (that is, the interaction between image features and text features), both can contain richer information.

[0051] The above-mentioned multimodal interactive processing can be implemented in a variety of ways. For example, one of the ways can be implemented through MM-DiT (Modulated Multi-scale Decomposition Transformer, a multimodal diffusion model based on the Transformer architecture). Among them, MM-DiT is a key technology in the text graph model. It combines the self-attention mechanism of the Transformer to enable the text graph model to better match the text conditions when generating images. Specifically, when the user enters text or image prompts, the text is encoded by a pre-trained text model, and the image is converted to a latent space representation by a pre-trained autoencoder. Then, MM-DiT uses a modulation mechanism to integrate this information, thereby improving the quality and accuracy of the generated image. In an embodiment of the present application, an adapter network is designed based on the MM-DiT structure. Through the adapter network, multiple different types of control signals can be feature extracted separately, and then the MM-DiT module performs multimodal interactive processing, so that various control signals can obtain richer information from the feature vectors of other types of control signals, thereby enhancing the feature vectors corresponding to each control signal. Then, it can be combined with the backbone network of the Vincent graph model which is also based on the MM-DiT structure to realize the unified control of the Vincent graph model by multiple different types of control signals.

[0052] In addition, after the above-mentioned control information is obtained through feature extraction, multimodal interaction, etc., this control information can be injected into the backbone network. There can also be multiple specific injection methods. For example, in one method, direct addition can be used for injection. This method is more friendly to the control of spatial structure, but for object control (that is, the conditional image is related to a specific object, and an image related to the object needs to be generated), the generation effect will be relatively poor. In another way, the injection of control information can be achieved by cross-attention. This method may lose the correspondence between pixels and pixels. Therefore, it is not friendly enough for the control of spatial structure, but it is more friendly for object control. In other words, both introduction methods have their own advantages and disadvantages. However, considering that the problem of object control is more difficult to solve, the loss of spatial structure information can be compensated by additionally introducing the spatial coding information of the conditional image. Therefore, in an embodiment of the present application, the control information can be introduced by cross-attention, and the spatial coding information of the conditional image can be introduced into the backbone network at the same time to solve the problem of losing spatial structure information.

[0053] In the specific implementation, after the feature extraction of the conditional image and the control instruction text content, multimodal interaction processing is performed, and the control information is introduced based on the cross-attention method, and the spatial encoding information of the conditional image is also introduced into the backbone network. These two improvements can be combined together or exist separately.

[0054] From the perspective of system architecture, see Figure 1 , the embodiment of the present application adjusts the structure of the AI ​​generative model, and an adapter network can be added on the basis of the backbone network. Specifically, during training, in addition to obtaining the target image and the corresponding text description content, a variety of different types of conditional images and corresponding control instruction content can also be obtained. This conditional image and control instruction text content can be input into the adapter network for feature extraction, and in an optional manner, multimodal interaction can also be performed. Afterwards, the obtained control information can be introduced into the backbone network of the AI ​​generative model. In a preferred manner, the cross-attention method can be used for introduction, and the spatial encoding information of the conditional image (which can reflect the position coordinates of each pixel in the image and other information) is introduced into the backbone network to make up for the loss of the corresponding relationship information between pixels and pixels in the cross-attention method. After the output is completed through the backbone network, the loss function can be calculated, and the training process of the model can be completed through multiple rounds of iterations.

[0055] The AI ​​generative model trained in the above manner can unify multiple different types of control signals. That is, when the user needs to generate an image, he can input text description content, and can also input multiple conditional images of different control types into the AI ​​generative model at the same time. For example, it can include depth maps and contour maps for controlling spatial structures, images for generating specified objects, or style maps for generating specified styles, etc. After that, the adapter network is used to extract features and then injected into the backbone network to generate images.

[0056] The specific implementation scheme provided in the embodiments of the present application is described in detail below.

[0057] Embodiment 1

[0058] First, this embodiment provides an AI generative model training method, see Figure 2 , the method may include:

[0059] S201: Acquire multiple training data groups for training an artificial intelligence (AI) generative model, wherein the training data groups include: descriptive text content describing a target image used to supervise a training process, at least one conditional image, and control instruction text content corresponding to the conditional image; the multiple training data groups include multiple conditional images corresponding to multiple different control types.

[0060] Among them, the training data is the data used to train a specific AI generative model. In an embodiment of the present application, such training data may be in the form of a quadruple, and each quadruple may include: a target image used to supervise the training process, a descriptive text content describing the target image, a conditional image, and a control instruction text content corresponding to the conditional image. Among them, regarding the target image, that is, the image used to supervise the model training process, that is, the model can be trained with the output image of the model as close to the target image as possible. The text description content of the target image is the content used as the model input, which is equivalent to allowing the model to generate an image based on the text description content. When training the basic text image model, it is usually sufficient to provide the target image and the text description content in the training data. However, in an embodiment of the present application, it is also necessary to provide the conditional image and the corresponding control instruction text content in the training data. Among them, the conditional image can be the aforementioned depth map, contour map, style map, and so on. The control instruction text content is used to describe the image type, control dimension, etc. corresponding to the conditional image. For example, if the conditional image is a depth map, the control instruction text content can be "generate an image so that it has the depth information in the image", etc.

[0061] Among them, the same quadruple may include one or more conditional images, and the multiple conditional images may also correspond to multiple different control types, for example, they may include depth maps, contour maps, object maps, style maps, and so on. The conditional images included in each quadruple may be different. In short, through multiple quadruple elements, multiple different types of conditional images that need to be unified can be covered. For example, in a specific implementation, if 15 different types of control signals need to be unified, the collected training data may also cover the conditional images corresponding to these 15 control signals, and so on.

[0062] For example, the input quadruple could be of the form:

[0063] Target image: a picture of an afternoon tea theme

[0064] Text description corresponding to the target image: Delicious afternoon tea (in actual applications, it may be necessary to use English or other languages ​​for description)

[0065] Conditional image: edge map

[0066] Task instruction: Generate an image based on the edge map.

[0067] Among them, various condition images of different control types may include:

[0068] Depth map (depth), Canny edge map (canny), HED (Holistically-Nested Network) edge map (hed), semantic segmentation map (segmentation map), normal surface map (normal surface map), human pose skeleton (human pose skeleton), sketch (sketch), bounding box (bounding box), image outpainting, image colorization (colorization), image sharpening / deblurring (deblur), image partial repainting (inpainting), style reference map, object reference map, etc.

[0069] Specifically, the above training data set can be collected in a variety of ways, for example, by obtaining it from some open source data sets, or by manually constructing some such training data sets, etc., which are not limited here.

[0070] S202: Through the adapter network of the AI ​​generative model, feature extraction is performed on the condition image and the control instruction text content respectively to obtain control information.

[0071] After obtaining multiple training data quadruplets, they can be used to train a specific AI generative model. Among them, in an embodiment of the present application, the AI ​​generative model may include two parts: an adapter network and a backbone network. Regarding the target image and text description information in the quadruplet, they can be directly input into the backbone network for feature extraction and other processing. As for the conditional image and the control instruction text content, they can first be input into the adapter network for feature extraction to obtain specific control information. Among them, regarding feature extraction, that is to say, the input conditional image and control instruction text content can be expressed in the form of a feature vector. Specifically, a first feature vector corresponding to the conditional image and a second feature vector corresponding to the control instruction text content can be generated.

[0072] As mentioned above, the first feature vector and the second feature vector can be directly injected into the backbone network as feature information, or, in a preferred implementation, the first feature vector and the second feature vector can also be processed by multimodal interaction through models such as MM-DiT. That is to say, since the first feature vector is the feature vector of the image and the second feature vector is the feature vector of the text, the image and the text belong to different modes, so the two can be multimodally interacted so that each can obtain richer information from the other's feature vector, and respectively obtain the third feature vector corresponding to the conditional image and the fourth feature vector corresponding to the control instruction text content. Afterwards, the third feature vector and the fourth feature vector can be used as specific control information.

[0073] S203: Inject the control information into the backbone network of the AI ​​generative model through cross-attention, introduce the spatial encoding information of the conditional image into the backbone network, and complete the training of the AI ​​generative model, so as to uniformly control the AI ​​generative model through conditional images of multiple different control types.

[0074] After obtaining the control information, the control information can be injected into the backbone network of the AI ​​generative model by means of cross-attention. Among them, cross-attention is a mechanism for allocating importance that focuses on the relationship between different modalities or different sequences, and is widely used in multimodal tasks and sequence-to-sequence tasks. It dynamically adjusts the attention of each modality (or sequence) to other modalities (or sequences) by calculating the similarity between the query (Query) vector of one modality (or sequence) and the key (Key) and value (Value) vector of another modality (or sequence), thereby realizing the comprehensive utilization of information. In an embodiment of the present application, it is through this cross-attention mechanism that the AI ​​generative model realizes the comprehensive utilization of the control information corresponding to the conditional image and the text content of the control instruction.

[0075] Among them, both the backbone network and the adapter network can be multi-layer networks. In a preferred manner, in order to better realize the injection of control information in a cross-attention manner, the number of layers of the two can be the same. In specific implementation, the number of layers of the adapter network can be determined according to the number of layers of the backbone network of the AI ​​generative model. For example, assuming that the backbone network is 30 layers, the adapter network can also be a 30-layer network, and so on. In this way, when injecting control information by cross-attention, it can be done in the following way: after the conditional image and the control instruction text content are input into the adapter network, they will first enter the first layer of the adapter network. The output of the first layer will be input to the second layer of the adapter network on the one hand, and to the first layer of the backbone network on the other hand; the second output of the adapter network will also be input to the third layer of the adapter network on the one hand, and to the second layer of the backbone network on the other hand, and so on, until the backbone network completes the output of the last layer, that is, generates a specific image, and then completes a round of iteration by calculating the loss function and other methods. After multiple rounds of iterations to reach the training target, the training process of the model can be completed.

[0076] Among them, as mentioned above, due to the introduction of control information in a cross-attention manner, spatial structural information such as the correspondence between pixels may be lost. Therefore, in an embodiment of the present application, the spatial coding information of the conditional image can also be introduced into the backbone network. Among them, in the spatial domain, the digital image is manifested as a spatially distributed dot matrix, and its statistical characteristics (the image signal (brightness, chrominance, etc.) itself, or the random statistical characteristics of the output value after they are processed in a certain way) are used to reflect the correlation between any two pixels. For example, there is often a strong correlation between adjacent pixels in the same row and between pixels corresponding to adjacent rows, and so on. The spatial coding information of the image is specifically a method of encoding image information into a spatial pattern to express the correlation between pixels. In specific implementation, there can be a variety of different spatial coding methods. In an embodiment of the present application, any of the spatial coding methods can be used to encode the conditional image to obtain specific spatial coding information.

[0077] For the backbone network, since control information, spatial coding information and descriptive text content can be obtained, this information can be combined to achieve joint control of the AI ​​generative model, and the training process can be supervised using the target image in the training data quadruple.

[0078] That is to say, for the backbone network, if there are N layers, the adapter network structure can also have N layers (N adapters) accordingly. In each layer, the output of the corresponding adapter will be combined with the output result of the corresponding layer in the backbone network (in a cross-attention manner, that is, injected into the backbone network as mentioned in the article. When calculating the cross-attention, position encoding can be introduced, that is, for the backbone network features as the query and the adapter output features as the key, position encoding must be added first). The result of the cross-attention is then added to the output of the backbone network to obtain the final output of the layer; in addition, the output of the n-1th layer adapter will be input to the nth layer adapter, and the output of the nth layer adapter will be combined with the output of the nth layer of the backbone network; the features of the combination of the output of the n-1th layer in the backbone network and the output of the n-1th layer of the adapter will continue to be input into the backbone network for the next step of processing; until all layers of the network are processed, the predicted output is obtained. This output is used to calculate the loss function during training and for sampling during inference.

[0079] Specifically, during training, given the input four elements, the image encoder in the backbone network is used to extract the features of the conditional image and the target image, and the text encoders in the backbone network are used to extract the text features of the control instruction text content and the text description content; for the target image features, a random Gaussian noise needs to be generated and fused with the image features in the way of the backbone network, and then processed by a convolution, and then expanded into a vector representation. For the features of the text description, a linear layer can be used for processing; the conditional image and the control instruction text content are processed using similar operations, except that the conditional image does not need to be denoised. These features are respectively input into the MM-DiT module and the adapter module for processing; assuming that the final output is y. Then, the common regression (MSE) loss function in deep learning can be used for loss calculation. For example, loss = MSE (y, x_1-x_0), where x_1 is the feature of the target image before denoising, x_0 is the added noise, and MSE represents the common regression loss function.

[0080] In short, an adapter network can be added on the basis of the backbone network of the AI ​​generative model. When collecting specific training data, it can be in the form of a quadruple, that is, it can include a target image for supervising the training process, a descriptive text content describing the target image, at least one conditional image, and a control instruction text content corresponding to the conditional image. Among them, multiple training data groups can include multiple conditional images corresponding to multiple different control types, so that the training data can cover multiple different types of control signals. Specifically, during training, the adapter network of the AI ​​generative model can first be used to extract features of the conditional image and the control instruction text content to obtain control information; then, the control information can be injected into the backbone network of the AI ​​generative model by cross-attention, and the spatial encoding information of the conditional image can be introduced into the backbone network, so as to combine the control information, spatial encoding information and the descriptive text content to achieve control of the AI ​​generative model. Among them, since multiple training data groups include multiple conditional images corresponding to multiple different control types, after the AI ​​generative model is supervised and trained in the above manner, it is possible to achieve unified control of the AI ​​generative model through conditional images of multiple different control types, that is, unified control can be achieved by inputting multiple different types of control signals into the same AI generative model, so as to meet the image generation control requirements in multiple different scenarios through the same AI generative model. Among them, when injecting control information into the backbone network, a cross-attention mechanism is adopted, and the spatial encoding information of the conditional image can also be introduced into the backbone network to compensate for the loss caused by the spatial attention mechanism in the correlation between pixels.

[0081] In a preferred implementation, after the conditional image and the control instruction text content are respectively feature extracted, the first feature vector corresponding to the conditional image and the second feature vector corresponding to the control instruction text content can also be subjected to multimodal interactive processing, so that the third feature vector corresponding to the conditional image and the fourth feature vector corresponding to the control instruction text content obtained by the multimodal interactive processing are determined as the control information. In this way, each can obtain richer information from the other, which is conducive to improving the final image generation quality.

[0082] Embodiment 2

[0083] The above-mentioned embodiment 1 mainly aims at injecting control information into the backbone network of the AI ​​generative model by cross-attention, and introducing the spatial encoding information of the conditional image into the backbone network to realize the training of the model. In this embodiment 2, it is mainly aimed at protecting the solution of using multimodal interaction alone. Specifically, an AI generative model training method is provided, see Figure 3 , the method may include:

[0084] S301: Acquire multiple training data groups for training an artificial intelligence (AI) generative model, wherein the training data groups include: a description text content describing a target image for supervising a training process, at least one conditional image, and a control instruction text content corresponding to the conditional image; the multiple training data groups include multiple conditional images corresponding to multiple different control types;

[0085] S302: extracting features of the condition image and the control instruction text content respectively through the adapter network of the AI ​​generative model, and performing multimodal interactive processing on a first feature vector corresponding to the condition image and a second feature vector corresponding to the control instruction text content to obtain control information;

[0086] S303: Inject the control information into the backbone network of the AI ​​generative model so as to combine the control information with the description text content and complete the training of the AI ​​generative model so as to achieve unified control of the AI ​​generative model.

[0087] Embodiment 3

[0088] This third embodiment corresponds to the first embodiment, and protects a solution for using a trained model to generate images. Specifically, after the AI ​​generative model is trained in the aforementioned manner, when the model is used for reasoning, a conditional image, a corresponding task instruction, and a text description are given, and the image is generated according to the sampling method in the backbone network. In addition, although only one task (depth map, edge map, style map, etc.) can be considered for each input during training, after training, the model also supports the joint control of multiple tasks, such as depth map + style map. During reasoning, the features of each conditional image and corresponding task instruction are extracted separately, and are calculated separately with the backbone features in a cross-attention manner, and then the results are added together with the backbone network features; the rest are consistent with the single condition generation. Specifically, this third embodiment provides a method for image generation through an AI generative model, see Figure 4 , the method may include:

[0089] S401: receiving a description text content input by a user to describe an image to be generated, a plurality of condition images, and a control instruction text content corresponding to the condition images, wherein the plurality of condition images correspond to a plurality of different control types;

[0090] S402: extracting features of the condition image and the control instruction text content respectively through the adapter network of the AI ​​generative model to obtain control information;

[0091] S403: Injecting the control information into the backbone network of the AI ​​generative model by means of cross-attention, and introducing the spatial encoding information of the conditional image into the backbone network, so as to jointly control the AI ​​generative model through conditional images of multiple different control types, and generate a target image by the AI ​​generative model.

[0092] In specific implementation, after feature extraction is performed on the conditional image and the control instruction text content respectively, multimodal interactive processing can be performed on the first feature vector corresponding to the conditional image and the second feature vector corresponding to the control instruction text content, so that the third feature vector corresponding to the conditional image and the fourth feature vector corresponding to the control instruction text content obtained by the multimodal interactive processing can be determined as the control information.

[0093] Embodiment 4

[0094] This fourth embodiment corresponds to the second embodiment, and protects a scheme for jointly controlling a trained model through conditional images of multiple different control types to generate an image. Specifically, this fourth embodiment provides a method for generating an image through an AI generative model, see Figure 5 , the method may include:

[0095] S501: receiving a description text content input by a user to describe an image to be generated, a plurality of condition images, and a control instruction text content corresponding to the condition images, wherein the plurality of condition images correspond to a plurality of different control types;

[0096] S502: extracting features of the condition image and the control instruction text content respectively through the adapter network of the AI ​​generative model, and performing multimodal interactive processing on a first feature vector corresponding to the condition image and a second feature vector corresponding to the control instruction text content to obtain control information;

[0097] S503: Injecting the control information into the backbone network of the AI ​​generative model so as to jointly control the AI ​​generative model through conditional images of multiple different control types, and generating a target image by the AI ​​generative model.

[0098] For the parts not described in detail in Embodiments 2 to 4, please refer to the records in Embodiment 1 and other parts of this specification, which will not be repeated here.

[0099] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0100] Corresponding to the first embodiment, the embodiment of the present application further provides an AI generative model training device, including:

[0101] A first training data acquisition unit is used to acquire multiple training data groups for training an artificial intelligence AI generative model, wherein the training data groups include: a description text content describing a target image used to supervise the training process, at least one conditional image, and a control instruction text content corresponding to the conditional image; the multiple training data groups include multiple conditional images corresponding to multiple different control types;

[0102] A first control information obtaining unit, configured to extract features of the condition image and the control instruction text content respectively through an adapter network of the AI ​​generative model to obtain control information;

[0103] The first control information introduction unit is used to inject the control information into the backbone network of the AI ​​generative model by cross-attention, introduce the spatial encoding information of the conditional image into the backbone network, and complete the training of the AI ​​generative model, so as to uniformly control the AI ​​generative model through conditional images of multiple different control types.

[0104] Wherein, the device may further include:

[0105] A multimodal interaction processing unit is used to perform multimodal interaction processing on a first feature vector corresponding to the conditional image and a second feature vector corresponding to the control instruction text content after respectively extracting features from the conditional image and the control instruction text content, so as to determine a third feature vector corresponding to the conditional image and a fourth feature vector corresponding to the control instruction text content obtained by the multimodal interaction processing as the control information.

[0106] The control instruction text content corresponding to the conditional image is used to describe the control type and / or target control dimension of the conditional image.

[0107] The training data set also includes the target image for supervising the training process of the AI ​​generative model.

[0108] The adapter network and the backbone network are both multi-layer networks with the same number of layers.

[0109] Corresponding to the first embodiment, the embodiment of the present application further provides an AI generative model training device, including:

[0110] A second training data acquisition unit is used to acquire multiple training data groups for training the artificial intelligence AI generative model, wherein the training data groups include: a description text content describing a target image used to supervise the training process, at least one conditional image, and a control instruction text content corresponding to the conditional image; the multiple training data groups include multiple conditional images corresponding to multiple different control types;

[0111] A second control information acquisition unit is used to perform feature extraction on the condition image and the control instruction text content respectively through the adapter network of the AI ​​generative model, and perform multimodal interactive processing on the first feature vector corresponding to the condition image and the second feature vector corresponding to the control instruction text content to obtain control information;

[0112] The second control information injection unit is used to inject the control information into the backbone network of the AI ​​generative model so as to combine the control information with the description text content and complete the training of the AI ​​generative model so as to achieve unified control of the AI ​​generative model.

[0113] Corresponding to the third embodiment, the embodiment of the present application further provides a device for generating an image through an AI generative model, which may include:

[0114] A first receiving unit is used to receive a description text content input by a user to describe an image to be generated, a plurality of condition images, and a control instruction text content corresponding to the condition images, wherein the plurality of condition images correspond to a plurality of different control types;

[0115] A first control information obtaining unit, configured to extract features of the condition image and the control instruction text content respectively through an adapter network of the AI ​​generative model to obtain control information;

[0116] The first generation unit is used to inject the control information into the backbone network of the AI ​​generative model by means of cross-attention, and introduce the spatial encoding information of the conditional image into the backbone network, so as to jointly control the AI ​​generative model through conditional images of multiple different control types, and generate a target image by the AI ​​generative model.

[0117] These may also include:

[0118] A multimodal interaction processing unit is used to perform multimodal interaction processing on a first feature vector corresponding to the conditional image and a second feature vector corresponding to the control instruction text content after respectively extracting features from the conditional image and the control instruction text content, so as to determine a third feature vector corresponding to the conditional image and a fourth feature vector corresponding to the control instruction text content obtained by the multimodal interaction processing as the control information.

[0119] Corresponding to the fourth embodiment, the embodiment of the present application further provides a device for generating an image through an AI generative model, which may include:

[0120] A second receiving unit is used to receive a description text content input by a user to describe the image to be generated, a plurality of condition images, and a control instruction text content corresponding to the condition images, wherein the plurality of condition images correspond to a plurality of different control types;

[0121] A second control information obtaining unit is used to perform feature extraction on the condition image and the control instruction text content respectively through the adapter network of the AI ​​generative model, and perform multimodal interactive processing on the first feature vector corresponding to the condition image and the second feature vector corresponding to the control instruction text content to obtain control information;

[0122] The second generation unit is used to inject the control information into the backbone network of the AI ​​generative model so as to jointly control the AI ​​generative model through conditional images of multiple different control types, and generate a target image by the AI ​​generative model.

[0123] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0124] And an electronic device, comprising:

[0125] one or more processors; and

[0126] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0127] A computer program product includes a computer program / computer executable instructions, which, when executed by a processor in an electronic device, implement the steps of the method described in the aforementioned method embodiment.

[0128] in, Figure 6 The architecture of the electronic device is shown as an example, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620 may be communicatively connected via a communication bus 630.

[0129] Among them, the processor 610 can be implemented by a general-purpose CPU (Central Processing Unit, processor), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.

[0130] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store an operating system 621 for controlling the operation of the electronic device 600, and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device 600. In addition, a web browser 623, a data storage management system 624, and a model training processing system 625, etc. can also be stored. The above-mentioned model training processing system 625 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0131] The input / output interface 613 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0132] The network interface 614 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0133] The bus 630 comprises a pathway for transmitting information between the various components of the device (eg, the processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620).

[0134] It should be noted that, although the above device only shows a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, a memory 620, a bus 630, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0135] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0136] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0137] The training of the AI ​​generative model, the image generation method and the electronic device provided by this application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core idea of ​​this application. At the same time, for those skilled in the art, according to the idea of ​​this application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. An AI generative model training method, characterized in that: include: Acquire multiple training data groups for training an artificial intelligence (AI) generative model, wherein the training data groups include: a description text content describing a target image used to supervise a training process, at least one conditional image, and a control instruction text content corresponding to the conditional image; the multiple training data groups include multiple conditional images corresponding to multiple different control types; Through the adapter network of the AI ​​generative model, feature extraction is performed on the condition image and the control instruction text content respectively to obtain control information; The control information is injected into the backbone network of the AI ​​generative model by means of cross-attention, and the spatial encoding information of the conditional image is introduced into the backbone network, and the training of the AI ​​generative model is completed, so that the AI ​​generative model can be uniformly controlled through conditional images of multiple different control types.

2. The method according to claim 1, characterized in that Also includes: After feature extraction is performed on the conditional image and the control instruction text content respectively, multimodal interaction processing is performed on the first feature vector corresponding to the conditional image and the second feature vector corresponding to the control instruction text content, so that the third feature vector corresponding to the conditional image and the fourth feature vector corresponding to the control instruction text content obtained by the multimodal interaction processing can be determined as the control information.

3. The method according to claim 1, characterized in that The control instruction text content corresponding to the conditional image is used to describe the control type and / or target control dimension of the conditional image.

4. The method according to claim 1, characterized in that: The training data set also includes the target image for supervising the training process of the AI ​​generative model.

5. The method according to claim 1, characterized in that The adapter network and the backbone network are both multi-layer networks with the same number of layers.

6. An AI generative model training method, characterized in that: include: Acquire multiple training data groups for training an artificial intelligence (AI) generative model, wherein the training data groups include: a description text content describing a target image used to supervise a training process, at least one conditional image, and a control instruction text content corresponding to the conditional image; the multiple training data groups include multiple conditional images corresponding to multiple different control types; Through the adapter network of the AI ​​generative model, feature extraction is performed on the condition image and the control instruction text content respectively, and multimodal interactive processing is performed on the first feature vector corresponding to the condition image and the second feature vector corresponding to the control instruction text content to obtain control information; The control information is injected into the backbone network of the AI ​​generative model so as to combine the control information with the description text content, and complete the training of the AI ​​generative model so as to achieve unified control of the AI ​​generative model.

7. A method for image generation using an AI generative model, characterized in that: include: Receive a description text content input by a user to describe an image to be generated, a plurality of condition images, and a control instruction text content corresponding to the condition images, wherein the plurality of condition images correspond to a plurality of different control types; Through the adapter network of the AI ​​generative model, feature extraction is performed on the condition image and the control instruction text content respectively to obtain control information; The control information is injected into the backbone network of the AI ​​generative model by means of cross-attention, and the spatial encoding information of the conditional image is introduced into the backbone network, so that the AI ​​generative model can be jointly controlled by conditional images of multiple different control types, and the target image can be generated by the AI ​​generative model.

8. The method according to claim 7, characterized in that Also includes: After feature extraction is performed on the conditional image and the control instruction text content respectively, multimodal interaction processing is performed on the first feature vector corresponding to the conditional image and the second feature vector corresponding to the control instruction text content, so that the third feature vector corresponding to the conditional image and the fourth feature vector corresponding to the control instruction text content obtained by the multimodal interaction processing can be determined as the control information.

9. A method for image generation using an AI generative model, characterized in that: include: Receive a description text content input by a user to describe an image to be generated, a plurality of condition images, and a control instruction text content corresponding to the condition images, wherein the plurality of condition images correspond to a plurality of different control types; Through the adapter network of the AI ​​generative model, feature extraction is performed on the condition image and the control instruction text content respectively, and multimodal interactive processing is performed on the first feature vector corresponding to the condition image and the second feature vector corresponding to the control instruction text content to obtain control information; The control information is injected into the backbone network of the AI ​​generative model so that the AI ​​generative model is jointly controlled through conditional images of multiple different control types, and the target image is generated by the AI ​​generative model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

11. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 9.

12. A computer program product comprising a computer program / computer executable instructions, characterized in that: When the computer program / computer executable instructions are executed by a processor in an electronic device, the steps of the method according to any one of claims 1 to 9 are implemented.

Images