Image generation method and device and electronic equipment

Abstract

The invention provides an image generation method and device and electronic equipment. The method comprises the following steps: inputting a target text into an image generation model, and generating a basic image; establishing a semantic feature space based on the image features of the basic image; obtaining guide semantic features of the guide image in the semantic feature space; inputting the target text into an image generation model to generate an intermediate feature; adjusting the intermediate features based on the guide semantic features, and processing the adjusted intermediate features through the image generation model until the image generation model outputs a final image; wherein the image content of the final image is matched with the image content indicated by the target text, and the image structure of the final image is matched with the image structure information contained in the guide image. The method does not need a training model, saves the training time cost, and improves the image generation efficiency.

Description

Technical Field

[0001] The present invention relates to the field of game technology, and in particular to an image generating method, device and electronic equipment. Background Art

[0002] Pre-trained models can generate images based on text, for example, the Stable Diffusion model. In order to make the generated image have a specific spatial layout or style, it is necessary to add preset spatial conditions in the process of generating the image. In related technologies, adding a trainable network layer on the basis of the pre-trained model, such as ControlNet, can integrate spatial conditions into the image generation process so that the final generated image meets the specific spatial layout or style. This method requires retraining the pre-trained model and the trainable network layer, which has a high training time cost and reduces the efficiency of image generation. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an image generation method, device and electronic device to save training time cost and improve image generation efficiency.

[0004] In a first aspect, an embodiment of the present invention provides an image generation method, comprising: inputting a target text into an image generation model to generate a basic image; wherein the basic image includes image content indicated by the target text; establishing a semantic feature space based on image features of the basic image; wherein the semantic feature space includes specified image features of the basic image; obtaining a guiding semantic feature of a guiding image in the semantic feature space; wherein the guiding image includes image structure information; inputting the target text into the image generation model to generate intermediate features; adjusting the intermediate features based on the guiding semantic features, and processing the adjusted intermediate features through the image generation model until the image generation model outputs a final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

[0005] In a second aspect, an embodiment of the present invention provides an image generating device, the device comprising: a basic image generating module, used to input a target text into an image generating model to generate a basic image; wherein the basic image includes image content indicated by the target text; a space establishing module, used to establish a semantic feature space based on image features of the basic image; wherein the semantic feature space includes specified image features of the basic image; a feature acquiring module, used to acquire guiding semantic features of a guiding image in the semantic feature space; wherein the guiding image includes image structure information; a feature generating module, used to input the target text into the image generating model to generate intermediate features; a final image output module, used to adjust the intermediate features based on the guiding semantic features, and process the adjusted intermediate features through the image generating model until the image generating model outputs a final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

[0006] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the above-mentioned image generation method.

[0007] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned image generation method.

[0008] The embodiments of the present invention bring the following beneficial effects:

[0009] The above-mentioned image generation method, device and electronic device input the target text into the image generation model to generate a basic image; wherein the basic image contains the image content indicated by the target text; based on the image features of the basic image, a semantic feature space is established; wherein the semantic feature space includes the specified image features of the basic image; the guiding semantic features of the guiding image in the semantic feature space are obtained; wherein the guiding image includes image structure information; the target text is input into the image generation model to generate intermediate features; the intermediate features are adjusted based on the guiding semantic features, and the adjusted intermediate features are processed by the image generation model until the image generation model outputs the final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

[0010] In this method, a semantic feature space is established through a basic image generated by the target text, and a guiding semantic feature provided by a guiding image in the semantic feature space is obtained. The image generation process is guided based on the guiding semantic feature so that the image features match the image structure of the guiding image. This method does not require a training model, saves training time costs, and improves image generation efficiency.

[0011] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0012] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A schematic diagram of an image generated using ControlNet guidance in related technologies;

[0015] Figure 2 A flowchart of an image generation method provided by an embodiment of the present invention;

[0016] Figure 3 A schematic diagram of using LDA to process an image provided by an embodiment of the present invention;

[0017] Figure 4 A schematic diagram of a process of generating a semantic feature space provided by an embodiment of the present invention;

[0018] Figure 5 A schematic diagram of a flow chart of an image generation method provided by an embodiment of the present invention;

[0019] Figure 6 A schematic diagram of an embodiment of the present invention providing an example of generating an image by applying the image generating method of the embodiment;

[0020] Figure 7 A schematic diagram of the structure of an image generating device provided by an embodiment of the present invention;

[0021] Figure 8A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0023] To facilitate understanding, the terms involved in the embodiments of the present invention are first explained.

[0024] 1.Stable Diffusion (also known as latent diffusion model): an open source AI (Artificial Intelligence) model for generating images from text, referred to as SD model.

[0025] 2. WebUI (Website User Interface): An open source visualization program based on Stable Diffusion, mainly used for AI drawing creation. It provides functions such as text-generated images, image-generated images, AI zooming, text inversion, etc., and has a large number of diverse functional plug-ins.

[0026] 3.DreamMaker: AI drawing cloud platform based on Stable Diffusion.

[0027] 4.AIP: The integration package of the above WebUI reduces the difficulty of WebUI installation.

[0028] 5. Controlnet: An enhanced neural network model that controls image generation by adding additional conditions in Stable Diffusion. ControlNet has a variety of control methods to choose from to meet different painting needs, such as edge detection Canny, depth detection Depth, and action detection Openpose.

[0029] 6.LDA: Linear Discriminant Analysis; LDA is a supervised learning method for classification and dimensionality reduction. LDA maximizes the variance between categories and minimizes the variance within categories by finding linear combinations in the feature space, thereby achieving dimensionality reduction and classification of data. LDA is commonly used in pattern recognition, image processing, bioinformatics and other fields.

[0030] After the emergence of AI drawing technology, workers can use open source WEBUI, DreamMaker, AIP and other platforms or functions to generate images through text-generated images, image-generated images, etc. However, SD models can usually only generate images based on text descriptions, and it is difficult to control the spatial layout or style of the image.

[0031] In related technologies, images can be controlled through ControlNet, but ControlNet requires additional training for each guidance condition and model architecture, which is not only costly but also difficult to adapt to the ever-changing model architecture and user needs.

[0032] Specifically, ControlNet is a neural network architecture used to enhance the pre-trained text-to-image diffusion model, which improves the image generation process of the aforementioned SD model by adding spatial conditions. Here is how ControlNet works in detail:

[0033] ControlNet implements conditional control by adding a trainable network layer on top of a pre-trained model such as the Stable Diffusion model. This network layer, called ControlNet, can handle additional spatial conditional inputs such as edge maps, depth maps, or human pose maps.

[0034] ControlNet uses zero convolution layers to connect the fixed (or locked) part of the pre-trained model and the trainable copy. The zero convolution layer is a 1x1 convolution layer whose weights and biases are initialized to zero, which can avoid introducing harmful noise in the early stages of training and protect the deep features of the pre-trained model.

[0035] Users can provide additional conditional images, such as edge maps, depth maps, etc. ControlNet encodes these conditional images into feature vectors that match the latent space of the pre-trained model through a small network.

[0036] The following is the workflow of ControlNet:

[0037] The conditional image, e.g., edge map or depth map, is converted into a conditional vector through a small network so that the conditional vector matches the latent space of the pre-trained model.

[0038] During image generation, ControlNet inputs the conditional vector together with the textual cues into a diffusion model, such as the SD model. This makes the generation process guided not only by the textual cues but also by the spatial structure of the conditional image.

[0039] ControlNet adjusts the intermediate features in the generation process so that the generated image is consistent with the conditional image in spatial structure while maintaining the content of the text description.

[0040] The trainable copy of ControlNet gradually learns how to incorporate the spatial information of the conditional image into the generated image during training. This process does not require retraining the entire pre-trained model, but is achieved through fine-tuning.

[0041] In the inference stage, ControlNet uses the trained parameters to integrate the conditional information into the generation process of the diffusion model, generating an image that is consistent with the conditional image in spatial structure and conforms to the text description.

[0042] When using ControlNet, different SD models need to be retrained, and new guidance methods also need to be retrained. For example, point cloud data, Mseh data, etc. If ControlNet uses these data for guidance, ControlNet needs to be retrained, which has a high training time cost and reduces the efficiency of image generation.

[0043] In addition, the images generated using ControlNet guidance tend to be guided by pictures and weaken the guidance of text descriptions, resulting in the mismatch between the generated images and the text descriptions.

[0044] refer to Figure 1 In the example, the reference image is detected by edge detection to obtain edge features, which are then input into ControlNet and the output of ControlNet into the SD model to obtain the final image, whose structure refers to the structure in the reference image. When the description of the text prompt is more complex, the image generated in this way may not accurately reflect the content of the text prompt, and the image does not match the description of the text prompt.

[0045] Based on this, the embodiments of the present invention provide an image generation method, device and electronic device, which can be applied to generate various types of images.

[0046] See also Figure 2 An image generation method is shown, the method comprising the following steps:

[0047] Step S202, inputting the target text into the image generation model to generate a basic image; wherein the basic image contains the image content indicated by the target text;

[0048] The image generation model can be a diffusion model, such as the aforementioned SD model, or it can be other types of AI models that can generate images.

[0049] The image generation model can generate a basic image based on the target text. For example, if the target text includes 'a man', the basic image includes a man. For another example, if the target text includes 'a red hat', the man in the basic image wears a red hat.

[0050] Step S204, establishing a semantic feature space based on the image features of the basic image; wherein the semantic feature space includes the specified image features of the basic image;

[0051] After the target text is input into the image generation model, the encoder and decoder in the image generation model first generate latent space features, and then generate the above-mentioned basic image based on the latent space. The image features of the above-mentioned basic image can be obtained from the latent space features of the image generation model, or the features of the basic image can be reversed to obtain the image features.

[0052] The image features of the basic image usually include sub-features of multiple temperatures and multiple categories corresponding to the basic image, and some features can be filtered out from these sub-features as designated image features; for example, some main features can be filtered out as designated image features.

[0053] A semantic feature space is established based on the specified image features. The semantic feature space is a specific feature space that can represent the basic semantics of the basic image or the basic semantics of the same category as the basic image. For example, if the basic image includes a man, the semantic feature space represents the basic semantics of the image that includes a man.

[0054] Step S206, obtaining the guiding semantic features of the guiding image in the semantic feature space; wherein the guiding image includes image structure information;

[0055] The guide image includes image structure information, such as character posture, object shape, etc. In actual implementation, the image features of the guide image can be guided in advance, and then the image features are mapped to the above semantic feature space to obtain the guided semantic features.

[0056] Step S208, inputting the target text into the image generation model to generate intermediate features;

[0057] After the target text is input into the image generation model, a multi-step processing process is usually required. The steps can be represented by time steps. After each time step, an intermediate feature is generated until the final image is output.

[0058] The intermediate feature is usually a latent space feature. The image generation model includes multiple computing layers, and different computing layers output different features. The intermediate feature can be the feature output by all computing layers in the image generation model, or the feature output by a specified computing layer.

[0059] Step S210, adjusting the intermediate features based on the guiding semantic features, and processing the adjusted intermediate features through the image generation model until the image generation model outputs a final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

[0060] As mentioned above, the target text input into the image generation model is processed through multiple time steps, and each time step corresponds to an intermediate feature; the aforementioned guided semantic features can be adjusted for the intermediate features of each time step, and can also be adjusted for the intermediate features of some time steps.

[0061] The image generation model processes the adjusted intermediate features, for example, denoising, to generate a final image. The image content of the final image matches the content of the target text, and the image structure of the final image matches the image structure information contained in the guide image; for example, the pose of the person in the final image is the same or similar to the pose of the person in the guide image.

[0062] The above-mentioned image generation method inputs the target text into the image generation model to generate a basic image; wherein the basic image contains the image content indicated by the target text; based on the image features of the basic image, a semantic feature space is established; wherein the semantic feature space includes the specified image features of the basic image; the guiding semantic features of the guiding image in the semantic feature space are obtained; wherein the guiding image includes image structure information; the target text is input into the image generation model to generate intermediate features; the intermediate features are adjusted based on the guiding semantic features, and the adjusted intermediate features are processed by the image generation model until the image generation model outputs the final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

[0063] In this method, a semantic feature space is established through a basic image generated by the target text, and a guiding semantic feature provided by a guiding image in the semantic feature space is obtained. The image generation process is guided based on the guiding semantic feature so that the image features match the image structure of the guiding image. This method does not require a training model, saves training time costs, and improves image generation efficiency.

[0064] In a specific implementation method, the features of the base image are inferred to obtain the output features of the self-attention layer of the decoder in the image generation model; and a semantic feature space is established based on the output features.

[0065] Feature inversion tools can be used to infer features from the base image, for example, DDIM Inversion.

[0066] In order to avoid training the image generation model, it is necessary to find a unified semantic representation of the image structure, and this semantic identification is not affected by the image modality. Taking the SD model as an example, the SD model can obtain the composition features of the image and the conceptual features of the target text during the image generation process. Therefore, these features need to be remodeled, such as mapping them to the same space, to achieve a unified semantic representation of the features.

[0067] Among them, the concept feature is usually a noun, for example, a man, a car, etc.

[0068] During the research, the inventors found that the features output by the self-attention layer in the SD model have highly consistent semantic expressions in different image modalities. Based on this, the features of the self-attention layer of the decoder U-Net decoder in the SD model can be inferred through the aforementioned feature inversion tool to obtain the output features of the self-attention layer, and then establish a semantic feature space based on the output features.

[0069] Specifically, linear discriminant analysis is performed on the output features to obtain the semantic feature space.

[0070] Linear discriminant analysis is referred to as LDA. It can reduce the dimension of output features, extract main features, etc. After processing the output features, linear discriminant analysis outputs a set of eigenvalues ​​and the eigenvectors corresponding to each eigenvalue. The eigenvalue indicates the importance of the corresponding eigenvector. The eigenvector with a larger eigenvalue is used as the principal component feature in the output feature, and a semantic feature space is established based on the principal component feature.

[0071] exist Figure 3 In the example, the features of the original image after LDA processing have high consistency, where LDA clusters similar features in the original image; in the features after LDA processing, the foreground area appears purple-red, that is, the foreground area is mainly high-frequency features, and the background area appears blue-green, that is, the background area is mainly low-frequency features.

[0072] Figure 4 The generation process of the semantic feature space is shown. The target text is input into the SD model and the base image is output; the feature inversion tool can extract the diffusion features of each time step denoised in the self-attention layer of the SD model. When the base images generated based on the target text include multiple images, each base image corresponds to a set of diffusion features, which constitute the aforementioned output features.

[0073] The output features of the self-attention layer of the basic image are processed by LDA, such as dimensionality reduction, to obtain multiple feature vectors, and after filtering out the principal component features, a semantic feature space is established based on the principal component features. The semantic feature space links multiple different image modalities, so the guided semantic features in the subsequent guided image can be passed to the generation process of the final image.

[0074] This semantic feature space can be applied to the same target text, or target texts with the same concept, greatly reducing the data processing cost.

[0075] Furthermore, the guide image is feature inversely deduced to obtain the specified image features of the guide image; the specified image features are mapped to the semantic feature space to obtain the guide semantic features.

[0076] The feature inversion tool can be used to invert the features of the guide image, for example, DDIM Inversion. Specifically, the feature inversion tool is used to invert the features in the self-attention layer of the decoder U-Net decoder in the SD model to obtain the output features of the self-attention layer, that is, the aforementioned specified image features are the output features of the self-attention layer, and the processing method is the same as that of the base image.

[0077] The specified image features are subjected to linear discriminant analysis to obtain principal component features of the guided image. The principal component features are mapped to the semantic feature space to obtain guided semantic features.

[0078] After obtaining the guided semantic features, at each time step of processing the intermediate features, the intermediate features are adjusted by comparing the gap between the intermediate features and the guided semantic features so that the composition of the final image is consistent with that of the guided image.

[0079] In the process of adjusting the intermediate features based on the guiding semantic features, the intermediate features are mapped to the semantic feature space to obtain the intermediate semantic features; the feature distance between the intermediate semantic features and the guiding semantic features is determined; the intermediate features are adjusted until the feature distance reaches a minimum value, and the adjustment of the intermediate features is stopped.

[0080] When the intermediate features are adjusted, the intermediate semantic features change accordingly, and thus the feature distance changes; by adjusting the intermediate features multiple times, the intermediate features when the feature distance reaches the minimum value are used as the final intermediate features.

[0081] In actual implementation, the original noise elimination estimate of the image generation model is obtained; a first distance function between the guided semantic feature and the intermediate feature is established; the intermediate feature is adjusted until the first distance function reaches a first minimum value; based on the original noise elimination estimate and the gradient of the first distance function relative to the intermediate feature, the noise estimate of the image generation model is determined; and a final image is generated based on the noise estimate.

[0082] Among them, the first distance function g(S t , S gt ) is expressed as:

[0083]

[0084] Among them, S t is the intermediate feature, S gt To guide semantic features; [S t ] ij is the feature value at the coordinate position [i, j] in the intermediate semantic feature corresponding to the intermediate feature; [s gt ] ij is the feature value at the [i, j] coordinate position in the guided semantic feature.

[0085] The purpose of adjusting the intermediate features is to minimize the function value of the first distance function.

[0086] Noise estimate It is expressed as:

[0087]

[0088] Among them, ∈ t is the original noise-removed estimate, s is the guided weight coefficient, The gradient of the code first distance function with respect to the intermediate features.

[0089] In the above method, a guide image is provided, which contains the composition structure information that is expected to be reflected in the generated final image. The guide image is processed, its features are extracted and projected into the semantic feature space, and the guide image is converted into a form that matches the semantic feature space. The guided semantic features can dynamically adjust the image features at each time step of the generated image, so that the position of the image features in the semantic feature space is closer to the guided semantic features, thereby controlling the final image to produce an expected composition structure, ensuring that the image structure of the final image is consistent with the image structure of the guide image.

[0090] The guiding semantic features are used to make the image structure of the final image consistent with the image structure of the guiding image. In addition, in order to make the image appearance of the final image consistent with the target text, this embodiment also needs to obtain the appearance semantic features of the target text in the semantic feature space.

[0091] In order to ensure that the appearance of the final image is consistent with the description of the target text, it is necessary to extract appearance features from the image generation process based only on the target text without guidance from the guide image, and import them into the final image generation process guided by the guide image to ensure that the appearance features of the final image are the same as the description of the target text.

[0092] In actual implementation, the target text is input into the image generation model to obtain the latent space features of the target text; based on the latent space features and the semantic feature space, the appearance semantic features are generated.

[0093] When generating the latent space features of the target text, it is not necessary to input the aforementioned guide image or guide semantic features into the image generation model, and only the target text is input.

[0094] In a specific implementation method, the target text is input into an image generation model to obtain a diffusion feature corresponding to a specified time step; the diffusion feature is mapped to a semantic feature space to obtain a mapping feature; the diffusion feature corresponding to the specified time step and the weighted spatial mean of the mapping feature are determined, and the weighted spatial mean is determined as the appearance semantic feature.

[0095] When the target text is input into the image generation model, diffusion features of multiple time steps will be generated. The aforementioned specified time steps can be all time steps or part of the time steps. The diffusion features of the specified time steps contain the appearance information of different semantic parts corresponding to the target text.

[0096] For the diffusion feature Ft at a specified time step t, its weighted spatial mean It can be expressed as:

[0097]

[0098] Where σ is the sigmoid function, It is the mapping feature obtained after the diffusion feature is mapped to the semantic feature space. The feature value of the kth channel of the mapping feature at the spatial position (i, j), [f t ] ij is the characteristic value of the diffusion feature Ft at the spatial position (i, j) at the specified time step t. Encodes the appearance of different semantic components.

[0099] Specifically, the intermediate feature is mapped to the semantic feature space to obtain the intermediate semantic feature; the feature distance between the intermediate semantic feature and the appearance semantic feature is determined; the intermediate feature is adjusted until the feature distance reaches a minimum value, and the adjustment of the intermediate feature is stopped.

[0100] When the intermediate features are adjusted, the intermediate semantic features change accordingly, and thus the feature distance changes; by adjusting the intermediate features multiple times, the intermediate features when the feature distance reaches the minimum value are used as the final intermediate features.

[0101] In one approach, a second distance function between the appearance semantic feature and the intermediate feature is established, and the second distance function is expressed as:

[0102]

[0103] in, Represents the appearance semantic features, represents the intermediate features, that is, the image features generated without the guidance of appearance semantic features, N a Represents the number of appearance semantic features.

[0104] In order to maintain the conceptual appearance in the target text, visual guidance is introduced to guide the appearance details in the final image through appearance semantic features, so that the visual appearance of the final image is consistent with the conceptual appearance in the target text.

[0105] In one implementation, the intermediate features are adjusted based on the guiding semantic features and the appearance semantic features corresponding to the target text. By adjusting the intermediate features together with the guiding semantic features and the appearance semantic features, that is, jointly guiding the generation of the final image, the final image can be made consistent not only with the structure of the guiding image, but also with the appearance information of the target text.

[0106] Specifically, based on the specified time step and the target text, a first noise component of the image generation model is determined; based on the specified time step, a second noise component of the image generation model is determined; a first distance function between the guide semantic feature and the intermediate feature, and a second distance function between the appearance semantic feature and the intermediate feature are established; and the intermediate feature is adjusted until the first distance function reaches a first minimum value and the second distance function reaches a second minimum value.

[0107] Among them, the first noise component is represented by ∈ θ (X t ; t, c), refers to the noise component predicted by the image generation model such as the pre-trained diffusion model at the specified time step t and target text c.

[0108] The second noise component is expressed as refers to the same specified time step t, but without any text condition (i.e., the empty set ), the noise component predicted by the image generation model in the case of . The second noise component represents the unconditional generation process, that is, the process of the image generation model generating images without text guidance.

[0109] Furthermore, based on the first noise component, the second noise component, the first minimum value of the first distance function and the second minimum value of the second distance function, a noise estimation value of the image generation model is determined; and a final image is generated based on the noise estimation value.

[0110] The above noise estimate It can be expressed as:

[0111]

[0112] Among them, s is the guided weight coefficient, ∈ θ(X t ; t, c) is the aforementioned first noise component, is the aforementioned second noise component, λ s is the intensity parameter of composition guidance, g s is the first minimum value of the first distance function, λ a is the visually guided intensity parameter, g a is the second minimum value of the second distance function.

[0113] Through the above method, the image features are dynamically adjusted during the image generation process, and the final image generated is not only consistent with the guide image in composition, but also consistent with the target text in appearance; this method effectively combines the control of structure and appearance, and improves the quality and accuracy of image generation.

[0114] Figure 5 The overall process diagram of image generation is shown, which is divided into two pipelines, the generation pipeline and the guidance pipeline. In the guidance pipeline, the guidance image generates guidance semantic features based on the SD model and the pre-established semantic feature space, and the target text generates appearance semantic features based on the SD model and the semantic feature space. In the generation pipeline, the target text is input into the SD model to generate intermediate features, which are adjusted by the appearance semantic features and the guidance semantic features to generate the final image.

[0115] Figure 6 An example of the final image generated by the image generation method of this embodiment is shown; wherein the first row of images is a guide image, the last row is a target text, the second row, the third row and the fourth row are final images output by multiple image generation models, and the image generation models are SD1.5 model, SD2.1 model and SD-XL1.0 model respectively.

[0116] The image generation method provided in this embodiment does not require additional training for each new guidance condition or model architecture, which greatly saves training cost and time. Through the construction and application of semantic feature space, it can flexibly adapt to different guidance images and text descriptions, realize highly customized image generation, and support a variety of guidance images, such as sketches, normal maps, depth maps, edge maps, human postures, segmentation masks, natural images, etc., as well as various model architectures and customized training models; it can also better achieve the alignment of images and texts. When faced with a conflict between the guidance image and the text description, it can better balance the two and generate an image that conforms to both the spatial structure and the text description.

[0117] Corresponding to the above method embodiment, see Figure 7 An image generating device as shown, the device comprising:

[0118] A basic image generation module 70 is used to input the target text into the image generation model to generate a basic image; wherein the basic image contains the image content indicated by the target text;

[0119] A space establishment module 72, used to establish a semantic feature space based on the image features of the basic image; wherein the semantic feature space includes the specified image features of the basic image;

[0120] A feature acquisition module 74 is used to acquire the guiding semantic features of the guiding image in the semantic feature space; wherein the guiding image includes image structure information;

[0121] A feature generation module 76, for inputting the target text into the image generation model to generate intermediate features;

[0122] The final image output module 78 is used to adjust the intermediate features based on the guiding semantic features, and process the adjusted intermediate features through the image generation model until the image generation model outputs the final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

[0123] The above-mentioned image generation device inputs the target text into the image generation model to generate a basic image; wherein the basic image contains the image content indicated by the target text; based on the image features of the basic image, a semantic feature space is established; wherein the semantic feature space includes the specified image features of the basic image; the guiding semantic features of the guiding image in the semantic feature space are obtained; wherein the guiding image includes image structure information; the target text is input into the image generation model to generate intermediate features; the intermediate features are adjusted based on the guiding semantic features, and the adjusted intermediate features are processed by the image generation model until the image generation model outputs the final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

[0124] In this method, a semantic feature space is established through a basic image generated by the target text, and a guiding semantic feature provided by a guiding image in the semantic feature space is obtained. The image generation process is guided based on the guiding semantic feature so that the image features match the image structure of the guiding image. This method does not require a training model, saves training time costs, and improves image generation efficiency.

[0125] The above-mentioned space establishment module is used to: perform feature inversion on the basic image to obtain the output features of the self-attention layer of the decoder in the image generation model; and establish a semantic feature space based on the output features.

[0126] The above-mentioned space establishment module is used to: perform linear discriminant analysis on the output features to obtain a semantic feature space.

[0127] The feature acquisition module is used to: perform feature inversion on the guide image to obtain the specified image features of the guide image; and map the specified image features to the semantic feature space to obtain the guide semantic features.

[0128] The above device also includes an appearance feature acquisition module, which is used to acquire the appearance semantic features of the target text in the semantic feature space.

[0129] The appearance feature acquisition module is used to input the target text into the image generation model to obtain the latent space features of the target text; and generate appearance semantic features based on the latent space features and the semantic feature space.

[0130] The above-mentioned appearance feature acquisition module is used to input the target text into the image generation model to obtain the diffusion feature corresponding to the specified time step; map the diffusion feature to the semantic feature space to obtain the mapping feature; determine the diffusion feature corresponding to the specified time step and the weighted spatial mean of the mapping feature, and determine the weighted spatial mean as the appearance semantic feature.

[0131] The final image output module is used to map the intermediate features to the semantic feature space to obtain the intermediate semantic features; determine the feature distance between the intermediate semantic features and the guiding semantic features; adjust the intermediate features until the feature distance reaches the minimum value, and stop adjusting the intermediate features.

[0132] The above-mentioned final image output module is used to: obtain the original noise elimination estimation value of the image generation model; establish a first distance function between the guided semantic feature and the intermediate feature; adjust the intermediate feature until the first distance function reaches a first minimum value; determine the noise estimation value of the image generation model based on the original noise elimination estimation value and the gradient of the first distance function relative to the intermediate feature; and generate a final image based on the noise estimation value.

[0133] The final image output module is used to adjust the intermediate features based on the guiding semantic features and the appearance semantic features corresponding to the target text.

[0134] The above-mentioned final image output module is used to: map the intermediate features to the semantic feature space to obtain the intermediate semantic features; determine the feature distance between the intermediate semantic features and the appearance semantic features; adjust the intermediate features until the feature distance reaches the minimum value, and stop adjusting the intermediate features.

[0135] The above-mentioned final image output module is used to: determine the first noise component of the image generation model based on the specified time step and the target text; determine the second noise component of the image generation model based on the specified time step; establish a first distance function between the guiding semantic feature and the intermediate feature, and a second distance function between the appearance semantic feature and the intermediate feature; adjust the intermediate feature until the first distance function reaches a first minimum value and the second distance function reaches a second minimum value.

[0136] The final image output module is used to: determine the noise estimation value of the image generation model based on the first noise component, the second noise component, the first minimum value of the first distance function and the second minimum value of the second distance function; and generate the final image based on the noise estimation value.

[0137] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the above-mentioned image generation method. The electronic device can be a server or a terminal device.

[0138] See also Figure 8 As shown, the electronic device includes a processor 100 and a memory 101 , wherein the memory 101 stores computer executable instructions that can be executed by the processor 100 , and the processor 100 executes the computer executable instructions to implement the above-mentioned image generating method.

[0139] Further, Figure 8 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 100 , the communication interface 103 and the memory 101 are connected via the bus 102 .

[0140] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0141] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 100. The above processor 100 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the method of the above embodiment in combination with its hardware.

[0142] The processor in the above electronic device can implement the following operations in the above image generation method by executing computer executable instructions:

[0143] A method for generating an image comprises inputting a target text into an image generation model to generate a basic image; wherein the basic image contains image content indicated by the target text; establishing a semantic feature space based on image features of the basic image; wherein the semantic feature space contains specified image features of the basic image; obtaining a guiding semantic feature of a guiding image in the semantic feature space; wherein the guiding image contains image structure information; inputting the target text into an image generation model to generate intermediate features; adjusting the intermediate features based on the guiding semantic features, and processing the adjusted intermediate features through the image generation model until the image generation model outputs a final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

[0144] The features of the base image are inferred to obtain the output features of the self-attention layer of the decoder in the image generation model; and a semantic feature space is established based on the output features.

[0145] Perform linear discriminant analysis on the output features to obtain the semantic feature space.

[0146] The features of the guide image are inversely deduced to obtain the specified image features of the guide image; the specified image features are mapped to the semantic feature space to obtain the guide semantic features.

[0147] Obtain the appearance semantic features of the target text in the semantic feature space.

[0148] The target text is input into the image generation model to obtain the latent space features of the target text; based on the latent space features and the semantic feature space, the appearance semantic features are generated.

[0149] The target text is input into the image generation model to obtain the diffusion features corresponding to the specified time step; the diffusion features are mapped to the semantic feature space to obtain the mapping features; the diffusion features corresponding to the specified time step and the weighted spatial mean of the mapping features are determined, and the weighted spatial mean is determined as the appearance semantic feature.

[0150] The intermediate feature is mapped to the semantic feature space to obtain the intermediate semantic feature; the feature distance between the intermediate semantic feature and the guiding semantic feature is determined; the intermediate feature is adjusted until the feature distance reaches a minimum value, and the adjustment of the intermediate feature is stopped.

[0151] Obtain an original noise elimination estimate of an image generation model; establish a first distance function between a guided semantic feature and an intermediate feature; adjust the intermediate feature until the first distance function reaches a first minimum value; determine a noise estimate of the image generation model based on the original noise elimination estimate and a gradient of the first distance function relative to the intermediate feature; and generate a final image based on the noise estimate.

[0152] The intermediate features are adjusted based on the guided semantic features and the corresponding appearance semantic features of the target text.

[0153] The intermediate features are mapped to the semantic feature space to obtain the intermediate semantic features; the feature distance between the intermediate semantic features and the appearance semantic features is determined; the intermediate features are adjusted until the feature distance reaches a minimum value, and the adjustment of the intermediate features is stopped.

[0154] Based on a specified time step and a target text, a first noise component of an image generation model is determined; based on a specified time step, a second noise component of an image generation model is determined; a first distance function between a guide semantic feature and an intermediate feature, and a second distance function between an appearance semantic feature and an intermediate feature are established; and the intermediate feature is adjusted until the first distance function reaches a first minimum value and the second distance function reaches a second minimum value.

[0155] Based on the first noise component, the second noise component, the first minimum value of the first distance function, and the second minimum value of the second distance function, a noise estimation value of the image generation model is determined; and a final image is generated based on the noise estimation value.

[0156] In this method, a semantic feature space is established through a basic image generated by the target text, and a guiding semantic feature provided by a guiding image in the semantic feature space is obtained. The image generation process is guided based on the guiding semantic feature so that the image features match the image structure of the guiding image. This method does not require a training model, saves training time costs, and improves image generation efficiency.

[0157] This embodiment also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned image generation method.

[0158] The computer executable instructions stored in the computer readable storage medium can implement the following operations in the image generation method by executing the computer executable instructions:

[0159] A method for generating an image comprises inputting a target text into an image generation model to generate a basic image; wherein the basic image contains image content indicated by the target text; establishing a semantic feature space based on image features of the basic image; wherein the semantic feature space contains specified image features of the basic image; obtaining a guiding semantic feature of a guiding image in the semantic feature space; wherein the guiding image contains image structure information; inputting the target text into an image generation model to generate intermediate features; adjusting the intermediate features based on the guiding semantic features, and processing the adjusted intermediate features through the image generation model until the image generation model outputs a final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

[0160] The features of the base image are inferred to obtain the output features of the self-attention layer of the decoder in the image generation model; and a semantic feature space is established based on the output features.

[0161] Perform linear discriminant analysis on the output features to obtain the semantic feature space.

[0162] The features of the guide image are inversely deduced to obtain the specified image features of the guide image; the specified image features are mapped to the semantic feature space to obtain the guide semantic features.

[0163] Obtain the appearance semantic features of the target text in the semantic feature space.

[0164] The target text is input into the image generation model to obtain the latent space features of the target text; based on the latent space features and the semantic feature space, the appearance semantic features are generated.

[0165] The target text is input into the image generation model to obtain the diffusion features corresponding to the specified time step; the diffusion features are mapped to the semantic feature space to obtain the mapping features; the diffusion features corresponding to the specified time step and the weighted spatial mean of the mapping features are determined, and the weighted spatial mean is determined as the appearance semantic feature.

[0166] The intermediate feature is mapped to the semantic feature space to obtain the intermediate semantic feature; the feature distance between the intermediate semantic feature and the guiding semantic feature is determined; the intermediate feature is adjusted until the feature distance reaches a minimum value, and the adjustment of the intermediate feature is stopped.

[0167] Obtain an original noise elimination estimate of an image generation model; establish a first distance function between a guided semantic feature and an intermediate feature; adjust the intermediate feature until the first distance function reaches a first minimum value; determine a noise estimate of the image generation model based on the original noise elimination estimate and a gradient of the first distance function relative to the intermediate feature; and generate a final image based on the noise estimate.

[0168] The intermediate features are adjusted based on the guided semantic features and the corresponding appearance semantic features of the target text.

[0169] The intermediate features are mapped to the semantic feature space to obtain the intermediate semantic features; the feature distance between the intermediate semantic features and the appearance semantic features is determined; the intermediate features are adjusted until the feature distance reaches a minimum value, and the adjustment of the intermediate features is stopped.

[0170] Based on a specified time step and a target text, a first noise component of an image generation model is determined; based on a specified time step, a second noise component of an image generation model is determined; a first distance function between a guide semantic feature and an intermediate feature, and a second distance function between an appearance semantic feature and an intermediate feature are established; and the intermediate feature is adjusted until the first distance function reaches a first minimum value and the second distance function reaches a second minimum value.

[0171] Based on the first noise component, the second noise component, the first minimum value of the first distance function, and the second minimum value of the second distance function, a noise estimation value of the image generation model is determined; and a final image is generated based on the noise estimation value.

[0172] In this method, a semantic feature space is established through a basic image generated by the target text, and a guiding semantic feature provided by a guiding image in the semantic feature space is obtained. The image generation process is guided based on the guiding semantic feature so that the image features match the image structure of the guiding image. This method does not require a training model, saves training time costs, and improves image generation efficiency.

[0173] The computer program products of the image generation method, device and electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0175] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0176] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0177] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0178] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image generation method, characterized in that: The method comprises: Inputting the target text into the image generation model to generate a basic image; wherein the basic image contains the image content indicated by the target text; Based on the image features of the basic image, a semantic feature space is established; wherein the semantic feature space includes the specified image features of the basic image; Acquire a guiding semantic feature of a guiding image in the semantic feature space; wherein the guiding image includes image structure information; Inputting the target text into the image generation model to generate intermediate features; The intermediate features are adjusted based on the guiding semantic features, and the adjusted intermediate features are processed by the image generation model until the image generation model outputs a final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

2. The method according to claim 1, characterized in that The step of establishing a semantic feature space based on the image features of the basic image comprises: Performing feature inversion on the base image to obtain output features of the self-attention layer of the decoder in the image generation model; A semantic feature space is established based on the output features.

3. The method according to claim 2, characterized in that The step of establishing a semantic feature space based on the output features comprises: Performing linear discriminant analysis on the output features to obtain a semantic feature space.

4. The method according to claim 1, characterized in that: The step of obtaining the guiding semantic feature of the guiding image in the semantic feature space comprises: Performing feature inversion on the guide image to obtain specified image features of the guide image; The designated image feature is mapped to the semantic feature space to obtain a guided semantic feature.

5. The method according to claim 1, characterized in that After the step of establishing a semantic feature space based on the image features of the basic image, the method further comprises: Acquire the appearance semantic features of the target text in the semantic feature space.

6. The method according to claim 5, characterized in that The step of obtaining the appearance semantic features of the target text in the semantic feature space comprises: Inputting the target text into the image generation model to obtain latent space features of the target text; Based on the latent space features and the semantic feature space, appearance semantic features are generated.

7. The method according to claim 6, characterized in that The step of inputting the target text into the image generation model to obtain the latent space features of the target text includes: Inputting the target text into the image generation model to obtain diffusion features corresponding to a specified time step; The step of generating appearance semantic features based on the latent space features and the semantic feature space comprises: Mapping the diffusion feature to the semantic feature space to obtain a mapping feature; Determine the diffusion feature corresponding to the specified time step and the weighted spatial mean of the mapping feature, and determine the weighted spatial mean as the appearance semantic feature.

8. The method according to claim 1, characterized in that: The step of adjusting the intermediate feature based on the guiding semantic feature comprises: Mapping the intermediate feature to the semantic feature space to obtain an intermediate semantic feature; Determining a feature distance between the intermediate semantic feature and the guiding semantic feature; The intermediate feature is adjusted until the feature distance reaches a minimum value, and the adjustment of the intermediate feature is stopped.

9. The method according to claim 8, characterized in that The step of adjusting the intermediate features based on the guiding semantic features, and processing the adjusted intermediate features through the image generation model until the image generation model outputs a final image includes: Obtaining an original noise-removed estimate of the image generation model; Establishing a first distance function between the guiding semantic feature and the intermediate feature; Adjusting the intermediate feature until the first distance function reaches a first minimum value; determining a noise estimate of the image generation model based on the original noise-removed estimate and a gradient of the first distance function relative to the intermediate feature; A final image is generated based on the noise estimate.

10. The method according to claim 1, characterized in that The step of adjusting the intermediate feature based on the guiding semantic feature comprises: The intermediate feature is adjusted based on the guiding semantic feature and the appearance semantic feature corresponding to the target text.

11. The method according to claim 10, characterized in that The step of adjusting the intermediate feature based on the appearance semantic feature corresponding to the target text comprises: Mapping the intermediate feature to the semantic feature space to obtain an intermediate semantic feature; Determining a feature distance between the intermediate semantic feature and the appearance semantic feature; The intermediate feature is adjusted until the feature distance reaches a minimum value, and the adjustment of the intermediate feature is stopped.

12. The method according to claim 10, characterized in that The step of adjusting the intermediate feature based on the guiding semantic feature and the appearance semantic feature corresponding to the target text comprises: Determine a first noise component of the image generation model based on a specified time step and the target text; determine a second noise component of the image generation model based on the specified time step; Establishing a first distance function between the guide semantic feature and the intermediate feature, and a second distance function between the appearance semantic feature and the intermediate feature; The intermediate feature is adjusted until the first distance function reaches a first minimum value and the second distance function reaches a second minimum value.

13. The method according to claim 12, characterized in that The step of processing the adjusted intermediate features by the image generation model until the image generation model outputs a final image comprises: determining a noise estimate of the image generation model based on the first noise component, the second noise component, a first minimum value of the first distance function, and a second minimum value of the second distance function; A final image is generated based on the noise estimate.

14. An image generating device, characterized in that: The device comprises: A basic image generation module, used for inputting the target text into the image generation model to generate a basic image; wherein the basic image contains the image content indicated by the target text; A space establishment module, used to establish a semantic feature space based on the image features of the basic image; wherein the semantic feature space includes the specified image features of the basic image; A feature acquisition module, used to acquire the guiding semantic features of the guiding image in the semantic feature space; wherein the guiding image includes image structure information; A feature generation module, used for inputting the target text into the image generation model to generate intermediate features; A final image output module is used to adjust the intermediate features based on the guiding semantic features, and process the adjusted intermediate features through the image generation model until the image generation model outputs a final image; wherein the image content of the final image matches the image content indicated by the target text, and the image structure of the final image matches the image structure information contained in the guiding image.

15. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the image generating method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the image generation method according to any one of claims 1 to 13.

Images