Image generation method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an image generation method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring a line reference picture, a style reference picture and text description; fusing the line image features obtained based on the line reference image and the text features obtained based on the text description to obtain image-text control features; fusing the image-text control feature, the text feature and a style image feature obtained based on the style reference image to obtain an image-text fusion feature; and obtaining a noise image used for generating a target image, and inputting the noise image and the structure control feature, the image-text control feature, the style image feature and the image-text fusion feature obtained by the line reference image into the diffusion model for iteration to obtain a target image output by the diffusion model. According to the method, generation of the image can be continuously adjusted and optimized in combination with multiple features, the quality of the generated image is guaranteed, and meanwhile the generated image better conforms to the real intention of a user.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image generation method, device, electronic device and readable storage medium. Background Art

[0002] With the application and progress of artificial intelligence technology in the field of images, image generation has become more efficient and convenient. In related technologies, with the continuous advancement of multimodal technology, the technology of creating images required by users based on text and reference images is changing with each passing day. However, existing technologies still have limitations in interpreting the true intentions of users, resulting in the inability to generate high-quality images that meet user needs. Summary of the invention

[0003] In view of this, embodiments of the present invention provide an image generation method, device, electronic device and readable storage medium to solve the problem that the prior art cannot interpret the user's true intention when generating images, resulting in the inability to generate high-quality images that meet user needs.

[0004] According to a first aspect of an embodiment of the present invention, there is provided an image generation method, the method comprising: A line reference image, a style reference image and a text description are obtained; the line image features obtained based on the line reference image and the text features obtained based on the text description are fused to obtain image-text control features; the image-text control features, the text features and the style image features obtained based on the style reference image are fused to obtain image-text fusion features; a noise image used to generate a target image is obtained, and the noise image, the structure control features obtained based on the line reference image, the image-text control features, the style image features and the image-text fusion features are input into a diffusion model for iteration to obtain a target image output by the diffusion model.

[0005] According to a second aspect of an embodiment of the present invention, there is provided an image generating device, the device comprising: The acquisition module is configured to acquire a line reference image, a style reference image and a text description; the first fusion module is configured to fuse the line image features obtained based on the line reference image and the text features obtained based on the text description to obtain the image-text control features; the second fusion module is configured to fuse the image-text control features, the text features and the style image features obtained based on the style reference image to obtain the image-text fusion features; the image generation module is configured to acquire a noise image for generating a target image, and input the noise image, the structure control features obtained based on the line reference image, the image-text control features, the style image features and the image-text fusion features into the diffusion model for iteration to obtain the target image output by the diffusion model.

[0006] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of an embodiment of the present invention, a readable storage medium is provided, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0008] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: The embodiment of the present invention obtains a line reference image, a style reference image and a text description to determine the multi-faceted real demand expression for the target image, fuses the line image features obtained based on the line reference image and the text features obtained based on the text description to obtain the image-text control features; fuses the image-text control features, the text features and the style image features obtained based on the style reference image to obtain the image-text fusion features; so that the obtained image-text control features and image-text fusion features meet both the structural requirements and the text content requirements, and more accurately fit the user's expectations for the overall look and feel and specific details of the target image. A noise image for generating the target image is obtained, and the structural control features, image-text control features, style image features and image-text fusion features obtained from the noise image and the line reference image are input into the diffusion model for iteration to obtain the target image output by the diffusion model. In this way, the generation of the image can be continuously adjusted and optimized in combination with multiple features to ensure the quality of the generated image, and at the same time make the generated image more in line with the user's real intention. The limitation of the prior art that it is difficult to fully and accurately interpret the user's intention and cannot generate high-quality images that meet the needs is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 is a flow chart of an image generating method provided by an embodiment of the present invention; Figure 2 is a structural example diagram of a fusion module provided by an embodiment of the present invention; Figure 3 is a structural schematic diagram of a diffusion submodule provided by an embodiment of the present invention; Figure 4 is a structural schematic diagram of an image generating device provided by an embodiment of the present invention; Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0012] An image generating method, an apparatus, an electronic device and a readable storage medium according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0013] Figure 1 FIG. 1 is a flow chart of an image generation method provided by an embodiment of the present invention. Figure 1 As shown, the method includes: S101, obtaining a line reference image, a style reference image, and a text description; S102, fusing the line image features obtained based on the line reference image and the text features obtained based on the text description to obtain the image-text control features; S103, fusing the image-text control feature, the text feature, and the style image feature obtained based on the style reference image to obtain an image-text fusion feature; S104, obtaining a noise image for generating a target image, and inputting the noise image, structural control features, image-text control features, style image features, and image-text fusion features obtained based on a line reference image into a diffusion model for iteration to obtain a target image output by the diffusion model.

[0014] Specifically, the image generation method of this embodiment can be executed by a client or a server, or by a client and a server together. The following is an example in which the execution subject is a client. The line reference map, style reference map and text description are reference information for generating a target image, wherein the line reference map can be a line drawing and / or a simple drawing, the line reference map can represent the general structure, outline and other information of the generated target image, the style reference map can represent the style characteristics of the target image that the user expects to generate, such as realistic, cartoon, abstract and other styles; the text description can supplement the user's expectations of the target image from more specific semantic information and other aspects. By comprehensively obtaining these different forms of input, the user's intention can be more comprehensively understood.

[0015] It should be noted that after obtaining the line reference image, style reference image and text description, the corresponding encoder can be used to perform feature processing on the line reference image, style reference image and text description to obtain the line image features and structural control features corresponding to the line reference image, the style image features corresponding to the style reference image and the text features corresponding to the text description. Exemplarily, the line reference image and the style reference image can be processed by an image encoder, such as a convolutional neural network encoder, a Transformer encoder, etc., to obtain the line image features corresponding to the line reference image and the style image features corresponding to the style reference image, wherein the line image features contain the visual information of the line reference image, and the style image features contain the style and content information of the style reference image, which will be used to affect the style of the generated image. Further, the line reference image is feature processed by a Controlnet network to obtain the structural control features corresponding to the line reference image, and the text description is processed by a corresponding text encoder to obtain the text features corresponding to the text description, and the structural control features will be used to guide the subsequent image generation process to ensure that the image structure of the generated target image is consistent with the line reference image. The text features contain the semantic information in the text description, which will be used to guide the generated image to conform to the content of the text description.

[0016] It can be understood that the line image features obtained based on the line reference image and the text features obtained based on the text description are fused to obtain the image-text control features. This realizes the organic combination of the image's structural information (line image features) and the semantic information (text features) contained in the text description. The line image features reflect the basic framework of the image, and the text features contain the user's more detailed requirements for the image. The fused image-text control features can consider both the structural layout of the image and the specific expectations of the user through the text description at a comprehensive feature level, so that the subsequent generation process can proceed in a direction that meets both the structural requirements and the text content requirements based on this fused feature.

[0017] Furthermore, the image-text control features, text features, and style image features obtained based on the style reference graph are fused to obtain image-text fusion features. On the basis of the fusion of line image features and text features, style image features are further added for fusion. Style is an important factor affecting image quality and whether it meets user preferences. By fusing image-text control features (line image features and text features have been fused), text features (text requirements are further strengthened), and style image features (clear style requirements) into image-text fusion features, the generated images can closely meet the user's various needs in terms of content, structure, and style, and more accurately meet the user's expectations for the overall look and feel and specific details of the target image.

[0018] Furthermore, a noise image is introduced and combined with the multiple features obtained by the previous fusion (structural control features, image-text control features, style image features, image-text fusion features) to input the diffusion model for iteration. As the iteration process progresses, the generation of the image is continuously adjusted and optimized based on the various features input. The comprehensive input of multiple features enables the diffusion model to fully consider the needs of users from different angles in each iteration, and gradually generate high-quality images that meet the user's true intentions from multiple aspects such as image structure construction, content filling, and style shaping. Through this iterative generation method, the target image obtained by utilizing the synergy of multiple features avoids the limitation of existing technologies that cannot generate high-quality images that meet the needs due to the difficulty in fully and accurately interpreting the user's intentions.

[0019] According to the technical solution provided by the embodiment of the present invention, by obtaining a line reference image, a style reference image and a text description, the multi-faceted real demand expression for the target image is determined, and the line image features obtained based on the line reference image and the text features obtained based on the text description are integrated to obtain the image-text control features; the image-text control features, the text features and the style image features obtained based on the style reference image are integrated to obtain the image-text fusion features; so that the obtained image-text control features and image-text fusion features meet both the structural requirements and the text content requirements, and more accurately fit the user's expectations for the overall look and feel and specific details of the target image. A noise image for generating the target image is obtained, and the structural control features, image-text control features, style image features and image-text fusion features obtained from the noise image and the line reference image are input into the diffusion model for iteration to obtain the target image output by the diffusion model. In this way, the generation of the image can be continuously adjusted and optimized in combination with multiple features to ensure the quality of the generated image, and at the same time make the generated image more in line with the user's real intention. The limitation of the prior art that it is difficult to fully and accurately interpret the user's intention and cannot generate high-quality images that meet the needs is avoided.

[0020] In some embodiments, line image features obtained based on a line reference image and text features obtained based on a text description are fused to obtain image-text control features, including: semantically enhancing text features to obtain enhanced text features; fusing line image features and enhanced text features to obtain cross-fusion features; and determining image-text control features based on the cross-fusion features, text features, and line image features.

[0021] Specifically, before fusing the line image features and the text features, the text features are first semantically enhanced to obtain enhanced text features. For example, the text features can be processed using a self-attention mechanism to capture long-distance dependencies within the text features, enhance the semantic representation of the text, and obtain enhanced text features.

[0022] Furthermore, the line image features and the enhanced text features are fused to obtain cross-fusion features. The line image features mainly reflect the structural information such as the line shape, direction, and connection relationship in the line reference image, while the enhanced text features carry the user's richer and more accurate semantic requirements for the target image. The two features are fused to organically combine the structural information of the image and the semantic requirements described by the text at a new feature level, so that the generated image can meet the basic structural framework set by the line and the content conveyed by the text. In the fusion process, a simple splicing method or a cross-attention method can be used. Exemplarily, this embodiment preferably uses a cross-attention method to fuse the line image features and the enhanced text features. Specifically, the enhanced text features can be used as queries (Q), and the line image features can be used as keys (K) and values ​​(V) to fuse the enhanced text features and the line image features. For each dimension of the line image features and each dimension of the enhanced text features, their contribution during fusion is adjusted according to the attention weight. In this way, the relationship between the two features can be more reasonably reflected in the cross-fusion features obtained by fusion, so that those feature parts that are more relevant to user intentions can receive more attention, thereby achieving more accurate and richer multimodal information expression and generation.

[0023] Furthermore, after obtaining the cross-fusion features, the image-text control features are determined based on the cross-fusion features, text features, and line image features, so as to balance the image structure (from the line image features), the user's semantic needs (from the text features), and the information on the combination of structure and semantics embodied by the cross-fusion features that have been initially integrated, to ensure that when generating images, one aspect will not be overly emphasized while other aspects will not be neglected, such as only focusing on the line structure and generating an image lacking semantic connotations, nor will only emphasizing semantic needs and causing a messy generated image structure. This will more accurately guide image generation in the direction of meeting user needs.

[0024] In some embodiments, the image-text control feature is determined based on the cross-fusion feature, the text feature and the line image feature, including: using a learnable cross-attention module to process the line image feature to obtain a learnable image feature; fusing the learnable image feature with the cross-fusion feature to obtain an image cross-fusion feature; and performing feature connection on the image cross-fusion feature and the text feature to obtain the image-text control feature.

[0025] Specifically, the learnable cross-attention module can characterize that the parameters in the module can be continuously adjusted and optimized during the model training process. By learning from a large amount of labeled data or unsupervised data, the module can automatically find the best association method and fusion weight between different modal data, so as to better adapt to specific tasks and data characteristics.

[0026] It can be understood that the line image features are input into the learnable cross-attention module. This module will re-weight the line image features according to its own learnable parameters and the input line image features by calculating the attention weights between each part of the line image features and other possible related information to obtain learnable image features. For example, a line image feature may represent the features of a portrait of a person outlined by simple lines, including the contour lines, body lines and other information of the person. After being processed by the cross-attention module, the line features of the person's head may be dynamically assigned different weights according to the subsequent fusion requirements with text features, etc., so that when they are subsequently fused with other features, certain line parts can be more prominent or weakened to better fit the overall generation goal.

[0027] Furthermore, the learnable image features are fused with the cross-fusion features to obtain image cross-fusion features. Through this fusion, the structural information represented by the line image features, the learnable image feature information after cross-attention adjustment, and the semantic information conveyed by the text features can be more comprehensively and deeply combined at a new feature level. For example, the learnable image features may have a weighted representation that is more in line with the overall goal for certain key structural parts in the line image (such as the eye lines in the portrait of a person). When fused with the cross-fusion features, this optimized representation for the key structure can be better integrated with other elements in the cross-fusion features (such as the features corresponding to the "vivid image" description related to the text semantics, etc.), thereby obtaining an image cross-fusion feature that can reflect the key points of the image structure and take into account the text semantics.

[0028] Furthermore, the image cross-fusion feature and the text feature are feature-connected to obtain the image-text control feature. In this way, the image structure and semantic comprehensive information contained in the image cross-fusion feature after fusion again are directly connected with the most original and pure text semantic information retained in the text feature, so that the image-text control feature can be in a unified feature representation, while covering all the key information of the image after multiple processing and fusion and the complete semantic information of the text. For example, the image cross-fusion feature may contain all the information about the character image shaping from the line structure to the fusion with the text semantics, such as the character's outline, expression and other fusion information related to the text description "lively and cute". The text feature may be the feature vector corresponding to the original description such as "lively and cute". Through feature connection, the image-text control feature can fully integrate this information together, so that in the subsequent image generation process, as a comprehensive control factor, it can guide the generated image to meet the structural and fusion semantic requirements of the image, and accurately reflect the semantic connotation conveyed by the text, so as to more accurately meet the user's needs for generating images.

[0029] It should be noted that the feature connection can use the residual connection method to connect the image cross-fusion features and the text features to prevent the gradient disappearance and gradient explosion problems of the network during the training process. In some examples, the image cross-fusion features and the text features are connected, including: using a feedforward network to extract the image cross-fusion features, and performing a residual connection on the image cross-fusion features after feature extraction and the text features to obtain the image-text control features. In some embodiments, the image-text control feature, the text feature, and the style image feature obtained based on the style reference graph are fused to obtain the image-text fusion feature, including: A cross-attention calculation is performed on the text features and the style image features to obtain the initial fusion features; a cross-attention calculation is performed on the initial fusion features and the image-text control features to obtain the image-text fusion features.

[0030] Specifically, separate text features and style image features can only describe part of the image information from their respective perspectives. The initial fusion features obtained by cross-attention calculation fuse information of two different dimensions together, enriching the feature representation while avoiding the problem of semantic and style separation that may be caused by simple splicing. The image-text control feature is obtained by integrating multiple aspects of information such as line image features, text features, and cross-fusion features after fusion, and controls the content and structure of image generation to a certain extent. The initial fusion feature is mainly a fusion of text and style. The image-text fusion feature obtained by cross-attention calculation can integrate the fusion results of these two important stages. The generated image has both the correct structure and the appropriate style and semantic atmosphere.

[0031] In some embodiments, the diffusion model includes a plurality of diffusion submodules arranged in succession; a noise image, a structure control feature, a picture-text control feature, a style image feature, and a picture-text fusion feature obtained based on a line reference image are input into the diffusion model for iteration to obtain a target image output by the diffusion model, including: taking the noise image as the target input of the first diffusion submodule, and inputting the structure control feature, the picture-text control feature, the style image feature, and the picture-text fusion feature into each diffusion submodule; using the first diffusion submodule to process the noise image, the structure control feature, the picture-text control feature, the style image feature, and the picture-text fusion feature to obtain the target output of the first diffusion submodule; using the target output of the first diffusion submodule as the target input of the second diffusion submodule, and using the second diffusion submodule to process the target output, the structure control feature, the picture-text control feature, the style image feature, and the picture-text fusion feature of the first diffusion submodule to obtain the target output of the second diffusion submodule; using the target output of the second diffusion submodule as the target input of the third diffusion submodule, and repeatedly executing the processing flow of the diffusion submodule until the target image output by the last diffusion submodule is obtained.

[0032] Specifically, multiple diffusion submodules in the diffusion model are connected in sequence. The first diffusion submodule takes the noise image as the target input, and uses the structure control feature, image-text control feature, style image feature and image-text fusion feature to denoise the noise image and obtain the corresponding target output. The input features each carry important information about different aspects of the target image. The structure control feature helps to determine the basic structure and layout of the image, such as the general outline structure of a character portrait or the overall framework of an architectural image; the image-text control feature integrates the fusion information of the text description and image-related features, and can accurately control the content and semantic direction of image generation; the style image feature clarifies the style characteristics of the image, such as realistic, cartoon, retro and other styles; the image-text fusion feature further integrates the results of the fusion of the previous multiple features, affecting image generation from a more comprehensive perspective, and multiple features can work together to guide the image generation process.

[0033] Furthermore, the target output of the first diffusion submodule will be used as the target input of the second diffusion submodule. The target input will be denoised using the structural control features, image-text control features, style image features, and image-text fusion features to obtain the target output of the second diffusion submodule. The target output of the second diffusion submodule will be used as the target input of the third diffusion submodule, and the processing flow of the diffusion submodule will be repeated until the target image output by the last diffusion submodule is obtained. By inputting the noise image and multiple features (structural control features, image-text control features, style image features, and image-text fusion features) into each diffusion submodule for processing, each diffusion submodule can further optimize the image based on the output of the previous submodule and in combination with these features. In this iterative process, the feature representation of the image will increasingly meet user needs and the desired target image.

[0034] In some embodiments, the processing flow of each diffusion submodule includes: The target input, image-text fusion features and style image features are fused to obtain the initial fusion features; the image-text control features and the time steps of the corresponding diffusion submodules are processed using the multi-layer perceptron module respectively, and the processed image-text control features and the corresponding time steps are fused to obtain the time fusion features; the target output is determined according to the target input, initial fusion features, structural control features and time fusion features.

[0035] Specifically, in the processing process of each diffusion submodule, the target input, the image-text fusion feature and the style image feature are first fused to obtain the initial fusion feature. Specifically, the target input, the image-text fusion feature and the style image feature are fused to obtain the initial fusion feature, including: using the self-attention mechanism to process the target input, and fusing the processed target input with the image-text fusion feature to obtain the initial image-text fusion feature; fusing the initial image-text fusion feature with the style image feature to obtain the initial fusion feature. When fusing the processed target input with the image-text fusion feature, the processed target input can be used as a query (Q), and the image-text fusion feature can be used as a key (K) and a value (V) for cross-attention calculation to obtain the initial image-text fusion feature. When fusing the initial image-text fusion feature with the style image feature to obtain the initial fusion feature, the initial image-text fusion feature can be used as a query (Q), and the style image feature can be used as a key (K) and a value (V) for cross-attention calculation to obtain the initial fusion feature.

[0036] Furthermore, the multi-layer perceptron module (MLP) is used to process the image-text control feature and the time step of the corresponding diffusion submodule respectively, and the processed image-text control feature and the corresponding time step are fused to obtain the time fusion feature; after being processed by their respective MLPs, the time step and the global image-text control feature will obtain the corresponding output feature vector. And the two output feature vectors are fused, so that the information about the generation stage contained in the time step and the image generation control information contained in the image-text control feature can be combined to form a new time fusion feature. For example, if the feature vector obtained after the time step is processed by MLP represents the information such as the degree of attention to the image contour in the current generation stage, and the feature vector obtained after the global image-text control feature is processed by MLP contains information about the specific content, style, etc. of the image, then the fused time fusion feature has both information, providing a more comprehensive feature basis for further integration into the network.

[0037] Further, according to the target input, the initial fusion feature, the structural control feature and the time fusion feature, the output target output corresponding to each diffusion submodule is obtained. How to obtain the output target output will be described in detail in the subsequent embodiments, and will not be described in detail here.

[0038] In some embodiments, a target output is determined based on a target input, initial style image features, structural control features, and temporal fusion features, including: normalizing the initial style image features and the temporal fusion features, and feature-connecting the normalized initial style image features and the temporal fusion features with the target input to obtain residual fusion features; extracting features from the residual fusion features, and normalizing the extracted residual fusion features and the temporal fusion features to obtain target fusion features; fusing the structural control features, the temporal fusion features, and the target fusion features to obtain the target output.

[0039] Specifically, the initial style image features and the time fusion features are first normalized to ensure that the feature values ​​are operated on the same scale, which helps to improve the stability and convergence speed of the model. Normalization can be achieved through a variety of methods, such as adaptive layer normalization (AdaLN), standardization (z-score normalization), min-max normalization, etc. In this embodiment, adaptive layer normalization (AdaLN) is preferably used to normalize the style image features and the time fusion features. Then, the normalized features are connected to the target input to form residual fusion features, which can retain the original information of the target input, while introducing the guidance of style and time fusion features, and exemplarily performing residual connection with the target input to form residual fusion features.

[0040] Furthermore, feature extraction is performed on the residual fusion features. Specifically, a feedforward network can be used to extract features from the residual fusion features. More valuable feature information that can better reflect the target output requirements can be mined from the residual fusion features. At the same time, some redundant information is removed and key features are highlighted, so that subsequent fusion operations can be performed based on more refined and effective features, thereby improving the quality of the final target output. The residual fusion features and time fusion features after feature extraction are normalized to obtain the target fusion features. The reason for normalization again is similar to the previous one, that is, the residual fusion features and time fusion features after feature extraction are kept in a suitable state in terms of feature value range, etc., so as to better perform subsequent fusion operations, further stabilize the training process of the model, and improve the training effect of the model and the quality of the final output.

[0041] Finally, the structural control features, temporal fusion features, and target fusion features are fused to obtain the final target output. The fusion process can adopt a variety of strategies, such as weighted summation and concatenation, to ensure that different features can effectively interact and complement each other. In this way, the image generation method can comprehensively consider information such as structure, style, content, and time, and finally generate high-quality images that meet user needs through multiple iterations.

[0042] Figure 2 FIG. 1 is a schematic diagram of a fusion module provided in an embodiment of the present invention. Figure 2 As shown: It can be understood that the fusion module can be used to fuse line image features and text features to obtain image and text control features. The specific processing flow is as follows: First, the self-attention layer is used to semantically enhance the text features to obtain enhanced text features; Then, the line image features and enhanced text features are fused using the cross attention layer to obtain the cross fusion features; Then, the line image features are processed using a learnable cross-attention module to obtain learnable image features; Secondly, the learnable image features are fused with the cross-fusion features to obtain the image cross-fusion features; Finally, the feedforward network is used to extract the image cross-fusion features, and the image cross-fusion features after feature extraction are residually connected with the text features to obtain the image-text control features.

[0043] Figure 3 Schematic diagram of a diffusion submodule provided by an embodiment of the present invention. Figure 3 As shown, It can be understood that the diffusion submodule can iterate the noise image at each time step according to the input data. The specific processing flow of each diffusion submodule is as follows: First, the target input is processed using the self-attention layer, and the processed target input is fused with the image-text fusion feature using the cross-attention layer to obtain the initial image-text fusion feature; Then, the cross attention layer is used again to fuse the initial image-text fusion features and the style image features to obtain the initial fusion features; Then, MLP is used to process the image-text control features and the time steps of the corresponding diffusion submodules, and the processed image-text control features and the corresponding time steps are fused to obtain the time fusion features; Secondly, the initial style image features and the time fusion features are input into the AdaLN layer for normalization, and the normalized initial style image features and the time fusion features are residually connected with the target input to obtain the residual fusion features; Then, the residual fusion features are extracted using the feedforward network layer, and the residual fusion features and time fusion features after feature extraction are input into the AdaLN layer for normalization to obtain the target fusion features; Finally, the structural control features, time fusion features and target fusion features are fused to obtain the target output.

[0044] It should be understood that the order of execution of the steps in the above embodiment does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiment of the present invention.

[0045] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present invention, which will not be described in detail here.

[0046] The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0047] Figure 4 Schematic diagram of the structure of an image generating device provided by an embodiment of the present invention. Figure 4 As shown, the device comprises: An acquisition module 401 is configured to acquire a line reference image, a style reference image, and a text description; The first fusion module 402 is configured to fuse the line image feature obtained based on the line reference image and the text feature obtained based on the text description to obtain the image-text control feature; The second fusion module 403 is configured to fuse the image-text control feature, the text feature, and the style image feature obtained based on the style reference image to obtain an image-text fusion feature; The image generation module 404 is configured to obtain a noise image for generating a target image, and input the noise image, structural control features obtained based on a line reference image, image and text control features, style image features, and image and text fusion features into a diffusion model for iteration to obtain a target image output by the diffusion model.

[0048] In some embodiments, the first fusion module 402 is further configured to semantically enhance text features to obtain enhanced text features; fuse line image features and enhanced text features to obtain cross-fusion features; and determine graphic control features based on cross-fusion features, text features, and line image features.

[0049] In some embodiments, the first fusion module 402 is further configured to process line image features using a learnable cross-attention module to obtain learnable image features; fuse the learnable image features with the cross-fusion features to obtain image cross-fusion features; and perform feature connection on the image cross-fusion features and text features to obtain image-text control features.

[0050] In some embodiments, the second fusion module 403 is further configured to perform cross-attention calculation on the text features and the style image features to obtain initial fusion features; and perform cross-attention calculation on the initial fusion features and the image-text control features to obtain image-text fusion features.

[0051] In some embodiments, the image generation module 404 is also configured to use the noise image as the target input of the first diffusion sub-module, and input the structural control features, image-text control features, style image features and image-text fusion features into each diffusion sub-module; use the first diffusion sub-module to process the noise image, structural control features, image-text control features, style image features and image-text fusion features to obtain the target output of the first diffusion sub-module; use the target output of the first diffusion sub-module as the target input of the second diffusion sub-module, and use the second diffusion sub-module to process the target output, structural control features, image-text control features, style image features and image-text fusion features of the first diffusion sub-module to obtain the target output of the second diffusion sub-module; use the target output of the second diffusion sub-module as the target input of the third diffusion sub-module, and repeat the processing flow of the diffusion sub-module until the target image output by the last diffusion sub-module is obtained.

[0052] In some embodiments, the image generation module 404 is further configured to fuse the target input, the image-text fusion features and the style image features to obtain an initial fusion feature; respectively process the image-text control features and the time steps of the corresponding diffusion submodule using a multi-layer perceptron module, and fuse the processed image-text control features and the corresponding time steps to obtain a time fusion feature; determine the target output based on the target input, the initial fusion feature, the structural control feature and the time fusion feature.

[0053] In some embodiments, the image generation module 404 is also configured to normalize the initial style image features and the time fusion features, and feature connect the normalized initial style image features and the time fusion features with the target input to obtain residual fusion features; perform feature extraction on the residual fusion features, and normalize the residual fusion features and the time fusion features after feature extraction to obtain target fusion features; fuse the structural control features, the time fusion features and the target fusion features to obtain the target output.

[0054] Figure 5 Schematic diagram of an electronic device 5 provided by an embodiment of the present invention. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0055] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0056] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0057] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0058] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0059] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0060] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. An image generation method, characterized in that: include: Get line references, style references and text descriptions; Fusing the line image features obtained based on the line reference image and the text features obtained based on the text description to obtain image-text control features; The image-text control feature, the text feature, and the style image feature obtained based on the style reference graph are fused to obtain an image-text fusion feature; A noise image for generating a target image is obtained, and the noise image, the structural control features obtained based on the line reference image, the image-text control features, the style image features, and the image-text fusion features are input into a diffusion model for iteration to obtain the target image output by the diffusion model.

2. The method according to claim 1, characterized in that The step of fusing the line image features obtained based on the line reference image and the text features obtained based on the text description to obtain the image and text control features includes: Performing semantic enhancement on the text features to obtain enhanced text features; Fusing the line image feature and the enhanced text feature to obtain a cross-fusion feature; The graphic and text control feature is determined according to the cross-fusion feature, the text feature and the line image feature.

3. The method according to claim 2, characterized in that The step of determining the graphic control feature according to the cross-fusion feature, the text feature and the line image feature includes: Processing the line image features using a learnable cross-attention module to obtain learnable image features; Fusing the learnable image feature with the cross-fusion feature to obtain an image cross-fusion feature; The image cross-fusion feature and the text feature are feature-connected to obtain the image-text control feature.

4. The method according to claim 1, characterized in that The step of fusing the image-text control feature, the text feature, and the style image feature obtained based on the style reference graph to obtain an image-text fusion feature includes: Performing cross attention calculation on the text features and the style image features to obtain initial fusion features; A cross-attention calculation is performed on the initial fusion feature and the image-text control feature to obtain the image-text fusion feature.

5. The method according to claim 1, characterized in that The diffusion model includes a plurality of diffusion submodules arranged in series; the noise image, the structure control feature obtained based on the line reference image, the image-text control feature, the style image feature and the image-text fusion feature are input into the diffusion model for iteration to obtain the target image output by the diffusion model, including: The noise image is used as the target input of the first diffusion submodule, and the structure control feature, the image-text control feature, the style image feature and the image-text fusion feature are input into each diffusion submodule; Using a first diffusion submodule to process the noise image, the structure control feature, the image-text control feature, the style image feature, and the image-text fusion feature, to obtain a target output of the first diffusion submodule; The target output of the first diffusion submodule is used as the target input of the second diffusion submodule, and the target output of the first diffusion submodule, the structure control feature, the image-text control feature, the style image feature, and the image-text fusion feature are processed by the second diffusion submodule to obtain the target output of the second diffusion submodule; The target output of the second diffusion submodule is used as the target input of the third diffusion submodule, and the processing flow of the diffusion submodule is repeatedly executed until the target image output by the last diffusion submodule is obtained.

6. The method according to claim 5, characterized in that The processing flow of each diffusion submodule includes: Fusing the target input, the image-text fusion feature, and the style image feature to obtain an initial fusion feature; The image-text control feature and the time step of the corresponding diffusion submodule are processed by a multi-layer perceptron module respectively, and the processed image-text control feature and the corresponding time step are fused to obtain a time fusion feature; The target output is determined according to the target input, the initial fusion feature, the structural control feature and the time fusion feature.

7. The method according to claim 6, characterized in that The determining the target output according to the target input, the initial style image feature, the structure control feature and the time fusion feature includes: Normalizing the initial style image features and the time fusion features, and performing feature connection on the normalized initial style image features and the time fusion features and the target input to obtain residual fusion features; Extracting features from the residual fusion features, and normalizing the residual fusion features and the time fusion features after feature extraction to obtain target fusion features; The structural control feature, the time fusion feature and the target fusion feature are fused to obtain the target output.

8. An image generating device, characterized in that: include: An acquisition module, configured to acquire a line reference image, a style reference image, and a text description; A first fusion module is configured to fuse the line image feature obtained based on the line reference image and the text feature obtained based on the text description to obtain an image-text control feature; A second fusion module is configured to fuse the image-text control feature, the text feature, and the style image feature obtained based on the style reference graph to obtain an image-text fusion feature; The image generation module is configured to obtain a noise image for generating a target image, and input the noise image, the structural control features obtained based on the line reference image, the image and text control features, the style image features and the image and text fusion features into a diffusion model for iteration to obtain the target image output by the diffusion model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images