Image generation method, apparatus, device, storage medium, and computer program product

Abstract

The present disclosure relates to an image generation method, device, equipment, storage medium and computer program product. The image generation method comprises: obtaining an input image, input text and a reference image, wherein the input image is used to describe the content of a target image in the form of an image, the input text is used to describe the content of the target image in the form of text, and the reference image is used to determine the style of the target image; obtaining a first feature of the input image, a second feature of the input text and a third feature of the reference image; performing feature fusion on the second feature and the third feature to obtain a text-image fusion feature; and generating the target image based on the first feature and the text-image fusion feature. The present method can retain rich visual information in the reference image when generating a target image with the image content feature of the reference image according to a control condition, thereby improving the accuracy of the generated target image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and in particular to image generation methods, apparatus, devices, storage media, and computer program products. Background Technology

[0002] With the development of artificial intelligence (AI) technology, various AI models are being widely used in a variety of application scenarios. For example, in image generation scenarios, images can be generated based on text-to-image generators.

[0003] In related technologies, one image generation method involves generating an image that contains some features of the reference image and has a semantic similarity to the given text, based on given text or a reference image. However, this method cannot accurately generate the image required by the user. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an image generation method, apparatus, device, storage medium, and computer program product.

[0005] According to some embodiments of this disclosure, an image generation method is provided, comprising: acquiring an input image, input text, and a reference image, wherein the input image is used to describe the content of a target image in image form, the input text is used to describe the content of the target image in text form, and the reference image is used to determine the style of the target image; acquiring a first feature of the input image, a second feature of the input text, and a third feature of the reference image; fusing the second feature and the third feature to obtain an image-text fusion feature; and generating the target image based on the first feature and the image-text fusion feature.

[0006] According to some embodiments of this disclosure, the third feature of the reference image is obtained by extracting image content features that suppress semantic information in the reference image as the third feature, wherein the suppressed semantic information indicates that the semantic information in the reference image does not participate in feature extraction.

[0007] According to some embodiments of this disclosure, the step of extracting image content features that suppress semantic information in the reference image includes: dividing the reference image into multiple sub-blocks, randomly sorting the sub-blocks and extracting image features from each sub-block to obtain sub-block content features; and determining the image content features that suppress semantic information based on the sub-block content features.

[0008] According to some embodiments of this disclosure, determining the image content features of the suppressed semantic information based on the sub-block content features includes: encoding each sub-block to obtain image encoding features; randomly interpolating and fusing the image encoding features with the randomly rearranged sub-block content features to obtain randomly interpolated and fused sub-block content features; and determining the image content features of the suppressed semantic information based on the randomly interpolated and fused sub-block content features.

[0009] According to some embodiments of this disclosure, determining the image content features of the suppressed semantic information based on the content features of the randomly interpolated and fused sub-blocks includes: performing global feature extraction on the content features of the randomly interpolated and fused sub-blocks belonging to the same reference image to obtain global features; and determining the image content features of the suppressed semantic information based on the global features.

[0010] According to some embodiments of this disclosure, feature fusion of the second feature and the third feature includes: integrating the information of the second feature through cross-attention to obtain a first control condition, and integrating the information of the third feature through cross-attention to obtain a second control condition; performing convolution calculation and max pooling on the first control condition and the second control condition to obtain a third control condition; and performing feature fusion through an encoder based on the first control condition, the second control condition, and the third control condition to obtain the image-text fusion feature.

[0011] According to some embodiments of this disclosure, generating the target image based on the first feature, the second feature, and the third feature includes: determining a diffusion generation model and initial noise of the diffusion generation model, wherein the diffusion generation model is used to generate the target image; obtaining control conditions of the diffusion generation model based on the image-text fusion feature and the first feature; and denoising the initial noise based on the control conditions of the diffusion generation model and the image-text fusion feature to generate the target image.

[0012] According to some embodiments of this disclosure, the diffusion generation model includes a control model for generating control conditions and a backbone model for generating images according to the control conditions; the diffusion generation model is trained as follows: acquiring first training data and second training data, wherein the first training data includes input text training data and reference image training data, and the second training dataset includes input image training data, input text training data, and reference image training data; inputting the first training data into the backbone model of the diffusion generation model to be trained, adjusting the parameters of the diffusion generation model until the target image generated by the diffusion generation model to be trained meets the set first condition; inputting the second training data into the backbone model and control model of the diffusion generation model to be trained, adjusting the parameters of the diffusion generation model until the image generated by the diffusion generation model to be trained meets the set second condition, thereby obtaining a trained diffusion generation model.

[0013] According to some embodiments of this disclosure, an image generation apparatus is provided, comprising: an acquisition unit configured to acquire an input image, input text, and a reference image, wherein the input image is used to describe the content of a target image in image form, the input text is used to describe the content of the target image in text form, and the reference image is used to determine the style of the target image; an extraction unit configured to acquire a first feature of the input image, a second feature of the input text, and a third feature of the reference image; further configured to fuse the second feature and the third feature to obtain an image-text fusion feature; and a generation unit configured to generate the target image based on the first feature and the image-text fusion feature.

[0014] According to some embodiments of this disclosure, the extraction unit obtains the third feature of the reference image by extracting the image content feature of the reference image that suppresses semantic information as the third feature, wherein the suppressed semantic information indicates that the semantic information in the reference image does not participate in feature extraction.

[0015] According to some embodiments of this disclosure, the extraction unit extracts image content features of suppressed semantic information in the reference image in the following manner: the reference image is divided into multiple sub-blocks, the sub-blocks are randomly sorted and image features are extracted for each sub-block to obtain sub-block content features; based on the sub-block content features, the image content features of suppressed semantic information are determined.

[0016] According to some embodiments of this disclosure, the extraction unit determines the image content features of the suppressed semantic information based on the sub-block content features in the following manner: encoding each sub-block to obtain image encoding features; randomly interpolating and fusing the image encoding features with the randomly rearranged sub-block content features to obtain randomly interpolated and fused sub-block content features; and determining the image content features of the suppressed semantic information based on the randomly interpolated and fused sub-block content features.

[0017] According to some embodiments of this disclosure, the extraction unit determines the image content features of the suppressed semantic information based on the content features of the randomly interpolated and fused sub-blocks in the following manner: global feature extraction is performed on the content features of the randomly interpolated and fused sub-blocks belonging to the same reference image to obtain global features; based on the global features, the image content features of the suppressed semantic information are determined.

[0018] According to some embodiments of this disclosure, the generation unit fuses the second feature and the third feature to obtain image-text fusion features in the following manner: integrating the information of the second feature through cross-attention to obtain a first control condition, and integrating the information of the third feature through cross-attention to obtain a second control condition; performing convolution calculation and max pooling on the first control condition and the second control condition to obtain a third control condition; and performing feature fusion through an encoder based on the first control condition, the second control condition, and the third control condition to obtain the image-text fusion features.

[0019] According to some embodiments of this disclosure, the generation unit generates the target image based on the first feature and the image-text fusion feature in the following manner: determining a diffusion generation model and the initial noise of the diffusion generation model, wherein the diffusion generation model is used to generate the target image; obtaining the control conditions of the diffusion generation model based on the image-text fusion feature and the first feature; and denoising the initial noise based on the control conditions of the diffusion generation model and the image-text fusion feature to generate the target image.

[0020] According to some embodiments of this disclosure, the diffusion generation model includes a control model for generating control conditions and a backbone model for generating images according to the control conditions; the diffusion generation model is trained as follows: acquiring first training data and second training data, wherein the first training data includes input text training data and reference image training data, and the second training dataset includes input image training data, input text training data, and reference image training data; inputting the first training data into the backbone model of the diffusion generation model to be trained, adjusting the parameters of the diffusion generation model until the target image generated by the diffusion generation model to be trained meets the set first condition; inputting the second training data into the backbone model and control model of the diffusion generation model to be trained, adjusting the parameters of the diffusion generation model until the image generated by the diffusion generation model to be trained meets the set second condition, thereby obtaining a trained diffusion generation model.

[0021] According to some embodiments of the present disclosure, another image generation apparatus is provided, comprising: a processor; and a memory for storing a processor-executable computer program or instructions; wherein the processor is configured to execute the computer program or instructions to implement the steps of the image generation method described in any one of the embodiments of the present disclosure.

[0022] According to some embodiments of the present disclosure, a storage medium is provided, comprising: the storage medium storing a computer program or instructions, wherein when the computer program or instructions in the storage medium are executed by a processor of an image generating device, the image generating device is enabled to perform the image generating method described in any one of the embodiments of the present disclosure.

[0023] According to some embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image generation method described in any one of the embodiments of the present disclosure.

[0024] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: On the one hand, since both the input image and the reference image are visual data, and by fusing the features of the reference image with the features of the input text, control conditions are generated based on the features of the input image. This allows the generation of a target image with the image content features of the reference image based on the control conditions, while preserving the rich visual information in the reference image, thereby improving the accuracy of the generated target image. On the other hand, by using the reference image as input information for generating control conditions, the generated target model acquires the style features of the reference image, increasing the types of style images that can be generated.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0027] Figure 1A This is a flowchart illustrating an image generation method according to some embodiments of the present disclosure.

[0028] Figure 1B This is a schematic diagram illustrating a scenario of an image cross-bracing method according to some embodiments of the present disclosure.

[0029] Figure 2 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 2 .

[0030] Figure 3 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 3 .

[0031] Figure 4 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 4 .

[0032] Figure 5 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 5 .

[0033] Figure 6 The flowchart of an image generation method shown according to some embodiments of this disclosure Figure 6 .

[0034] Figure 7A This is flowchart seven, illustrating an image generation method according to some embodiments of the present disclosure.

[0035] Figure 7B This is a scene illustration of an image generation method according to some embodiments of the present disclosure. Figure 2 .

[0036] Figure 8 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 8 .

[0037] Figure 9 This is a block diagram of an image generation apparatus 100 shown according to some embodiments of the present disclosure.

[0038] Figure 10 This is a block diagram illustrating an image generation apparatus 200 according to some embodiments of the present disclosure.

[0039] Figure 11 This is a block diagram illustrating an image generation apparatus 300 according to some embodiments of the present disclosure. Detailed Implementation

[0040] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0041] With the development of artificial intelligence (AI) technology, various AI models are being widely used in a variety of application scenarios. For example, in image generation scenarios, images can be generated based on text-to-image generators.

[0042] In related technologies, using text-to-image generation models to generate images based on given text and reference images is a common image generation method, thereby transferring some image content features corresponding to the reference image to the generated target image. For ease of understanding, the application scenarios of this image generation method will be illustrated through the following examples:

[0043] In one example, taking the transfer of image content features (e.g., style features) of a reference image to a target image as an example, a common application scenario is as follows: the reference image is an "image in the style of ink painting", the user input text (e.g., the image generation requirement) is "generate an image with a kitten", the reference image and the input text are input into model A, and model A outputs the final target image of a kitten with the image content features (e.g., style features) of the reference image.

[0044] However, text-to-image generation models in related technologies only generate target images with partial image content features corresponding to a reference image based on the content of the given text. To ensure the target image possesses these features, the visual data of the reference image needs to be combined with the text description. Since the projection of the reference image onto the text information is unlikely to retain the inherent rich details of the reference image, the generated image will lose some details from the reference image, resulting in a blurred style. For example, continuing the previous example, when generating a "kitten image with an ink painting style" based on the user-input text description, if the stylistic features of varying shades and textures in an ink painting are lost, and only the black and white features are retained, the target image output by model A might be a black and white style kitten image, which is then used as the target image. This would make the generated image content too coarse and lack the visual style of the reference image.

[0045] Therefore, it is evident that when using image generation methods in related technologies to generate an image based on given text and a reference image, the generated target image may not necessarily possess the accurate style features of the reference image. For example, there may be cases where an image with only partial style features of the reference image is generated. In such cases, the image perceived by the naked eye may be completely different in style, thus affecting the user experience.

[0046] To address the aforementioned issues, this disclosure proposes an image generation method. This method extracts features from a reference image to obtain image content features for generating a target image. The features of the reference image are then fused with features from given text information and used together with features from the input image to generate the target image. This approach preserves rich information about the image content features in the reference image for target image generation, resulting in a more precise style for the generated target image and a more accurate representation of the user's image generation needs.

[0047] It should be noted that the image generation method provided in this disclosure can be applied to an image generation device. The image generation device may include, for example, a terminal. The terminal includes, but is not limited to, at least one of the following: mobile phone, wearable device, Internet of Things device, smart car, computer, tablet computer, virtual reality (VR) terminal device, and augmented reality (AR) terminal device.

[0048] For ease of understanding, some technical terms appearing in the embodiments of this disclosure will be explained illustratively below.

[0049] Stable Diffusion (SD) models are a class of generative models based on latent diffusion. They efficiently generate high-quality images by applying a diffusion process to a low-dimensional latent space. SD models are commonly used for tasks such as text-to-image generation, image inpainting, and artistic creation. Their core architecture consists of an encoder-decoder structure, a diffusion process, and a conditional guidance mechanism. The workflow is divided into three stages: latent space compression, diffusion denoising, and image reconstruction. By using generated control conditions, initial noise can be diffused and denoised to obtain a target image that conforms to the target distribution.

[0050] Latent space compression could involve using a variational autoencoder (VAE) to compress a high-resolution image into a low-dimensional latent space, reducing computational complexity. Diffusion denoising could involve progressively denoising the purely noisy data in the latent space to generate a clean latent representation that conforms to the target distribution. Image reconstruction could involve using the decoder in a variational autoencoder (VAE) to restore the target image from the clean latent representation, thus recovering visual details.

[0051] For the diffusion denoising process, the generation direction can be guided by conditional information from text, images, or other modalities, thus achieving controllable content generation.

[0052] Style-ControlNet is a model that introduces style-conditional inputs on top of ControlNet to generate conditional information. This generated conditional information allows for precise control over both the structure and artistic style of the generated content. It is commonly used for tasks such as multi-style image synthesis, artistic creation, and advertising design, supporting the simultaneous specification of content structure and image style through text, images, or semantic graphs.

[0053] Style-controlled diffusion generation (SD+ControlNet) is a technical solution that integrates a control network on top of a stable diffusion model. By introducing structural control signals, it achieves precise guidance on the spatial layout and geometric structure of the generated image content. Its core objective is to give users fine-grained control over image composition, object position, and shape while retaining SD's high-quality generation capabilities. It is suitable for image generation tasks that require strict adherence to preset structures.

[0054] Feature fusion is a technique that integrates feature representations from different sources or levels, aiming to improve the expressive power and performance of a model through information complementarity. Compared to feature concatenation, feature fusion places greater emphasis on modeling the relationships between features and optimizing fusion strategies.

[0055] Image style: The visual characteristics and artistic expression of an image, reflecting its visual appeal and aesthetic features. Common image styles can be represented from several perspectives, such as color style, texture style, and line style.

[0056] Color styles can include, for example, vibrant color styles (e.g., images using high saturation and strong contrast), soft color styles (e.g., images using low saturation and low contrast), black and white color styles (e.g., images using only grayscale levels), or monochrome color styles (e.g., images using only a single color).

[0057] Texture styles can include, for example, smooth texture styles (e.g., the colors and brightness in the image change gently with no obvious texture) and coarse texture styles (e.g., the image has obvious texture and graininess), etc.

[0058] As for line styles, they can include fine lines as well as bold lines.

[0059] The content of an image includes information such as the object, scene, style, etc., that the image records.

[0060] It should be noted that the content of an image may be defined with different meanings in different application scenarios. In the embodiments of this disclosure, the exemplary descriptions of relevant image content, unless otherwise specified, can be understood as including image content that may be involved in all application scenarios.

[0061] Global features: In image processing, global features refer to attributes that represent the overall characteristics of an image. These features, in contrast to local features, primarily describe the overall characteristics of an image or object, such as color, texture, shape, and style.

[0062] Channels: These typically refer to different dimensions or components of data. In the field of deep learning, especially in Convolutional Neural Networks (CNNs), feature maps also have the concept of channels. Each channel represents a set of features extracted by neurons in the previous layer through different convolutional kernels. For example, after a convolutional layer, there might be 32 channels, each representing features extracted from the input image through a different convolutional kernel.

[0063] Image encoder: This usually refers to a neural network structure that transforms image data into a compressed, numerical representation that captures the key features and attributes of the image while discarding unnecessary information.

[0064] Contrastive Language-Image Pre-training (CLIP): This model is pre-trained through large-scale contrastive learning of image and text pairs. The core idea of ​​CLIP is to transform the image recognition task into a text-image matching task, enabling the model to understand image content and establish a connection with natural language descriptions.

[0065] Text encoder: A text encoder is a tool or model used to convert text data into a numerical form that can be efficiently processed by machine learning algorithms.

[0066] Variational autoencoders are probabilistic graphical generative models that utilize neural networks to parameterize probability distributions and learn latent representations of given data. Their core idea is to assume a series of parameterized distributions for the data and optimize these parameters to make the model generate data as close as possible to real-world data.

[0067] A vector encoder is a model component that maps input data to a vector space. It uses neural networks or other mathematical methods to transform data into a fixed-length vector representation in order to capture the key features and inherent structure of the data. Vector encoders are commonly used for tasks such as data compression, feature extraction, image generation, and natural language processing.

[0068] A multi-layer perceptron (MLP) is a fundamental and important neural network structure that consists of multiple layers of neurons, including an input layer, one or more hidden layers, and an output layer.

[0069] It should be noted that the embodiments disclosed herein can be applied to the following scenarios, including but not limited to: image restoration and recovery scenarios (including, for example, in the restoration of old photos, the technology can be used to generate image content of lost or damaged parts), image editing and compositing scenarios (including, for example, extracting the main content from different images and compositing them to generate images with different backgrounds), image generation and creation scenarios (including, for example, combining different elements or objects to generate entirely new works of art or designs), personalized emoticon generation scenarios (including, for example, generating personalized emoticons that match the main content of oneself or friends based on needs), virtual makeup scenarios (including, for example, generating images of target objects after makeup), and image or video stylization (including, for example, using stylized image generation technology to transform photos into works of art, or creating more artistic visual effects in entertainment products such as video games and movies), etc.

[0070] Figure 1A This is a flowchart illustrating an image generation method according to some embodiments of the present disclosure, such as... Figure 1A As shown, the image generation method used in the terminal includes the following steps.

[0071] In step S11, an input image, input text, and a reference image are obtained. The input image is used to describe the content of the target image in image form, the input text is used to describe the content of the target image in text form, and the reference image is used to determine the style of the target image.

[0072] In step S12, the first feature of the input image, the second feature of the input text, and the third feature of the reference image are obtained.

[0073] In step S13, the second feature and the third feature are fused to obtain the image-text fusion feature.

[0074] In step S14, a target image is generated based on the first feature and the image-text fusion feature.

[0075] In this embodiment, on the one hand, since both the input image and the reference image are visual data, and by fusing the features of the reference image with the features of the input text, control conditions are generated based on the features of the input image. This allows the generation of a target image with the image content features of the reference image, while preserving rich visual information in the reference image, thereby improving the accuracy of the generated target image. On the other hand, by using the reference image as input information for the generation control conditions, the generated target model acquires the style features of the reference image, increasing the number of style types that can be generated for the target image.

[0076] It should be noted that the input text can be presented in forms such as prompts or tags. The input image can be presented in forms such as doodles, lines, sketches, outlines, hand-drawn pictures, simplified diagrams, or images obtained by extracting outlines from images.

[0077] It should be noted that a reference image can be understood, for example, as an image used to determine the style of the target image. The reference image includes style features used to generate the target image.

[0078] To facilitate understanding, the following will be explained through... Figure 1B The above embodiments are described by way of example.

[0079] Figure 1B This is a scene illustration of an image generation method according to some embodiments of the present disclosure. Figure 2 ,like Figure 1B As shown, the method includes the following steps.

[0080] The process involves acquiring an input image, input text, and a reference image. Style features are extracted from the reference image, and the extracted style features, along with the input image and input text, jointly control the image generation process to produce the target image.

[0081] Understandably, since the reference image is not a meaningless picture—that is, it may contain semantic information—and the input text also contains semantic information, the semantic information in the reference image and the semantic information in the input text may conflict during feature fusion of the second and third features, affecting content fidelity. For example, continuing the previous example, based on the user's input text description, when generating an image of a kitten with a "watercolor style," if the watercolor style conflicts with the semantic information of the "kitten" input text, it might generate a "watercolor style landscape image," meaning the semantic information of the reference image masks the semantic information of the input text. Based on this, the user might suppress the semantic information of the reference image and extract features that suppress semantic information.

[0082] Therefore, to make the generated target image more accurate, semantic suppression can be performed when extracting the third feature. For ease of understanding, the following explanation is provided: Figure 2 The following example will be used to explain.

[0083] Figure 2 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 2 ,like Figure 2 As shown, the method includes the following steps.

[0084] In step S21, the input image, input text, and reference image are acquired.

[0085] In step S22, the first feature of the input image, the second feature of the input text, and the image content feature of the reference image with suppressed semantic information are obtained as the third feature. Suppressing semantic information means that the semantic information in the reference image does not participate in feature extraction.

[0086] In step S23, the second feature and the third feature are fused to obtain the image-text fusion feature.

[0087] In step S24, a target image is generated based on the first feature and the image-text fusion feature.

[0088] In this embodiment of the disclosure, since the semantic information in the third feature is disturbed, the image-text fusion information obtained after feature fusion through the third feature and the second feature only contains the semantic information of the input text.

[0089] For example, the semantic information may be disrupted because the semantic information in the third feature cannot be aligned with the image. Therefore, when fusing the second and third features, the semantic information in the reference image will not conflict with the semantic information in the second feature. This allows for the generation of a target image based on the input text while avoiding interference from the semantic information in the style image, thus improving the accuracy of the generated target image.

[0090] It should be noted that the implementation of step 21 is similar to that of step S11. For an interpretation of the relevant content or a detailed description of the implementation method, please refer to the description of the relevant content in the above embodiments, which will not be repeated here. The implementation of steps S23-S24 is similar to that of steps S13-S14. For an interpretation of the relevant content or a detailed description of the implementation method, please refer to the description of the relevant content in the above embodiments, which will not be repeated here.

[0091] In some embodiments, an input image can be understood as a description of the image content of a target image through an image representation. Similarly, input text can be understood as a description of the image content of a target image through a textual description. The content of the target image described in image form is usually not entirely the same as the content described in text form; for example, it may describe the same target from different angles or describe different characteristics of the same target.

[0092] For example, the input image is "a doodle image containing a kitten," where the information about the kitten, including its target, pose, lines, and angle, can be understood as the content of the target image described in image form. The input text is "a kitten with gray stripes," where the semantic information of "gray," "stripes," and "kitten" can be understood as the content of the target image described in text form. Reference images could be, for example, "ink wash style images," "cartoon style images," or "realistic style images."

[0093] Understandably, the first feature can be obtained by understanding and extracting features from the input image using a Latent Convolutional Module (LCM). The second feature can be obtained by understanding and extracting features from the input text using a text encoder. Furthermore, the style features of the reference image can be obtained by understanding the content of the reference image and extracting its features using a Visual Large Language Module (V-LLM).

[0094] For ease of understanding, the following implementation methods will be described in conjunction with... Figure 2 The relevant embodiments will be described, that is, in the process of obtaining the third feature, semantic information is suppressed by randomly sorting the image or feature at each stage.

[0095] Figure 3 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 3 ,like Figure 3 As shown, the method includes the following steps.

[0096] In step S31, the input image, input text, and reference image are acquired.

[0097] In step S32, the reference image is divided into multiple sub-blocks, the sub-blocks are randomly sorted, and image features are extracted from each sub-block to obtain the sub-block content features.

[0098] In step S33, image content features that suppress semantic information are determined based on the sub-block content features.

[0099] In step S34, the image content features that suppress semantic information are used as the third feature.

[0100] For ease of understanding, an image with a resolution of H×H can be segmented into j sub-blocks. After randomly sorting the segmented sub-blocks, feature extraction is performed using a visual large language model to obtain the content features of each sub-block.

[0101] In this embodiment, by segmenting the reference image into multiple sub-blocks, the semantic information in the reference image cannot be perfectly aligned with the order of the sub-blocks, thereby suppressing semantic information. By fusing the sub-block content features with a second feature, the resulting image-text fusion feature reduces the interference of semantic information in the reference image, improving the expression of semantic information in the input text. Furthermore, this improves the accuracy of the final generated target image in conforming to style features.

[0102] Understandably, in some embodiments, users require that the input text be preserved as much as possible. Even by segmenting the reference image into multiple sub-blocks and reordering these sub-blocks before extracting features, some semantic information will still be retained. To further improve the accuracy of the generated target image, semantic information can be further suppressed by randomly sorting the content features of the sub-blocks at the feature level. Additionally, semantic information can be suppressed by inputting negative sample prompts to the model. For example, inputting negative sample prompts can instruct the model to avoid semantic information containing watermarks, signatures, poor quality, or inappropriate content in the generated image.

[0103] Figure 4 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 4 ,like Figure 4 As shown, the method includes the following steps.

[0104] In step S41, the reference image is divided into multiple sub-blocks, the sub-blocks are randomly sorted, and image features are extracted from each sub-block to obtain the sub-block content features.

[0105] In step S42, each sub-block is encoded to obtain image encoding features.

[0106] In step S43, the image coding features and the randomly rearranged sub-block content features are randomly interpolated and fused to obtain the randomly interpolated and fused sub-block content features.

[0107] In step S44, image content features that suppress semantic information are determined based on the sub-block content features obtained by random interpolation fusion.

[0108] In this embodiment of the disclosure, since the features obtained by encoding the sub-blocks are text semantic alignment features, the image encoding features obtained by encoding are randomly interpolated and fused with the previously obtained sub-block content features. This can further suppress semantic information on the basis that the sub-block content features have already suppressed semantic information, and the feature data is disturbed at the feature level. The resulting randomly interpolated and fused sub-block content features are features with invalid semantic information.

[0109] To facilitate understanding, the process of randomly interpolating and fusing the image coding features with the randomly rearranged sub-block content features in step S43 will be represented by a formula below.

[0110] For each sub-block of the reference image, feature extraction is performed using V-LLM to obtain sub-block content features fa containing certain semantic information. After random rearranging the sub-block content features, the sub-block content features are perturbed at the feature level to obtain semantically disordered sub-block content features fa′. The process of randomly interpolating and fusing the image coding features fb obtained after encoding each sub-block with the randomly rearranged sub-block content features, i.e., the semantically disordered sub-block content features fa′, can be expressed as the following mathematical expression (1):

[0111] i=Random(Conv1D(Random(fa′)+α·Random(fb)))

[0112] Mathematical expression (1)

[0113] Where α is the weight of the random interpolation fusion.

[0114] In this embodiment of the disclosure, for the sub-block content features based on random interpolation fusion, they can be input into a multi-channel one-dimensional convolution to achieve random sorting of multi-channel dimensions.

[0115] In the embodiments of this disclosure, the encoding may be implemented by inputting each sub-block into a language image pre-trained model encoder (Clip image encoder), or each sub-block may be input into other high-level encoding modules.

[0116] Understandably, in the process of suppressing the semantic information of the reference image, the reference image is segmented, causing the third feature to lose the global style features of the entire image. If the third feature, which has lost the global style features of the entire image, is used for image generation, the generated image may only have local style. To improve the global style of the final generated image, the content features of the sub-blocks fused by random interpolation can be globally modeled and output to enhance the style feature information in the third feature, weaken the semantic information, and remove redundancy.

[0117] Figure 5 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 5 See Figure 5 The method includes the following steps.

[0118] In step S51, the reference image is divided into multiple sub-blocks, the sub-blocks are randomly sorted, and image features are extracted from each sub-block to obtain the sub-block content features.

[0119] In step S52, each sub-block is encoded to obtain image encoding features.

[0120] In step S53, the image coding features and the randomly rearranged sub-block content features are randomly interpolated and fused to obtain the randomly interpolated and fused sub-block content features.

[0121] In step S54, global features are extracted from the content features of sub-blocks that are randomly interpolated and fused from the same reference image to obtain global features.

[0122] In step S55, image content features that suppress semantic information are determined based on global features.

[0123] In this embodiment, by extracting global features from the content features of randomly interpolated and fused sub-blocks belonging to the same reference image, the global features lost due to semantic information suppression are recovered. These global features contain the global style information of the reference image. Through these global features, the third feature can retain the style features of the reference image while suppressing semantic information. By guiding image generation with these third features, the generated target image can accurately represent the features of the reference image.

[0124] It should be noted that for global feature extraction, the sub-block content features of the same reference image can be obtained by randomly interpolating and fusing them into a vector encoder. The vector encoder includes convolutional layers and a multilayer perceptron, where the convolutional layers are used to acquire global features, and the multilayer perceptron is used to vectorize the features processed by the convolutional layers.

[0125] To facilitate understanding, the obtained global features will be explained below using specific numerical values. For each image, an H-dimensional vector representation i can be obtained. If there are k input reference images, then the k reference images are processed in parallel to obtain the third feature K, K = {i0, i1, ..., ik}.

[0126] Understandably, even with semantic information suppression of the third feature of the reference image, feature conflicts may still occur when directly fusing the second and third features, as it cannot be guaranteed that the third feature does not contain semantic information. To further reduce the potential feature conflicts that may arise when fusing the second and third features, the second and third features can be fused in parallel. The resulting fused feature is then passed on and modeled using an independent transform encoder module to further reduce feature conflicts between the second and third features.

[0127] Figure 6 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 6 See Figure 6The method includes the following steps.

[0128] In step S61, an input image, input text, and a reference image are obtained. The input image is used to describe the content of the target image in image form, the input text is used to describe the content of the target image in text form, and the reference image is used to determine the style of the target image.

[0129] In step S62, the first feature of the input image, the second feature of the input text, and the third feature of the reference image are obtained.

[0130] In step S63, the information of the second feature is integrated through cross-attention to obtain the first control condition, and the information of the third feature is integrated through cross-attention to obtain the second control condition.

[0131] In step S64, convolution calculation and max pooling are performed on the first and second control conditions to obtain the third control condition.

[0132] In step S65, based on the first control condition, the second control condition and the third control condition, feature fusion is performed by the encoder to obtain image-text fusion features.

[0133] In this embodiment of the disclosure, by integrating the information of the second feature and the information of the third feature in parallel, the problem of feature accumulation loss caused by superimposing the features of text description and image in a series manner is avoided. Furthermore, fusion modeling can be performed through an independent transformer encoder, which reduces feature conflicts, improves the semantic information representation of the image-text fusion features, and preserves style features.

[0134] It should be noted that the cross-attention approach integrating the second feature can be implemented by inputting the second feature into the row cross-attention module. The cross-attention approach integrating the third feature can be implemented by inputting the third feature into the image cross-attention module. It is worth noting that since the original SD already has good text feature representation capabilities, to reduce training costs, the parameters of the row cross-attention module in SD can be retained and not used in model training. The image cross-attention module, however, needs to be trained specifically for the image generation task.

[0135] In this embodiment of the disclosure, the key and value of the text cross-attention module are derived from the second feature, and the key and value of the style cross-attention module are derived from the third feature.

[0136] To facilitate understanding, the method for obtaining image-text fusion features will be explained below using formulas.

[0137] The formula for obtaining the image-text fusion features can be expressed as the following mathematical expression (2):

[0138] Style-Attention = Trans en (fea t +fea i +Max pooling (Conv 2d (fea t fea i )))

[0139] Mathematical expression (2)

[0140] Among them, fea t As the first control condition, fea i As the second control condition, Max pooling For max pooling, Conv 2d This indicates convolution processing.

[0141] It is understandable that the obtained image-text fusion features can retain rich visual information from the reference image. However, since some visual information is lost when projected onto the text information, directly using the image-text fusion information to guide the generation of the target image may result in insufficient accuracy of the generated target image, thus affecting the user experience. The generation of the target image can be guided by obtaining control conditions for generating the target image based on the image-text fusion features and the first feature.

[0142] Figure 7A This is a flowchart seven illustrating an image generation method according to some embodiments of the present disclosure, such as... Figure 7A As shown, the method includes the following steps.

[0143] In step S71, the input image, input text, and reference image are acquired.

[0144] In step S72, the first feature of the input image, the second feature of the input text, and the third feature of the reference image are obtained.

[0145] In step S73, the second feature and the third feature are fused to obtain the image-text fusion feature.

[0146] In step S74, the diffusion generation model and the initial noise of the diffusion generation model are determined.

[0147] In step S75, the control conditions of the diffusion generation model are obtained based on the image-text fusion features and the first feature.

[0148] In step S76, based on the control conditions of the diffusion generation model and the image-text fusion features, the initial noise is denoised to generate the target image.

[0149] In this embodiment, since the image-text fusion feature retains both the style information of the reference image and the semantic information of the input text, and the input image contains the content of the target image, and both the first feature and the image-text fusion feature are visual data, the control conditions generated based on the first feature and the image-text fusion feature can guide the diffusion generation model to generate a target image that conforms to the text information description and has the style of the reference image. Furthermore, since the reference image is used as the basis for the generation control conditions, rather than training the diffusion generation model with the corresponding reference image, it is possible to generate new style images without needing to fine-tune or retrain the model for each new style, thereby increasing the types of style images that can be generated.

[0150] To facilitate understanding, the following will be explained through... Figure 7B The above embodiments are described by way of example.

[0151] Figure 7B This is a scene illustration of an image generation method according to some embodiments of the present disclosure. Figure 2 ,like Figure 7B As shown, the method includes the following steps.

[0152] Step S11-1: Obtain input information.

[0153] Step S12-1: Obtain the first feature and the second feature.

[0154] Step S12-2: Obtain the third feature.

[0155] Step S13-1, Feature fusion.

[0156] Step S14-1: Obtain the control conditions for the diffusion generation model.

[0157] Step S14-2: Generate the target image.

[0158] Step S11-1 can be understood, for example, as acquiring an input image and input text used to describe the content of the target image.

[0159] Step S12-1 can be understood as extracting features from the input image using a latent convolutional module to obtain the first feature. Then, a text encoder extracts features from the input text to obtain the second feature. The latent convolutional module can be, for example, a module comprising four 3D convolutional layers and multiple dropout regularization layers. Using a latent convolutional module reduces the number of model parameters, thereby reducing computational cost.

[0160] Step S12-2 can be understood as follows: Each reference image in a batch is segmented into multiple sub-blocks, and these sub-blocks are randomly ordered. Sub-block content features are extracted using a visual large language model. Each sub-block is encoded using a pre-trained language image encoder to obtain image-encoded features. These image-encoded features are then randomly interpolated and fused with the randomly rearranged sub-block content features to obtain randomly interpolated and fused sub-block content features. The randomly interpolated and fused sub-block content features belonging to the same reference image are then used to extract global features using a vector encoder to obtain global features. For a given batch of K reference images, a third feature K can be obtained, K = {i0, i1, ..., ik}.

[0161] Step S13-1 can be understood as follows: The second feature is processed by the text cross-attention module to obtain the first control condition. The third feature is processed by the style cross-attention module to obtain the second control condition. The first and second control conditions are passed through a convolutional layer and a max-pooling layer to obtain the third control condition. Based on the first, second, and third control conditions, feature fusion is performed by an encoder to obtain the image-text fused features.

[0162] Regarding step S14-1, it should be noted that the diffusion generation model in this embodiment can be based on SD+ControlNet, replacing the cross-attention layer in SD+ControlNet with a style-attention layer used to execute steps S11-1 to S13-1. SD+ControlNet includes a backbone model and a control model. The backbone model can be understood as reducing initial noise to obtain an image conforming to the target distribution based on the control conditions of the diffusion generation model generated by the control model. The backbone model can be a U-Net structure, which includes a multi-level encoder and a decoder corresponding to each encoder level. Each encoder level includes a self-attention layer and a style-attention layer in sequence. The order of the self-attention layer and the style-attention layer in the decoder is reversed compared to that in the encoder. The control model adopts the same structure as the backbone model, and the output of the self-attention layer of the decoder in the control model is used as the input of the self-attention layer of the decoder in the backbone model. For the style-attention layer in the backbone model and the control model, a style-attention layer with the same structure but different parameter settings is used. Using the first feature as input to the control model, and since the style attention layer can provide image-text fusion features, the control conditions for the diffusion generation model are obtained.

[0163] For step S14-2, it should be noted that by using initialization noise as input to the backbone model, providing image-text fusion features through the style attention layer, and receiving the control conditions of the diffusion generation model through the self-attention layer of the backbone model, a target image that conforms to the user description and has the style of the reference image can be generated.

[0164] It is understood that the diffusion generation model in this embodiment includes two stages: control condition generation and image generation, corresponding to the control model and the backbone model, respectively. If the entire model is trained simultaneously, there may be a situation where the diffusion generation model can be used to generate images of a fixed style, but needs to be retrained when generating images of a new style. Therefore, it is necessary to train the model in two stages.

[0165] To facilitate understanding, the following will be explained... Figure 8 The description illustrates the training process of the diffusion generative model in this disclosure.

[0166] Figure 8 This is a flowchart of an image generation method shown according to some embodiments of the present disclosure. Figure 8 ,like Figure 8 As shown, the method includes the following steps.

[0167] In step S81, first training data and second training data are obtained, wherein the first training data includes input text training data and reference image training data, and the second training dataset includes input image training data, input text training data and reference image training data.

[0168] In step S82, the first training data is input into the backbone model of the diffusion generation model to be trained, and the parameters of the diffusion generation model are adjusted until the target image generated by the diffusion generation model to be trained meets the set first condition.

[0169] In step S83, the second training data is input into the backbone model and control model of the diffusion generation model to be trained, and the parameters of the diffusion generation model are adjusted until the images generated by the diffusion generation model to be trained meet the set second condition, thus obtaining the trained diffusion generation model.

[0170] In step S84, the target image is generated based on the first feature and the image-text fusion feature using the trained diffusion generation model.

[0171] In this embodiment of the disclosure, since the first training stage is trained using the first training data for generating text-image style features, the model is able to learn the ability to generate target images that conform to the text description. Furthermore, since the second training stage adds the input image to the training data, the diffusion generation model is able to learn feature representations that adapt to mainstream styles. Thus, the trained diffusion generation model can accurately generate target images with multiple mainstream style features.

[0172] It should be noted that the first training phase can use reference image-text pairs for training, while the second phase can use aligned stylized image-text pairs for training. During the second phase of training, the parameters of the SD model and CLIP can be frozen. Optimization can be performed using the Adamw optimizer with a learning rate of 0.001, a batch size of 32, and input images adjusted to H×H resolution. During training, style reference images are synthesized by augmenting the images in the training data through operations such as random cropping, resizing, horizontal flipping, and rotation. These augmentations help prevent shortcut learning and provide doodle image information, such as brushstroke line art, to improve the control model's adaptability to Style-Attention. The sampling step T = 32 is set for inference, and the CFG is set to 7.5.

[0173] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0174] Based on the same concept, embodiments of this disclosure also provide an image generation apparatus.

[0175] It is understood that the image generation apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0176] Figure 9 This is a block diagram of an image generation apparatus 100 according to some embodiments of the present disclosure. (Refer to...) Figure 9 The device includes an acquisition unit 101, an extraction unit 102, and a generation unit 103.

[0177] The acquisition unit 101 is configured to acquire an input image, input text, and a reference image, wherein the input image is used to describe the content of the target image in image form, the input text is used to describe the content of the target image in text form, and the reference image is used to determine the style of the target image.

[0178] The extraction unit 102 is configured to acquire a first feature of the input image, a second feature of the input text, and a third feature of the reference image; it is also configured to fuse the second and third features to obtain image-text fusion features.

[0179] The generation unit 103 is configured to generate a target image based on a first feature and an image-text fusion feature.

[0180] In some embodiments, the extraction unit 102 obtains the third feature of the reference image by extracting the image content feature of the reference image that suppresses semantic information as the third feature, wherein suppressing semantic information indicates that the semantic information in the reference image does not participate in feature extraction.

[0181] In some embodiments, the extraction unit 102 extracts image content features that suppress semantic information in the reference image in the following manner: the reference image is divided into multiple sub-blocks, the sub-blocks are randomly sorted and image features are extracted for each sub-block to obtain sub-block content features; based on the sub-block content features, the image content features that suppress semantic information are determined.

[0182] In some embodiments, the extraction unit 102 determines the image content features that suppress semantic information based on the sub-block content features in the following manner: encoding each sub-block to obtain image encoding features; randomly interpolating and fusing the image encoding features with the randomly rearranged sub-block content features to obtain randomly interpolated and fused sub-block content features; and determining the image content features that suppress semantic information based on the randomly interpolated and fused sub-block content features.

[0183] In some embodiments, the extraction unit 102 determines the image content features that suppress semantic information based on the content features of randomly interpolated and fused sub-blocks in the following manner: global feature extraction is performed on the content features of randomly interpolated and fused sub-blocks belonging to the same reference image to obtain global features; based on the global features, the image content features that suppress semantic information are determined.

[0184] In some embodiments, the generation unit 103 fuses the second feature and the third feature to obtain image-text fusion features in the following manner: integrating the information of the second feature through cross-attention to obtain a first control condition, and integrating the information of the third feature through cross-attention to obtain a second control condition; performing convolution calculation and max pooling on the first control condition and the second control condition to obtain a third control condition; and performing feature fusion through an encoder based on the first control condition, the second control condition, and the third control condition to obtain image-text fusion features.

[0185] In some embodiments, the generation unit 103 generates a target image based on a first feature and an image-text fusion feature in the following manner: determining a diffusion generation model and initial noise of the diffusion generation model, wherein the diffusion generation model is used to generate the target image; obtaining control conditions of the diffusion generation model based on the image-text fusion feature and the first feature; and denoising the initial noise based on the control conditions of the diffusion generation model and the image-text fusion feature to generate the target image.

[0186] In some embodiments, the diffusion generation model includes a control model for generating control conditions and a backbone model for generating images according to the control conditions. The diffusion generation model is trained as follows: first training data and second training data are acquired, wherein the first training data includes input text training data and reference image training data, and the second training dataset includes input image training data, input text training data, and reference image training data; the first training data is input into the backbone model of the diffusion generation model to be trained, and the parameters of the diffusion generation model are adjusted until the target image generated by the diffusion generation model to be trained meets the set first condition; the second training data is input into the backbone model and control model of the diffusion generation model to be trained, and the parameters of the diffusion generation model are adjusted until the image generated by the diffusion generation model to be trained meets the set second condition, thereby obtaining the trained diffusion generation model.

[0187] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0188] Figure 10 This is a block diagram illustrating an image generation apparatus 200 according to some embodiments of the present disclosure. For example, apparatus 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0189] Reference Figure 10The device 200 may include one or more of the following components: processing component 202, memory 204, power supply component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0190] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0191] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0192] Power supply component 206 provides power to various components of device 200. Power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 200.

[0193] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0194] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0195] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0196] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0197] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 216 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0198] In some embodiments of this disclosure, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0199] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by the processor 220 of the device 200 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0200] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0201] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of the image generating apparatus, the image generating apparatus is able to perform the methods described above.

[0202] Figure 11 This is a block diagram illustrating an apparatus 300 for image generation according to some embodiments disclosed in a publication. For example, apparatus 300 may be provided as a server. See also... Figure 11 The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the methods described above.

[0203] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0204] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory 332 including instructions, which can be executed by the processing component 322 of the device 300 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0205] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0206] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of the image generating apparatus, the image generating apparatus is able to perform the methods described above.

[0207] In some embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the image generation method involved in any of the above embodiments.

[0208] In one exemplary embodiment, the processor executing the computer program may be deployed in an electronic device, for example.

[0209] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0210] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, a first component, part, region, layer, or section mentioned in the examples may also be referred to as a second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature.

[0211] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0212] In this description, "multiple" means at least two, referring to two or more, such as two, three, etc., unless otherwise explicitly specified. Other quantifiers are similar. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, unless otherwise specified or clearly indicated from the context, the articles "a" and "an" as used in this application and the appended claims are generally understood to mean "one or more."

[0213] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term "and / or" includes any one of the related listed items and any combination of two or more; "and / or" describes the association relationship between related objects, indicating that three relationships may exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Similarly, "at least one of..." includes any one of the related listed items and any combination of two or more.

[0214] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0215] Furthermore, the term "exemplary" is used herein to indicate that it serves as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term "exemplary" is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to indicate an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to indicate any of the natural inclusive permutations. That is, if X applies A; X applies B; or X applies both A and B, then applying A or B satisfies the condition under any of the foregoing instances.

[0216] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if it is not structurally equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in this disclosure, such terms are intended to be inclusive in a manner similar to the term “including.”

[0217] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0218] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image generation method, characterized in that, include: The system acquires an input image, input text, and a reference image, wherein the input image is used to describe the content of the target image in image form, the input text is used to describe the content of the target image in text form, and the reference image is used to determine the style of the target image. Obtain the first feature of the input image, the second feature of the input text, and the third feature of the reference image; The second and third features are fused to obtain the image-text fusion feature; The target image is generated based on the first feature and the image-text fusion feature.

2. The method according to claim 1, characterized in that, The third feature of the reference image is obtained in the following manner: The image content features with suppressed semantic information in the reference image are extracted as the third feature. The suppressed semantic information indicates that the semantic information in the reference image does not participate in feature extraction.

3. The method according to claim 2, characterized in that, The step of extracting image content features with suppressed semantic information from the reference image includes: The reference image is divided into multiple sub-blocks, the sub-blocks are randomly sorted, and image features are extracted from each sub-block to obtain the sub-block content features; Based on the sub-block content features, the image content features for suppressing semantic information are determined.

4. The method according to claim 3, characterized in that, Based on the sub-block content features, the image content features for suppressing semantic information are determined, including: Each of the sub-blocks is encoded to obtain image coding features; The image coding features and the randomly rearranged sub-block content features are randomly interpolated and fused to obtain the randomly interpolated and fused sub-block content features; Based on the sub-block content features obtained by the random interpolation fusion, the image content features for suppressing semantic information are determined.

5. The method according to claim 4, characterized in that, Based on the sub-block content features obtained from the random interpolation fusion, the image content features for suppressing semantic information are determined, including: Global features are obtained by extracting global features from the content features of the sub-blocks that belong to the same reference image through random interpolation fusion. Based on the global features, the image content features for suppressing semantic information are determined.

6. The method according to claim 1, characterized in that, The second and third features are fused, including: By integrating information from the second feature through cross-attention, a first control condition is obtained; and by integrating information from the third feature through cross-attention, a second control condition is obtained. The first and second control conditions are subjected to convolution and max pooling to obtain the third control condition. Based on the first control condition, the second control condition, and the third control condition, feature fusion is performed by an encoder to obtain the image-text fusion feature.

7. The method according to claim 1, characterized in that, The step of generating the target image based on the first feature and the image-text fusion feature includes: Determine the diffusion generation model and the initial noise of the diffusion generation model, wherein the diffusion generation model is used to generate the target image; The control conditions for the diffusion generation model are obtained based on the image-text fusion features and the first feature; Based on the control conditions of the diffusion generation model and the image-text fusion features, the initial noise is denoised to generate the target image.

8. The method according to claim 7, characterized in that, The diffusion generation model includes a control model for generating control conditions and a backbone model for generating images based on the control conditions. The diffusion generation model is trained as follows: Acquire first training data and second training data, wherein the first training data includes input text training data and reference image training data, and the second training dataset includes input image training data, input text training data, and reference image training data; The first training data is input into the backbone model of the diffusion generation model to be trained, and the parameters of the diffusion generation model are adjusted until the target image generated by the diffusion generation model to be trained meets the set first condition. The second training data is input into the backbone model and control model of the diffusion generation model to be trained, and the parameters of the diffusion generation model are adjusted until the images generated by the diffusion generation model to be trained meet the set second condition, thus obtaining the trained diffusion generation model.

9. An image generation apparatus, characterized in that, include: The acquisition unit is configured to acquire an input image, input text, and a reference image, wherein the input image is used to describe the content of the target image in image form, the input text is used to describe the content of the target image in text form, and the reference image is used to determine the style of the target image; The extraction unit is configured to acquire a first feature of the input image, a second feature of the input text, and a third feature of the reference image; it is also configured to fuse the second feature and the third feature to obtain image-text fusion features. The generation unit is configured to generate the target image based on the first feature and the image-text fusion feature.

10. The apparatus according to claim 9, characterized in that, The extraction unit obtains the third feature of the reference image in the following manner: The image content features with suppressed semantic information in the reference image are extracted as the third feature. The suppressed semantic information indicates that the semantic information in the reference image does not participate in feature extraction.

11. The apparatus according to claim 10, characterized in that, The extraction unit extracts image content features with suppressed semantic information from the reference image in the following manner: The reference image is divided into multiple sub-blocks, the sub-blocks are randomly sorted, and image features are extracted from each sub-block to obtain the sub-block content features; Based on the sub-block content features, the image content features for suppressing semantic information are determined.

12. The apparatus according to claim 11, characterized in that, The extraction unit determines the image content features that suppress semantic information based on the sub-block content features in the following manner: Each of the sub-blocks is encoded to obtain image coding features; The image coding features and the randomly rearranged sub-block content features are randomly interpolated and fused to obtain the randomly interpolated and fused sub-block content features; Based on the sub-block content features obtained by the random interpolation fusion, the image content features for suppressing semantic information are determined.

13. The apparatus according to claim 12, characterized in that, The extraction unit determines the image content features that suppress semantic information based on the sub-block content features obtained by random interpolation fusion in the following manner: Global features are obtained by extracting global features from the content features of the sub-blocks that belong to the same reference image through random interpolation fusion. Based on the global features, the image content features for suppressing semantic information are determined.

14. The apparatus according to claim 9, characterized in that, The generation unit fuses the second feature and the third feature to obtain the image-text fusion feature in the following manner: By integrating information from the second feature through cross-attention, a first control condition is obtained; and by integrating information from the third feature through cross-attention, a second control condition is obtained. The first and second control conditions are subjected to convolution and max pooling to obtain the third control condition. Based on the first control condition, the second control condition, and the third control condition, feature fusion is performed by an encoder to obtain the image-text fusion feature.

15. The apparatus according to claim 9, characterized in that, The generation unit generates the target image based on the first feature and the image-text fusion feature in the following manner: Determine the diffusion generation model and the initial noise of the diffusion generation model, wherein the diffusion generation model is used to generate the target image; The control conditions for the diffusion generation model are obtained based on the image-text fusion features and the first feature; Based on the control conditions of the diffusion generation model and the image-text fusion features, the initial noise is denoised to generate the target image.

16. The apparatus according to claim 15, characterized in that, The diffusion generation model includes a control model for generating control conditions and a backbone model for generating images based on the control conditions. The diffusion generation model is trained as follows: Acquire first training data and second training data, wherein the first training data includes input text training data and reference image training data, and the second training dataset includes input image training data, input text training data, and reference image training data; The first training data is input into the backbone model of the diffusion generation model to be trained, and the parameters of the diffusion generation model are adjusted until the target image generated by the diffusion generation model to be trained meets the set first condition. The second training data is input into the backbone model and control model of the diffusion generation model to be trained, and the parameters of the diffusion generation model are adjusted until the images generated by the diffusion generation model to be trained meet the set second condition, thus obtaining the trained diffusion generation model.

17. An image generation device, characterized in that, include: processor; Memory used to store computer programs or instructions that can be executed by a processor; The processor is configured to execute the computer program or instructions to implement the steps of the image generation method according to any one of claims 1 to 8.

18. A storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by the processor of the image generating device, enable the image generating device to perform the image generating method according to any one of claims 1 to 8.

19. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the image generation method as described in any one of claims 1 to 8.

Images