Method and apparatus for image generation, and device and storage medium

Abstract

On the basis of the embodiments of the present disclosure, provided are a method and apparatus for image generation, and a device and a storage medium. The method comprises: on the basis of an input image comprising a target object, acquiring a group of intermediate image features for representing the target object. The method further comprises: on the basis of the group of intermediate image features, determining a target image feature for representing the target object. The method further comprises: on the basis of input text and the group of intermediate image features, determining a target text feature for representing an image generation target. The method further comprises: on the basis of the target image feature and the target text feature, generating an output image associated with the input image and the input text. In this way, high-quality image generation of image generation targets meeting various expectations can be advantageously implemented.

Description

Method, apparatus, device and storage medium for image generation

[0001] This application claims priority to the Chinese invention patent application entitled “Method, device, apparatus and storage medium for image generation” and application number 2023111616694, filed on September 8, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for image generation. Background Art

[0003] In the field of computer vision (CV), various image generation technologies based on machine learning have been significantly developed and have a wide range of applications. For example, in many application scenarios such as social networking, gaming, and image editing, it is expected to generate and use images that include target objects (e.g., target people). Image generation technologies based on machine learning can be used in such application scenarios to improve the user experience. In some example application scenarios, it is expected to generate images that match the user input based on the user's input information, such as text description information.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, a method for image generation is provided. The method includes obtaining a set of intermediate image features for characterizing the target object based on an input image including the target object. The method also includes determining target image features for characterizing the target object based on the set of intermediate image features. The method also includes determining target text features for characterizing a target for image generation based on input text and the set of intermediate image features. The method also includes generating an output image associated with the input image and the input text based on the target image features and the target text features.

[0006] In a second aspect of the present disclosure, a device for image generation is provided. The device includes: an intermediate feature acquisition module configured to acquire a set of intermediate image features for characterizing a target object based on an input image including the target object; an image feature determination module configured to determine target image features for characterizing the target object based on the set of intermediate image features; a text feature determination module configured to determine target text features for characterizing an image generation target based on input text and the set of intermediate image features; and an image generation module configured to generate an output image associated with the input image and input text based on the target image features and the target text features.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0009] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] FIG2 shows a schematic block diagram of an example architecture for image generation according to some embodiments of the present disclosure;

[0013] FIG3 shows another schematic block diagram of an example architecture for image generation according to some embodiments of the present disclosure;

[0014] FIG4 shows a schematic diagram of a diffusion model according to some embodiments of the present disclosure;

[0015] FIG5 shows a schematic diagram of an attention unit in a diffusion model according to some embodiments of the present disclosure;

[0016] FIG6 shows a flowchart of a process for image generation according to some embodiments of the present disclosure;

[0017] FIG7 shows a block diagram of an apparatus for image generation according to some embodiments of the present disclosure; and

[0018] FIG8 shows a block diagram of a device capable of implementing various embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0024] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0025] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0026] Herein, unless explicitly stated otherwise, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0027] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0028] As used herein, the term "machine learning model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, it can generate corresponding outputs for given inputs. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. In this article, a "machine learning model" may also be referred to as a "machine learning network" or "network", and these terms are used interchangeably in this article.

[0029] As used herein, the term "based on a set of intermediate image features" may refer to based on part of the intermediate image features in the set of intermediate image features (for example, one intermediate image feature or multiple intermediate image features in the set of intermediate image features) or based on all the intermediate image features in the set of intermediate image features.

[0030] Generally speaking, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the use stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and iterated continuously until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the inference stage, the model can be used to process the actual input based on the parameter values ​​obtained through training to determine the corresponding output.

[0031] As mentioned above, in the field of computer vision (CV), various image generation techniques based on machine learning have been significantly developed and widely used. In some example applications, it is desirable to use machine learning models to generate images that match the user input, such as text description information, based on the user's input.

[0032] Several image generation schemes based on text-to-image models have been proposed. These schemes use natural language prompts, such as text describing a scene, and employ trained text-to-image models to generate realistic images corresponding to the described scene. These text-to-image models are typically trained using large text-to-image databases. However, sometimes users want to integrate themselves, their family, or their friends into new scenes. These new scenes do not exist in the dataset used to train the model. In other words, the dataset used to train the model does not contain text and images corresponding to the new scenes. Therefore, it becomes very difficult to generate realistic images that match these new scenes using only text descriptions.

[0033] To achieve customized text-to-image generation for different scenarios, particularly customized facial image generation, a scheme based on image information preservation has been proposed. In this scheme, an input image of a target object, such as a person image or a facial image, is tokenized in the text space to obtain a unique token for the input image in the text space. Obtaining the tokens of the input image is also referred to as inversely transforming the input image into pseudowords. This input image tokenization process is typically a topic-based optimization process. This scheme employs a text-to-image model to generate an output image based on both the tokens of the input image and the text describing the scene entered by the user. This scheme uses the tokens of the input image during image generation, thus preserving the image information of the input image in the output image, i.e., it exhibits image preservation. Furthermore, this scheme can generate images corresponding to different scenarios based on a variety of different text inputs, thus being editable.

[0034] However, this approach typically requires significant computing and storage resources and a long processing time. This extended processing time and fine-tuning process make it difficult to meet users' needs for quickly acquiring desired images. Furthermore, due to the limited number of people in the training dataset, the model is trained only on images of people with specific appearances, leading to overfitting. This overfitting problem limits the editing capabilities of this approach.

[0035] To address these issues, several encoder-based approaches have been proposed to optimize customized text-to-image generation by optimizing pseudo-label embeddings, introducing sampling steps, or incorporating novel concept injection modules. However, these approaches struggle to achieve satisfactory results. Furthermore, these approaches require multiple input images and require fine-tuning during testing to ensure image preservation and editability.

[0036] In summary, these aforementioned solutions suffer from various drawbacks, including high cost, long generation time, lack of character similarity, and limited editing capabilities. Therefore, according to an embodiment of the present disclosure, a solution for generating image feature representations is provided, aiming to address one or more of the aforementioned and other potential issues.

[0037] In this solution, a set of intermediate image features representing a target object (e.g., a target person) is obtained from an input image including the target object. Based on the set of intermediate image features, target image features for representing the target object are determined. In addition, based on the user's input text and the set of intermediate image features, target text features for representing an image generation target are determined. For example, the image generation target may indicate a desired scene or a desired image style, etc. Further, based on the determined target image features and target text features, an output image associated with the input image and the input text is generated. For example, the output image may be an image of the target person that matches the scene or image style indicated by the image generation target.

[0038] In this way, an output image that meets the user's expectations can be generated based on both the input image and the input text entered by the user. Specifically, in determining the target text features used to characterize the image generation target, not only the user's input text is considered, but also intermediate image features related to the input image are taken into account. This allows the target text features to reflect relevant information about both the input image and the input text. The output image generated using these target text features combined with the target image features can better match the user's desired image generation target.

[0039] Some example embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0040] Sample Environment

[0041] FIG1 shows a block diagram of an example environment 100 in which multiple implementations of the present disclosure can be implemented. In the environment of FIG1 , an electronic device 110 is configured to generate an output image 112 based on input text 102 and an input image 104. The input image 104 includes a target object, such as a target person, animal, plant, or other static and dynamic objects, and may even include composite objects, etc. The target object may also be at least a part of the above-mentioned object, such as the face, half body, or whole body of the target person, or a part or all of other objects. In some embodiments below, a person or a person's face will be used as an example of a target object, but it should be understood that this is merely exemplary and not intended to be limiting. The embodiments described with reference to “person” or “person's face” are applicable to other types of objects.

[0042] The input text 102 may be descriptive information about the user's desired output image 104. For example, the input text 102 may describe the scene, environment, state, etc. of the desired output image 104. In another example, the input text 102 may also describe the style, etc. of the desired output image 104. It is desired that, based on the input text 102 and the input image 104, an output image 104 be generated that includes a target object (e.g., the head, half-body, or full-body of a target person) and matches the scene, environment, state, style, etc. described in the input text 102.

[0043] As an example, the input image 104 may be an image including a target person (represented by the placeholder S* in the examples herein), such as a photo of the person, a video screenshot, or the like. If the user desires to obtain a cartoon-style image of the person, an input text 102 similar to “S*in cartoon style” may be input to the electronic device 110. For another example, if the user desires to obtain an image of the person wearing a sci-fi spacesuit in a space environment, the input text 102 may be similar to “S*wears a sci-fi spacesuit in space”. In this way, by inputting different input texts 102, output images 112 of different scenes, environments, states, and styles can be generated for the target objects included in different input images 104. Several examples of generating output images 112 based on input text 102 and input image 104 will be described below in conjunction with Figures 2 to 5.

[0044] Although the input text 102 is shown as English in the examples herein, it should be understood that this is merely illustrative and is not intended to limit the scope of protection of the present disclosure. The embodiments of the present disclosure are applicable to texts in any language.

[0045] In some embodiments, the electronic device 110 may be a terminal device of a user. The electronic device 110 may include any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Alternatively, in some embodiments, the electronic device 110 may also be an external device or server that communicates remotely with the user's terminal device, including but not limited to a mainframe, an edge computing node, a computing device in a cloud environment, and the like. The electronic device 110 may be any electronic device with computing capabilities, including servers, blade servers, mainframes, edge devices, and other appropriate computing devices.

[0046] It should be understood that the components and arrangements of the environment 100 shown in FIG. 1 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements.

[0047] Overview of Image Generation

[0048] FIG2 shows a schematic block diagram of an architecture 200 for image generation according to some embodiments of the present disclosure. The architecture 200 of FIG2 can be implemented, for example, at the electronic device 110 of FIG1 . The architecture 200 includes at least a feature extraction subsystem 210 for extracting image features and text features. The architecture 200 further includes an image generation subsystem 220 for generating an output image 112 based on the features extracted by the feature extraction subsystem 210. The various components and models in the feature extraction subsystem 210 and the image generation subsystem 220 can be implemented by hardware, software, firmware, or any combination thereof.

[0049] 2 , feature extraction subsystem 210 determines target text features 212 and target image features 224 based on input text 102 and input image 104. Input image 104 includes target object 204. For example, input image 104 may include the face, half body, or full body of a target person.

[0050] Specifically, the feature extraction subsystem 210 obtains a set of intermediate image features 214 based on the input image 104 for characterizing the target object 204. For example, the feature extraction subsystem 210 may determine a region of interest (ROI) in the input image 104 based on the location of the target object 204 in the input image 104. The interesting portion of the target object 204, such as the face, is located within the ROI. The feature extraction subsystem 210 may perform feature extraction on the portion of the input image 104 located within the ROI to obtain the intermediate image features 214.

[0051] The feature extraction subsystem 210 determines target image features 224 for characterizing the target object based on a set of intermediate image features 214. As an example, the intermediate image features 214 and the target image features 224 can be determined by a model such as an image encoder or an image feature mapper. For example, different models can be used to determine the intermediate image features 214 and the target image features 224, respectively. For another example, a single model can be used to determine both the intermediate image features 214 and the target image features 224, wherein the intermediate image features 214 can be output by a hidden layer of the model, and the target image features 224 can be output by an output layer of the model.

[0052] The feature extraction subsystem 210 determines target text features 212 for characterizing an image generation target based on the input text 102 and a set of intermediate image features 214. For example, the feature extraction subsystem 210 may determine a text feature representation of the set of intermediate image features 214 and a text feature representation of at least a portion of the input text 102. The feature extraction subsystem 210 then generates target text features 212 based on the determined text feature representations. As an example, the text feature representations may be any suitable type of vectorized representation. Similarly, the intermediate image features 214, the target image features 224, and the target text features 212 may all be any suitable type of vectorized representation.

[0053] 2 , a target image feature 224 and a target text feature 212 may be determined based on a set of intermediate image features 214. In some embodiments, the target image feature 224 may be determined based on one intermediate image feature in the set of intermediate image features 214, and the target text feature 212 may be determined based on the remaining intermediate image features other than the intermediate image feature, as will be described below with reference to FIG. 3 .

[0054] After determining the target text features 212 and the target image features 224, the image generation subsystem 220 may generate the output image 112 based on the target text features 212 and the target image features 224. For example, the image generation subsystem 220 may use a trained machine learning model to generate the output image 112. As an example, the used machine learning model may include a diffusion model.

[0055] In this way, an output image 112 that meets the user's expectations can be generated based on both the user's input image 104 and input text 102. Specifically, in determining the target text features 212 used to characterize the image generation target, not only the user's input text 102 is considered, but also a set of intermediate image features 214 related to the input image 104 is considered. In this way, the target text features 212 can reflect relevant information about both the input image 104 and the input text 102. The output image generated using such target text features 212 in combination with the target image features 224 can better match the user's desired image generation target.

[0056] Furthermore, in the embodiments of the present disclosure, fine-tuning during testing can be avoided, and image generation can be achieved by only requiring the user to input a single image. In this way, fast and easy image generation can be achieved, for example, an image can be generated in just a few seconds.

[0057] Example architecture for image generation

[0058] An example architecture for image generation is described above in conjunction with FIG2 . Next, a more detailed example architecture 300 for image generation will be described in conjunction with FIG3 . Architecture 300 may be implemented, for example, at electronic device 110 of FIG1 . Similar to architecture 200 , architecture 300 includes a feature extraction subsystem 210 for extracting image features and text features, and an image generation subsystem 220 for generating images. The various components and models in architecture 300 may be implemented in hardware, software, firmware, or any combination thereof.

[0059] As shown in FIG3 , an input image 104 is input to a feature extraction subsystem 210 . The input image 104 includes a target object, such as a target person or a portion of a target person. In some embodiments, the feature extraction subsystem 210 may pre-process the input image 104 to obtain a processed image 304 . For example, the feature extraction subsystem 210 may determine a region of interest (ROI) in the input image 104 based on the location of the target object in the input image 104 . The ROI may be the region where the target object is located. Alternatively, the ROI may be the region where an interesting portion of the target object is located.

[0060] In the example of FIG. 3 , the region of interest is the area where the target person's face is located. In embodiments where the region of interest is the target person's face, the feature extraction subsystem 210 may employ face detection technology to identify and locate the face in the input image 104. For example, face detection may be performed based on one or more predetermined facial feature points. This ensures that subsequent feature extraction is focused on the region of interest.

[0061] Additionally or alternatively, in some embodiments, after determining the region of interest, the scope of the region of interest can be appropriately adjusted. For example, the scope of the region of interest including the facial area can be expanded according to a predetermined scale parameter. The predetermined scale parameter can be, for example, 1.3 or other values. By adjusting the scope of the region of interest, the region of interest can include images such as hair and part of the human body. In this way, it can be ensured that the characteristics of the target object, such as details of facial features (e.g., expression, hairstyle, hair color, etc.), are all included in the region of interest, thereby avoiding feature omission.

[0062] In some embodiments, the feature extraction subsystem 210 can extract a region of interest from the input image 104. By removing portions other than the target object from the extracted region of interest, a processed image 304 is obtained. For example, the feature extraction subsystem 210 can remove non-facial elements from the region of interest. For example, a mask can be applied to the region of interest to remove non-facial elements such as background and accessories. This ensures the accuracy and efficiency of subsequent feature extraction.

[0063] In some embodiments, the feature extraction subsystem 210 further scales the processed image 304 to obtain a processed image 304 of a desired size. For example, the desired size may be an image size required by a model subsequently used to generate the output image 112. The desired size may be preset or adjustable by a user or the feature extraction subsystem 210. The embodiments of the present disclosure are not limited in this respect.

[0064] After obtaining the processed image 304, the feature extraction subsystem 210 can generate a set of intermediate image features 214 based on the processed image 304. As an example, the feature extraction subsystem 210 can generate a set of intermediate image features 214 based on the processed image 304 using a predetermined number of network layers in the image encoder 310. The output of each of these predetermined number of network layers can be determined as an intermediate image feature. For example, the output features of the output layer in the image encoder 310 and the output features of m (m is an integer greater than or equal to 1) hidden layers close to the output layer can be selected as a set of intermediate image features 214. In this way, the representation ability of image feature representation (such as image labeling) and the editability of the model can be enhanced, and these image features capture spatial and semantic information at different levels of abstraction and concreteness.

[0065] The image encoder 310 may be a comparative language-image pre-trained (CLIP) image encoder or other types of image encoders. The image encoder 310 may be the same type as the image encoder used in the subsequent image generation subsystem 220. By using the CLIP image encoder, it is not necessary to train it, thereby simplifying the feature extraction process.

[0066] Additionally, in some embodiments, the feature extraction subsystem 210 may also obtain output image features 314 from the output layer of the image encoder 310. The feature extraction subsystem 210 may then utilize an image feature mapper 330 to convert the output image features 314 into target image features 224. For example, the image feature mapper 330 may be implemented using a multilayer perceptron, also known as an image adapter. The target image features 224 may also be referred to as target image labels.

[0067] In some embodiments, the image encoder 310 does not need to be trained. The image feature mapper 330 can be trained. By combining the image encoder 310 and the image feature mapper 330, the feature extraction process can be simplified.

[0068] Alternatively, in some embodiments, the output image features 314 of the image encoder 310 may be determined as the target image features 224. That is, in such embodiments, the image feature mapper 330 may be omitted. The scope of the present disclosure is not limited in this respect.

[0069] In some embodiments, the performance of feature extraction subsystem 210 is affected by the capabilities of the subsequent text encoder 340. Semantic information in the text space is often abstract, which places higher demands on text feature representation capabilities (i.e., tag representation capabilities). Therefore, by incorporating intermediate image features 214 in the image space as auxiliary features into the model, this makes the model's understanding of new concepts more concrete and helps effectively learn pseudo-tags in the text space.

[0070] Specifically, feature extraction subsystem 210 can determine target text features 212 based on a set of intermediate image features 214 and at least a portion of input text 102. In the example of FIG3 , input text 102 is "A photo of S*." It should be understood that this input text is merely exemplary and not limiting. Input text 102 can include any appropriate content.

[0071] In some embodiments, the feature extraction subsystem 210 can use a text feature mapper 320 (also known as a text perception machine) to convert a set of intermediate image features 214 into a first set of text feature representations 322. For example, a portion of the intermediate image features or all of the intermediate image features in the set of intermediate image features 214 can be converted into the first set of text feature representations 322. In this document, the first set of text feature representations 322 is also referred to as pseudo-label embeddings or pseudo-label embedding vectors. As an example only and not intended to be limiting, the text feature mapper 320 can convert the intermediate image features of the m hidden layers from the image encoder 310 into word embeddings to obtain the first set of text feature representations 322. The first set of text feature representations 322 can be represented as S * =S1,...,S m , that is, it can be represented as a pseudo-word “S*”, as shown in Figure 3. In this way, image features that are difficult to be properly described by existing words can be embedded into the text space, thereby obtaining a feature description of the image information of the target object (e.g., face).

[0072] In some embodiments, the text feature mapper 320 can be a model composed of two or more multi-layer perceptron (MLP) layers with nonlinear activation functions. The text feature mapper 320 can be trained or fine-tuned. Each MLP layer can convert the intermediate image features of a network layer from the image encoder 310 into a text feature representation, such as a word embedding. By adopting a text feature mapper 320 with an MLP layer, the text feature mapper 320 can be made lightweight and can quickly obtain a text feature representation.

[0073] The feature extraction subsystem 210 may determine a second set of text feature representations 324 based on at least a portion of the input text 102. For example, the feature extraction subsystem 210 may determine a first subtext representing a target object in the input text 102 and a second subtext other than the first subtext in the input text 102. In the example where the input text 102 is “A photo of S*”, the first subtext is “S*” and the second subtext is “A photo of”.

[0074] The feature extraction subsystem 210 can then generate target text features 212 based on the first set of text feature representations 322 and the second set of text feature representations 324. The target text features 212 can also be referred to as target text tokens. As an example, the feature extraction subsystem 210 can generate the target text features 212 based on the first set of text feature representations 322 and the second set of text feature representations 324 using a text encoder 340.

[0075] In some embodiments, the text encoder 340 can be a pre-trained model and can be used directly without training. As an example, the text encoder 340 can be a CLIP text encoder. In some embodiments, by obtaining the intermediate image features 214 by the CLIP image encoder and generating the target text features 212 by the CLIP text encoder, the image features can be converted to the text space more efficiently and accurately.

[0076] By obtaining a set of intermediate image features 214 using the image encoder 310 and mapping these features using the text feature mapper 320, the target text features 212 generated thereby can capture important image clues or visual clues related to the image of the target object. Utilizing these image clues related to the image of the target object, a desired output image 112 can be subsequently generated.

[0077] In some embodiments, the image feature mapper 330 and the text feature mapper 320 may use the same structural design so that the target image feature 224 and the target text feature 212 use the same or similar vectorized representation, which can facilitate subsequent image generation.

[0078] The above describes the feature extraction process of the feature extraction subsystem 210. As mentioned above, some modules in the feature extraction subsystem 210, such as the image encoder 310 and the text encoder 340, do not need to be trained or fine-tuned. This simplifies the training of the architecture 300.

[0079] Next, the image generation subsystem 220 can generate the output image 112 based on the extracted features. In some embodiments, the image generation subsystem 220 can generate the output image 112 from the initial noisy image using the diffusion model 350, conditioned on the target image features 224 and the target text features 212. Herein, the target image features 224 are also referred to as image conditions of the diffusion model 350, and the target text features 212 are also referred to as text conditions of the diffusion model 350.

[0080] In some embodiments, the initial noisy image may include a white noise image. Using a white noise image can improve the editability and richness of the output image 112. Additionally or alternatively, in some embodiments, the initial noisy image may include the input image 104 to which noise has been applied, where the applied noise is above a threshold noise level. The threshold noise level may be pre-set or dynamically set by a user or the image generation subsystem 220. By using the input image to which noise has been applied, the output image 112 can be made to more closely resemble or maintain the pose, perspective, etc., of the target object in the input image 104.

[0081] As will be described in detail below, the process of generating the output image 112 using the diffusion model 350 can include multiple denoising steps, for example, a total of T denoising steps, where T is an integer greater than or equal to 1. These denoising steps are performed in descending order, for example, first performing step T, then step T-1, then step T-2, and so on until step 1. The earlier the denoising step is performed, the earlier the noise level is addressed. Therefore, in this embodiment, the input image can be introduced early in the denoising process to ensure that the noise level applied to the input image is high, for example, above a threshold. In other words, the input image can be introduced in a denoising step close to step T.

[0082] In some embodiments, the target image features 224 and the target text features 212 may be input (also referred to as injected) into the diffusion model 350 at different ratios. Alternatively, the target image features 224 and the target text features 212 may be input into the diffusion model 350 as a whole at the same ratio. The ratio of the target image features 224 and the target text features 212 input into the diffusion model 350 may be pre-set, dynamically adjusted, or adjusted based on user input. The embodiments of the present disclosure are not limited in this respect.

[0083] In some embodiments, the diffusion model 350, as well as the image feature mapper 330 and the text feature mapper 340, can be trained based on the loss value 360. The loss value 360 ​​can include a loss associated with the target object and a loss associated with image reconstruction by the diffusion model 350. A more detailed example of the diffusion model 350 generating the output image 112 will be further described below in conjunction with Figures 4 and 5.

[0084] According to the above description with reference to FIG3 , in the embodiment of the present disclosure, the user does not need to input multiple images, which can advantageously achieve fast and efficient image generation.

[0085] Example Diffusion Model

[0086] The diffusion model, also known as the Denoising Diffusion Probabilistic Model, is a type of generative model. Common generative models (e.g., Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs)) primarily work by giving a "restricted" input (e.g., type, style, etc.) and gradually adding information to ultimately generate a result (e.g., an image, speech, etc.). Unlike common generative models, the diffusion model gradually "samples" a specific distribution from noise information (e.g., Gaussian noise) according to certain conditions, ultimately generating a result as the number of "sampling" rounds increases. In other words, the generation process of the diffusion model involves extracting the desired data from the noise through multiple iterations, and the quality of the generated data improves as the number of iterations increases. The input of the diffusion model includes Gaussian noise, and the output is the generated data.

[0087] In general, the diffusion model involves a forward diffusion process that continuously adds noise and a reverse diffusion process that continuously removes noise. The forward diffusion process can be represented as a Markov forward process. The desired data generation process corresponds to the reverse process, for example, by continuously reversing the Markov forward process to obtain the desired data. In this paper, the diffusion model is used to generate images.

[0088] In some embodiments, the diffusion model can use an attention mechanism to generate images. In some embodiments, the diffusion model can include a text encoder (denoted as c θ ), and the text encoder can be used to map the input text y to the intermediate representation c θ(y). This intermediate representation is then adopted in one or more intermediate layers of the denoising sub-network through an attention mechanism. As an example, the attention mechanism can be a cross-attention mechanism. Equation (1) shows an example of an attention mechanism adopted in an intermediate layer of the denoising sub-network:

[0089] Where Attn(Q, K, V) represents the value of the attention obtained, Represents query features, K = W K c θ (y) represents the key characteristic, V = W V c θ (y) represents the value feature, z t represents the noise latent representation at the t-th step (t is an integer greater than or equal to 1 and less than or equal to T) (i.e., the noise latent representation in the t-th iterative denoising step), represents the hidden state in the denoising subnetwork at step t, T is the total time or the total number of iterative denoising steps, d′ represents the scaling factor used for the attention mechanism, and Softmax() represents the normalized exponential function.

[0090] In this paper, the diffusion model using the denoising subnetwork structure is also called the latent diffusion model. The training goal of the latent diffusion model is to predict the noise added to the latent representation of the image. This process can be expressed as Equation (2):

[0091] in, represents the denoising network, ε represents the unscaled noise sample, ε(x) represents the use of encoder ε to map the image x from the pixel space to the latent space, and c θ (y) Input text y using text encoder c θ The intermediate representation to which it is mapped, ε θ (z t ,t,c θ (y)) represents the time t corresponding to z t and intermediate representation c θ (y) noise, L D represents the loss value in the diffusion process of the diffusion model (ie, the loss value of the diffusion model for image reconstruction). In some schemes, an image can be generated by optimizing the loss value of the diffusion model in the diffusion process.

[0092] The above describes a general example of a diffusion model using an attention mechanism to generate an image. In an embodiment of the present disclosure, the diffusion model 350 generates an output image 112 from an initial noisy image using an attention mechanism, conditional on the target image features 224 and the target text features 212.

[0093] FIG4 shows an example architecture of a diffusion model 350 for image generation according to an example embodiment of the present disclosure. It should be understood that the architecture of the diffusion model shown in FIG4 is only exemplary. As shown, the diffusion model 350 includes an autoencoder, which can be represented as where ε represents the encoder, and Denotes a decoder. The diffusion model 350 includes a denoising (also known as noise reduction) network (denoted as ).

[0094] In some embodiments, the autoencoder It is trained on a large-scale image dataset to compress images. Specifically, the encoder ε is used to map the image x from the pixel space to a low-dimensional latent space (also called hidden space), that is, to obtain the latent representation z = ε(x). The decoder is used to reconstruct the latent representation z back to an image with high fidelity in order to achieve

[0095] In some embodiments, the denoising network Multiple denoising sub-networks may be included, for example, denoising sub-network 450, denoising sub-network 455, and so on. Each denoising sub-network may be used for a denoising step. For example, a denoising sub-network architecture may be used to implement the denoising sub-network. It is trained to perform the back diffusion process in the latent space. Using the denoising network to perform the back diffusion process can reduce the consumption of computing resources and storage resources. As shown in Figure 4, the conditions (in the embodiment of the present disclosure, the target image features 224 and the target text features 212) are injected into the denoising network.

[0096] As previously described, the diffusion model 350 is used to generate the output image 112 from the initial noisy image 352 using the attention mechanism, conditioned on the target image features 224 and the target text features 212. In some embodiments, generating the output image 112 from the initial noisy image includes multiple denoising steps. Each denoising step uses a denoising sub-network. For example, assume that the multiple denoising steps include T iterative denoising steps. In the example of Figure 4, in the Tth denoising step, the denoising sub-network 450 is used, and in the T-1th denoising step, the denoising sub-network 455 is used. Each denoising sub-network may include one or more attention units, such as the attention unit 460.

[0097] In a given denoising step t in the plurality of denoising steps, by performing the denoising on the image feature 352z to be denoised in the given denoising step t tThe attention mechanism is applied to the target image features 224 to determine the visual attention features. If the given denoising step t is the first step of the denoising process, the image features to be denoised 352 are the features of the initial noisy image, otherwise the image features to be denoised 352 may be the output of the previous denoising step t+1. t The attention mechanism is applied to the target text feature 212 to determine the text attention feature. Based on the visual attention feature and the text attention feature, the output image feature 353z of the given denoising step t is generated. t-1 , as the input of the next denoising step t-1 of a given denoising step.

[0098] Example Attention Unit

[0099] In a given denoising step t, an attention unit 500 may be used to determine visual attention features and textual attention features, such as for implementing the respective attention units shown in FIG4 . The attention unit 500 may utilize a first conversion unit to convert the target text feature 212 into a first feature (also denoted as K T ), which can be used as a key in the attention mechanism. The attention unit 500 can use the second conversion unit to convert the target text feature 212 into a second feature (also denoted as V T ), which can be used as a value in the attention mechanism. The attention unit 500 can use the third conversion unit to convert the image feature to be denoised into a third feature (also represented as Q), which can be used as a query in the attention mechanism. The attention unit 500 can then determine the text attention feature based on the first feature, the second feature, and the third feature. In some embodiments, at least one conversion unit among the first conversion unit, the second conversion unit, and the third conversion unit includes a first subunit obtained by pre-training and a second subunit obtained by fine-tuning.

[0100] Similarly, the attention unit 500 can use, for example, the fourth conversion unit to convert the denoised image features into a fourth feature (also referred to as Q), which can be used as a query in the attention mechanism. The attention unit 500 can use the fifth conversion unit to convert the target image features 224 into a fifth feature (also referred to as K S ), which can be used as a key in the attention mechanism. The attention unit 500 can use the sixth conversion unit to convert the target image feature 224 into a sixth feature (also referred to as V S ), which can be used as a value in the attention mechanism. The attention unit 500 can then determine a visual attention feature based on the fourth feature, the fifth feature, and the sixth feature. In some embodiments, at least one of the fifth conversion unit and the sixth conversion unit can be obtained by fine-tuning.

[0101] FIG5 shows a schematic diagram of an attention unit 500. As shown, the attention unit 500 includes a text attention sub-unit 510 and a visual attention sub-unit 520. In some embodiments, a hidden state 530 may be input to at least a portion of the text attention sub-unit 510 and the visual attention sub-unit 520. Depending on the position of the attention unit 500 in the denoising sub-network, the hidden state 530 may be the image features to be denoised in the corresponding denoising step or the output of the previous attention unit.

[0102] In some embodiments, a fine-tunable solution can be adopted for the text attention unit 510. For example, the text attention unit 510 can include a base part and a fine-tunable part. For example, the model parameters of the base part can be determined through pre-training. Subsequently, the model parameters of the fine-tunable part can be determined during the fine-tuning phase. During the inference process, the attention mechanism can be applied based on the model parameters obtained through pre-training and the model parameters obtained through fine-tuning.

[0103] In the example of FIG5 , the fine-tunable subunits may be employed by the text attention subunit 510 to calculate the text attention features. Specifically, the text attention subunit 510 may utilize the first conversion unit (including subunits 540 and 545) to employ K T =W k T p +αΔW k T p To calculate the first feature used as the key, where α represents the weight and p represents the target text feature 212. Similarly, the text attention sub-unit 510 adopts the second conversion unit (including sub-unit 550 and sub-unit 555) to adopt V T =W v T p +αΔW v T p To calculate the second feature used as the value, where α represents the weight and p represents the target text feature 212. The text attention sub-unit 510 can also use the third conversion unit (including sub-unit 560 and sub-unit 565) to calculate the third feature used as the query heat. Based on the first feature, the second feature and the third feature, the text attention feature Attn(Q, K T , V T ). For example, ΔW in the above calculation k T and ΔW v TDenote the low-rank decomposition ΔW = BA, where A and B include fine-tunable parameters. In the example shown in Figure 5, subunits 540, 550, and 560 can be obtained through fine-tuning, while subunits 545, 555, and 565 can be obtained through pre-training. This approach can reduce the amount of training and improve training efficiency.

[0104] In some embodiments, the visual attention subunit 520 determines the visual attention feature by applying an attention mechanism to the denoised image feature and the target image feature 224 in a given denoising step. As an example, the visual attention subunit 520 can use the fourth conversion unit 570 to determine the fourth feature used as the query in The hidden state of the denoising sub-network in the denoising step t is shown in Table 1. The visual attention sub-unit 520 can use the fifth conversion unit 580 to adopt K S =W k S f determines the fifth feature used as a key, and uses the sixth conversion unit 590 to adopt V S =W v S f determines the sixth feature used as the value, where f represents the acquired target image feature 224. Based on the fourth, fifth and sixth features, the visual attention feature Attn (Q, K S , V S In some embodiments, the fifth conversion unit 580 and the sixth conversion unit 590 may be obtained through fine-tuning, and the fourth conversion unit 570 may be obtained through pre-training.

[0105] In some embodiments, based on the visual attention feature Attn(Q, K S , V S ) and text attention features Attn(Q, K T , V T ), generate the output image features of a given denoising step as the input of the next denoising step of the given denoising step. As an example, the text attention features Attn(Q, K) determined by the text attention subunit 510 can be T , V T ) and the visual attention feature Attn(Q, K S , V S ) are weighted summed to generate the output image features for a given denoising step.

[0106] In some embodiments, the diffusion model 350 may obtain a first weight for the image feature and a second weight for the text feature. The visual attention feature and the text attention feature may be weighted using the first weight and the second weight, respectively. For example, the following formula may be used to determine: T , V T )+σAttn(Q,K S , V S ) (3) Where O represents the feature output by the current attention unit, Attn(Q, K T , V T ) represents the text attention feature, Attn(Q, K S , V S ) represents the visual attention feature, γ and σ represent the second weight and the first weight, respectively (also known as a scaling factor). During inference, the values ​​of γ and σ can be fixed, dynamic, or determined based on user input. The relative magnitudes of γ and σ can indicate whether the generated output image is more consistent with the input text or the input image.

[0107] In some embodiments, during the training of the diffusion model, the value of the first weight and the value of the second weight are determined based on random numbers. For example, a random seed may be introduced during the process of determining the output image features. The random seed may be uniformly distributed. In the example of introducing random seeds, the output image features can be obtained as follows:

[0108] Where r1 and r2 are thresholds of random seeds, which can be pre-set or dynamically configured, for example. Using a random approach can prevent overfitting. In this way, a random fusion strategy can be adopted for each multimodal branch of the attention unit 500 to determine the output image features.

[0109] Additionally or alternatively, in some embodiments, during the reasoning process, the first weight and the second weight may be pre-set or set by the user. For example, the user may set different weights according to needs. In an example where the user expects to obtain an output image 112 that is more consistent with the input text 102, the second weight may be set to be larger, while the first weight may be set to be smaller. For another example, in an example where the user expects to obtain an output image 112 that is more consistent with the input text 102, the first weight may be set to be larger, while the second weight may be set to be smaller. The user may also adjust the first weight and the second weight based on other factors. The embodiments of the present disclosure are not limited in this respect.

[0110] Several examples of denoising steps of the diffusion model 350 are described above. In some embodiments, during the inference process of the diffusion model 350, random noise, such as random Gaussian noise z T is sampled, and through a series of T iterative denoising steps, z T is converted to z′0. Then, using the decoder To generate the output image 112, that is, to generate the output image

[0111] In some embodiments, the diffusion model 350 determines the output image 112 by optimizing a loss (eg, minimizing a loss value).

[0112] In some embodiments, the first set of text feature representations 322 (also referred to as pseudo-label embeddings p) obtained by the text feature mapper 320 is f ) and the target image feature V S Specifically, the following formulas (5) and (6) can be used to regularize p f and V S Regularization:

[0113] Where Mean() represents the mean value function, and They represent the regularized p f and V S .

[0114] In some embodiments, a loss value associated with the target object is determined. For example, a loss value associated with the face or image of the target object (also known as face image preservation loss) is determined. As an example, the loss value associated with the target object can be determined using the following formula:

[0115] Wherein, f() represents a domain-specific feature extractor, x represents a reference image, and x' is a denoised image corresponding to a text input 102 such as "A photo of S*". represents the similarity function. As an example, It can be used to calculate the cosine similarity between features extracted from the face area in the image.

[0116] Taking into account the above loss value associated with the target object, the total training loss of the diffusion model 350 can be obtained as follows:

[0117] in represents the total loss value, represents the loss value of the diffusion model 350 during the diffusion process, λ face Represents the weight, λ rt and λ rv are hyperparameters that control the sparsity level for the corresponding features. In some examples, It can be derived from the above-listed formula (2). The diffusion model 350 can generate the output image 112 by optimizing the loss, for example, minimizing the loss value.

[0118] By combining the loss value associated with the target object with the loss value associated with the image reconstruction of the diffusion model, the generated output image can include more accurate image features of the target object. In this way, the generated image can be made to better meet the user's expectations.

[0119] Example Training

[0120] Several examples of generating output image 112 using diffusion model 350 are described above in conjunction with FIG. 4 and FIG. 5 . In some embodiments, diffusion model 350 can be trained in a forward diffusion process. For example, diffusion model 350 can be continuously fine-tuned during the image generation process, such as in diffusion process 440.

[0121] In some embodiments, in the training step (or fine-tuning step) of the diffusion model 350, an input sample image and an input sample text including a sample object are obtained. Based on the input sample image and the input sample text, the diffusion model 350 is used to generate an output sample image associated with the input sample image and the input sample text. A first object feature representing the sample object is extracted from the input sample image. A second object feature representing the sample object is extracted from the output sample image. In some embodiments, the diffusion model 350 can be updated based on the difference between the first object feature and the second object feature. As an example, the second subunit obtained by fine-tuning in the attention unit 500, such as the subunits 540, 550, and 560 in Figure 5, can be updated based on the difference between the first object feature and the second object feature. In addition, the fifth conversion unit 580 and the sixth conversion unit 590 can also be updated based on the difference between the first object feature and the second object feature.

[0122] In this way, the diffusion model 350 can be continuously fine-tuned and updated, and the sub-units or conversion units in the attention unit can be updated to obtain a more accurate output image 112. This can improve the efficiency and accuracy of image generation, thereby further improving the user experience.

[0123] Example application scenario

[0124] The above describes several examples of generating output images according to the image generation scheme of the present disclosure in combination with Figures 2 to 5. In order to more clearly understand the embodiments of the present disclosure, an example application scenario is described below. The electronic device 110 obtains an input image, such as a static image input by a user, or a frame in a video. Alternatively, the electronic device 110 can also obtain an input image captured by an image acquisition device, for example, the electronic device 110 can obtain an input image taken by the user using a camera. The input image includes a target object, such as the face of a target person or the face of the user himself. In addition, the electronic device 110 obtains input text. Exemplarily, the input text can be input by the user, or acquired by a text acquisition device. The input text can describe the scene, environment, state or style of the desired output image, etc.

[0125] Then, through the example process described above with reference to FIG. 2 or FIG. 3 , an output image of the target object in the scene, environment, or state described by the input text can be generated, or an output image of the target object in the style described by the input text can be generated.

[0126] It should be understood that if the morphology (e.g., posture, expression, hairstyle, etc.) of the target object in different input images is different, the morphology of the target object in the generated output images may also be different. By adopting this solution, a variety of output images can be generated based on different input images and different input texts.

[0127] The image generation scheme disclosed herein is based not only on the image features of the input image, but also on text features associated with both the input text and the input image, thereby being able to meet the various different scenarios, environments, states, styles, and so on described by the input text. In addition, because in some embodiments, the text features are based on the features of the region of interest corresponding to the target object, such as a face, in the input image, the generated image can retain features of the target object's face, such as expression, posture, hairstyle, hair color, and so on, thereby obtaining a satisfactory output image. Such an image generation scheme can meet the different personalized needs of users, thereby improving the user experience.

[0128] Additionally, in some embodiments, the user can also set or adjust parameters in the image generation process, such as weights or other parameters used for textual attention features and visual attention features. In this way, the obtained output image can be made to better meet the user's expectations.

[0129] Furthermore, in some embodiments utilizing diffusion models and attention mechanisms, units or subunits within the model can be continuously optimized, thereby continuously improving the efficiency and accuracy of the model. The disclosed solution enables fast, efficient, and accurate image generation.

[0130] Example Process

[0131] FIG6 shows a flow chart of a process 600 for image generation according to some embodiments of the present disclosure. The process 600 may be implemented at the electronic device 110 of FIG1.

[0132] In block 610 , the electronic device 110 obtains a set of intermediate image features for characterizing a target object based on an input image including the target object.

[0133] In some embodiments, the electronic device 110 may determine a region of interest in the input image based on the location of the target object in the input image; extract the region of interest from the input image; remove portions of the non-target object from the extracted region of interest to obtain a processed image; and generate a set of intermediate image features based on the processed image. For example, the electronic device 110 may generate the set of intermediate image features based on the processed image using a predetermined number of network layers in an image encoder.

[0134] At block 620, electronic device 110 determines target image features for representing the target object based on the set of intermediate image features. For example, in an embodiment where a predetermined number of network layers in an image encoder are used to generate a set of intermediate image features, electronic device 110 may obtain output image features from an output layer of the image encoder. Electronic device 110 may then utilize an image feature mapper to convert the output image features into target image features.

[0135] At block 630, electronic device 110 determines target text features for representing an image generation target based on the input text and the set of intermediate image features. In some embodiments, electronic device 110 may utilize a text feature mapper to convert the set of intermediate image features into a first set of text feature representations; determine a second set of text feature representations based on at least a portion of the input text; and generate target text features based on the first set of text feature representations and the second set of text feature representations.

[0136] In some embodiments, the electronic device 110 may determine a first subtext representing a target object in the input text and a second subtext other than the first subtext in the input text, and may further determine a second set of text feature representations based on the second subtext.

[0137] At block 640, the electronic device 110 generates an output image associated with the input image and the input text based on the target image features and the target text features. In some embodiments, the electronic device 110 may generate the output image from the initial noisy image using a diffusion model, conditioned on the target image features and the target text features.

[0138] In some embodiments, the initial noisy image comprises at least one of the following: a white noise image, or the input image with noise applied thereto, the applied noise being higher than a threshold noise level.

[0139] In some embodiments, generating an output image from an initial noisy image includes multiple denoising steps. A given denoising step in the multiple denoising steps includes determining visual attention features by applying an attention mechanism to features of the image to be denoised and features of a target image in the given denoising step. The given denoising step also includes determining textual attention features by applying an attention mechanism to features of the image to be denoised and features of a target text. The given denoising step also includes generating output image features of the given denoising step based on the visual attention features and the textual attention features as input to a denoising step next to the given denoising step.

[0140] In some embodiments, the text attention feature can be determined by the following steps: using a first conversion unit to convert the target text feature into a first feature; using a second conversion unit to convert the target text feature into a second feature; using a third conversion unit to convert the image feature to be denoised into a third feature; and determining the text attention feature based on the first feature, the second feature, and the third feature. In some embodiments, at least one of the first conversion unit, the second conversion unit, and the third conversion unit includes a first subunit obtained through pre-training and a second subunit obtained through fine-tuning.

[0141] In some embodiments, the electronic device 110 may obtain an output image feature by weighting the visual attention feature and the text attention feature using a first weight and a second weight, respectively. In some embodiments, during the training process of the diffusion model, the value of the first weight and the value of the second weight are determined based on random numbers.

[0142] In some embodiments, the training steps of the diffusion model include: obtaining an input sample image and input sample text including a sample object; based on the input sample image and the input sample text, using the diffusion model, generating an output sample image associated with the input sample image and the input sample text; extracting a first object feature representing the sample object from the input sample image; extracting a second object feature representing the sample object from the output sample image; and updating the second subunit based on the difference between the first object feature and the second object feature.

[0143] Example devices and equipment

[0144] 7 shows a schematic structural block diagram of an apparatus 700 for image generation according to certain embodiments of the present disclosure. Apparatus 700 may be implemented as or included in electronic device 110. Each module / component in apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0145] As shown in the figure, the device 700 includes an intermediate feature acquisition module 710, which is configured to acquire a set of intermediate image features for characterizing the target object based on an input image including the target object. The device 700 also includes an image feature determination module 720, which is configured to determine target image features for characterizing the target object based on the set of intermediate image features. The device 700 also includes a text feature determination module 730, which is configured to determine target text features for characterizing the image generation target based on the input text and the set of intermediate image features. The device 700 also includes an image generation module 740, which is configured to generate an output image associated with the input image and the input text based on the target image features and the target text features.

[0146] In some embodiments, the intermediate feature acquisition module 710 is further configured to determine a region of interest in the input image based on the location of the target object in the input image; extract the region of interest from the input image; obtain a processed image by removing portions of the non-target object from the extracted region of interest; and generate a set of intermediate image features based on the processed image. In some embodiments, the intermediate feature acquisition module 710 can be further configured to generate the set of intermediate image features based on the processed image using a predetermined number of network layers in the image encoder.

[0147] In some embodiments, the image feature determination module 720 is further configured to obtain output image features of an output layer of the image encoder; and convert the output image features into target image features using an image feature mapper.

[0148] In some embodiments, the text feature determination module 730 can be further configured to utilize a text feature mapper to convert a set of intermediate image features into a first set of text feature representations; determine a second set of text feature representations based on at least a portion of the input text; and generate target text features based on the first set of text feature representations and the second set of text feature representations.

[0149] In some embodiments, the text feature determination module 730 can be further configured to determine a first subtext representing the target object in the input text and a second subtext in the input text other than the first subtext; and determine a second set of text feature representations based on the second subtext.

[0150] In some embodiments, the image generation module 740 is further configured to generate an output image from the initial noisy image using a diffusion model, conditioned on the target image features and the target text features. In some embodiments, the initial noisy image includes at least one of the following: a white noise image, or the input image with noise applied, the applied noise being greater than a threshold noise level.

[0151] In some embodiments, generating an output image from an initial noisy image includes multiple denoising steps. A given denoising step in the multiple denoising steps includes determining visual attention features by applying an attention mechanism to features of the image to be denoised and features of a target image in the given denoising step. The given denoising step also includes determining textual attention features by applying an attention mechanism to features of the image to be denoised and features of a target text. The given denoising step also includes generating output image features of the given denoising step based on the visual attention features and the textual attention features as input to a denoising step next to the given denoising step.

[0152] In some embodiments, the apparatus 700 may further include a text attention feature determination module configured to: convert the target text feature into a first feature using a first conversion unit; convert the target text feature into a second feature using a second conversion unit; convert the to-be-denoised image feature into a third feature using a third conversion unit; and determine the text attention feature based on the first feature, the second feature, and the third feature. In some embodiments, at least one of the first conversion unit, the second conversion unit, and the third conversion unit includes a first subunit obtained through pre-training and a second subunit obtained through fine-tuning.

[0153] In some embodiments, the apparatus 700 may further include an output image feature module configured to obtain an output image feature by weighting the visual attention feature and the text attention feature using a first weight and a second weight, respectively. In some embodiments, during the training of the diffusion model, the value of the first weight and the value of the second weight are determined based on random numbers.

[0154] In some embodiments, the device 700 may also include a diffusion model training module, which is configured to obtain an input sample image and an input sample text including a sample object; based on the input sample image and the input sample text, using a diffusion model, generate an output sample image associated with the input sample image and the input sample text; extract a first object feature representing the sample object from the input sample image; extract a second object feature representing the sample object from the output sample image; and update the second subunit based on the difference between the first object feature and the second object feature.

[0155] FIG8 shows a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 800 shown in FIG8 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 800 shown in FIG8 can be used to implement the electronic device 110 of FIG1 .

[0156] As shown in FIG8 , electronic device 800 is a general-purpose electronic device. Components of electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. Processing unit 810 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 820. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 800.

[0157] The electronic device 800 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 830 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 800.

[0158] The electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG8 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0159] The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0160] The input device 850 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 860 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 800 may also communicate with one or more external devices (not shown) via the communication unit 840 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 800, or with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0161] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0162] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0163] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0164] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0165] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0166] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating an image, comprising: Based on an input image including a target object, obtaining a set of intermediate image features for characterizing the target object; Determining target image features for characterizing the target object based on the set of intermediate image features; Determining target text features for characterizing an image generation target based on the input text and the set of intermediate image features; as well as Based on the target image feature and the target text feature, an output image associated with the input image and the input text is generated.

2. The method according to claim 1, wherein determining the target text feature comprises: Using a text feature mapper, converting the set of intermediate image features into a first set of text feature representations; Determining a second set of text feature representations based on at least a portion of the input text; as well as The target text features are generated based on the first set of text feature representations and the second set of text feature representations.

3. The method of claim 2, wherein determining the second set of text feature representations comprises: Determine a first subtext in the input text representing the target object and a second subtext in the input text except the first subtext; as well as Based on the second subtext, the second set of text feature representations is determined.

4. The method according to claim 1, wherein obtaining a set of intermediate image features for characterizing the target object comprises: Determining a region of interest in the input image based on a position of the target object in the input image; extracting the region of interest from the input image; Obtaining a processed image by removing a portion other than the target object from the extracted region of interest; as well as The set of intermediate image features is generated based on the processed image.

5. The method of claim 4, wherein generating the set of intermediate image features based on the processed image comprises: Based on the processed image, the set of intermediate image features is generated using a predetermined number of network layers in an image encoder.

6. The method according to claim 5, wherein determining the target image feature comprises: Obtaining output image features of an output layer of the image encoder; as well as The output image features are converted into the target image features using an image feature mapper.

7. The method of claim 1 , wherein generating the output image comprises: The output image is generated from an initial noisy image using a diffusion model and taking the target image features and the target text features as conditions.

8. The method according to claim 7, wherein the initial noisy image comprises at least one of the following: White noise image, or The input image is applied with noise, the applied noise being above a threshold noise level.

9. The method of claim 7, wherein generating the output image from the initial noisy image comprises a plurality of denoising steps, and a given denoising step of the plurality of denoising steps comprises: Determining visual attention features by applying an attention mechanism to the features of the image to be denoised in the given denoising step and the features of the target image; Determine text attention features by applying an attention mechanism to the features of the image to be denoised and the features of the target text; as well as Based on the visual attention feature and the text attention feature, an output image feature of the given denoising step is generated as the next denoising step of the given denoising step. enter.

10. The method of claim 9, wherein determining the text attention feature comprises: Using a first conversion unit to convert the target text feature into a first feature; Using a second conversion unit to convert the target text feature into a second feature; Using a third conversion unit, the image feature to be denoised is converted into a third feature; and Based on the first feature, the second feature and the third feature, the text attention feature is determined, wherein At least one of the first conversion unit, the second conversion unit, and the third conversion unit includes a first sub-unit obtained through pre-training and a second sub-unit obtained through fine-tuning.

11. The method of claim 9, wherein generating the output image features of the given denoising step comprises: The output image feature is obtained by weighting the visual attention feature and the text attention feature with a first weight and a second weight respectively. 12 . The method according to claim 10 , wherein in the training process of the diffusion model, the value of the first weight and the value of the second weight are determined according to random numbers.

13. The method according to claim 10, wherein the step of training the diffusion model comprises: Obtaining an input sample image and an input sample text including a sample object; Based on the input sample image and the input sample text, using the diffusion model, generating an output sample image associated with the input sample image and the input sample text; Extracting a first object feature representing the sample object from the input sample image; extracting a second object feature representing the sample object from the output sample image; as well as The second subunit is updated based on a difference between the first object feature and the second object feature.

14. A device for image generation, comprising: An intermediate feature acquisition module, configured to acquire a set of intermediate image features for characterizing the target object based on an input image including the target object; an image feature determination module, configured to determine a target image feature for characterizing the target object based on the set of intermediate image features; A text feature determination module, configured to determine a target text feature for characterizing an image generation target based on the input text and the set of intermediate image features; as well as The image generation module is configured to generate an output image associated with the input image and the input text based on the target image feature and the target text feature.

15. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 13 when executed by the at least one processing unit.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 13.

Images