Image generation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image generation method and device, equipment and a storage medium. The method includes acquiring a set of intermediate image features for characterizing a target object based on an input image including the target object. The method further includes determining a target image feature for characterizing the target object based on the set of intermediate image features. The method further includes determining a target textual feature for characterizing an image generation target based on the input text and the set of intermediate image features. The method further includes generating an output image associated with the input image and the input text based on the target image feature and the target text feature. In this way, high-quality image generation can be advantageously achieved in accordance with various desired image generation objectives.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for image generation. Background Art

[0002] In the field of computer vision (CV), various image generation technologies based on machine learning have been significantly developed and have a wide range of applications. For example, in many application scenarios such as social networking, games, and image editing, it is expected to generate and use images including target objects (e.g., target people). Image generation technology based on machine learning can be used in such application scenarios to improve the user's experience. In some example application scenarios, it is expected to generate an image that matches the user input based on the user's input information, such as text description information. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for image generation is provided. The method includes acquiring a set of intermediate image features for characterizing a target object based on an input image including a target object. The method also includes determining target image features for characterizing the target object based on the set of intermediate image features. The method also includes determining target text features for characterizing an image generation target based on input text and the set of intermediate image features. The method also includes generating an output image associated with the input image and the input text based on the target image features and the target text features.

[0004] In a second aspect of the present disclosure, a device for image generation is provided. The device includes: an intermediate feature acquisition module configured to acquire a set of intermediate image features for characterizing a target object based on an input image including the target object; an image feature determination module configured to determine target image features for characterizing the target object based on a set of intermediate image features; a text feature determination module configured to determine target text features for characterizing an image generation target based on input text and a set of intermediate image features; and an image generation module configured to generate an output image associated with the input image and the input text based on the target image features and the target text features.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A schematic block diagram showing an example architecture for image generation according to some embodiments of the present disclosure is shown;

[0011] Figure 3 Another schematic block diagram illustrating an example architecture for image generation according to some embodiments of the present disclosure;

[0012] Figure 4 A schematic diagram showing a diffusion model according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A schematic diagram showing an attention unit in a diffusion model according to some embodiments of the present disclosure is shown;

[0014] Figure 6 A flowchart showing a process for image generation according to some embodiments of the present disclosure is shown;

[0015] Figure 7 A block diagram showing an apparatus for image generation according to some embodiments of the present disclosure; and

[0016] Figure 8 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0018] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0019] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0020] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0021] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0024] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0025] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0026] As used herein, the term "machine learning model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, the corresponding output can be generated for a given input. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. In this article, a "machine learning model" may also be referred to as a "machine learning network" or "network", and these terms are used interchangeably in this article.

[0027] As used herein, the term "based on a set of intermediate image features" may refer to based on part of the intermediate image features in the set of intermediate image features (for example, one intermediate image feature or multiple intermediate image features in the set of intermediate image features) or based on all of the intermediate image features in the set of intermediate image features.

[0028] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the use stage (also called the reasoning stage). In the training stage, a given model can be trained using a large amount of training data, and iterated continuously until the model can obtain consistent reasoning that meets the expected goals from the training data. Through training, the model can be considered to be able to learn the association from input to output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the reasoning stage, the model can be used to process the actual input based on the parameter values ​​obtained through training to determine the corresponding output.

[0029] As mentioned above, in the field of CV, various image generation techniques based on machine learning have been significantly developed and have been widely used. In some example applications, it is expected that based on user input information, such as text description information, a machine learning model is used to generate an image that matches the user input.

[0030] At present, some image generation schemes based on text-to-image models have been proposed. These schemes are based on natural language prompts, such as text used to describe a scene, and use trained text-to-image models to generate realistic images corresponding to the described scene. These text-to-image models are usually trained using a large number of text-to-image databases. However, sometimes users want to integrate themselves, their family or their friends into new scenes. These new scenes do not exist in the dataset used to train the model. In other words, there is no text and image corresponding to the new scene in the dataset used to train the model. Therefore, for such new scenes, it becomes very difficult to generate a relatively realistic image that conforms to the scene only through text description.

[0031] In order to realize customized text-to-image image generation for different scenes, especially customized face image generation, a scheme based on image information retention has been proposed. In this scheme, the input image of the target object, such as a person image or a face image, is tokenized in the text space to obtain a unique token of the input image in the text space. Obtaining the token of the input image is also called inverse transformation of the input image into pseudowords. The tokenization process of this input image is usually a subject-based optimization process. The scheme adopts a text-to-image model to generate an output image based on both the token of the input image and the text describing the scene input by the user. This scheme is based on the token of the input image in the process of generating the image, so the image information of the input image is retained in the output image, that is, it has image retention. In addition, the scheme can generate images corresponding to different scenes based on various different text inputs, so it has editability.

[0032] However, this solution usually requires a lot of computing resources and storage resources, and takes a long time to process. Such a long processing time and a long fine-tuning process make it difficult for users to quickly obtain the desired image. On the other hand, due to the limitation of the number of people in the data set for training the model, the model is only trained on images of people with a specific image, so there will be an overfitting problem. This overfitting problem limits the editing ability of this solution.

[0033] To address the above issues, some encoder-based solutions have been proposed to optimize customized text-to-image generation by optimizing pseudo-label embedding, referencing sampling steps, or adding new concept injection modules. However, these solutions are difficult to achieve satisfactory results. In addition, these solutions require multiple input images and need to be fine-tuned during testing to meet image retention and editability.

[0034] In summary, the above solutions have many defects such as high cost, long generation time, lack of character image similarity, limited editing ability, etc. To this end, according to an embodiment of the present disclosure, a solution for generating image feature representation is provided, aiming to solve one or more of the above problems and other potential problems.

[0035] In this scheme, a group of intermediate image features characterizing a target object (e.g., a target person) is obtained from an input image including the target object. Based on the group of intermediate image features, target image features for characterizing the target object are determined. In addition, based on the user's input text and the group of intermediate image features, target text features for characterizing an image generation target are determined. For example, the image generation target may indicate a desired scene or a desired image style, and so on. Further, based on the determined target image features and target text features, an output image associated with the input image and the input text is generated. For example, the output image may be an image of the target person that matches the scene or image style indicated by the image generation target.

[0036] In this way, an output image that meets the user's expectations can be generated based on both the input image and the input text input by the user. In particular, in the process of determining the target text features used to characterize the image generation target, not only the user's input text is used, but also the intermediate image features related to the input image are taken into account. In this way, the target text features can reflect the relevant information of both the input image and the input text. The output image generated by combining such target text features with target image features can better match the image generation target expected by the user.

[0037] Some example embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0038] Example Environment

[0039] Figure 1 A block diagram of an example environment 100 is shown in which various implementations of the present disclosure can be implemented. Figure 1 In an environment, the electronic device 110 is configured to generate an output image 112 based on the input text 102 and the input image 104. The input image 104 includes a target object, such as a target person, an animal, a plant, or other static and dynamic objects, and may even include a composite object, etc. The target object may also be at least a part of the above-mentioned object, such as the face, half body or whole body of the target person, or a part or all of other objects. In some embodiments below, a person or a person's face will be used as an example of a target object, but it should be understood that this is merely exemplary and is not intended to be limiting. The embodiments described with reference to "person" or "person's face" are applicable to other types of objects.

[0040] The input text 102 may be descriptive information about the user's desired output image 104. For example, the input text 102 may describe the scene, environment, state, etc. in the desired output image 104. For another example, the input text 102 may also describe the style of the desired output image 104. It is desired that, based on the input text 102 and the input image 104, an output image 104 including a target object (e.g., the head, half body, or full body of a target person) is generated that matches the scene, environment, state, style, etc. described by the input text 102.

[0041] As an example, the input image 104 may be an image including a target person (represented by a placeholder S* in the example of this article), such as a photo of the person, a video screenshot, etc. If the user expects to obtain a cartoon-style image of the person, an input text 102 similar to "S*in cartoon style" may be input to the electronic device 110. For another example, if the user expects to obtain an image of the person wearing a sci-fi spacesuit in a space environment, the input text 102 may be similar to "S*wears a sci-fi spacesuit in space". In this way, by inputting different input texts 102, output images 112 of different scenes, environments, states, and styles can be generated for target objects included in different input images 104. This will be discussed below in conjunction with Figures 2 to 5 Several examples of generating output image 112 based on input text 102 and input image 104 are described.

[0042] Although in the examples herein, the input text 102 is shown as English, it should be understood that this is only illustrative and is not intended to limit the protection scope of the present disclosure. The embodiments of the present disclosure are applicable to texts in any language.

[0043] In some embodiments, the electronic device 110 may be a user's terminal device. The electronic device 110 may include any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Alternatively, in some embodiments, the electronic device 110 may also be an external device or server that communicates remotely with the user's terminal device, including but not limited to mainframes, edge computing nodes, computing devices in a cloud environment, and the like. The electronic device 110 may be any electronic device with computing capabilities, including servers, blade servers, mainframes, edge devices, and other appropriate computing devices.

[0044] It should be understood that Figure 1The components and arrangements of the environment 100 shown are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements.

[0045] Overview of Image Generation

[0046] Figure 2 A schematic block diagram of an architecture 200 for image generation according to some embodiments of the present disclosure is shown. Figure 2 The architecture 200 can be implemented, for example, in Figure 1 The architecture 200 includes at least a feature extraction subsystem 210 for image feature extraction and text feature extraction. The architecture 200 further includes an image generation subsystem 220 for generating an output image 112 based on the features extracted by the feature extraction subsystem 210. The various components and models in the feature extraction subsystem 210 and the image generation subsystem 220 can be implemented by hardware, software, firmware, or any combination thereof.

[0047] like Figure 2 As shown, the feature extraction subsystem 210 determines target text features 212 and target image features 224 based on the input text 102 and the input image 104. The input image 104 includes a target object 204. For example, the input image 104 may include the face, half body, or full body of a target person.

[0048] Specifically, the feature extraction subsystem 210 obtains a set of intermediate image features 214 for characterizing the target object 204 based on the input image 104. For example, the feature extraction subsystem 210 may determine a region of interest of the input image 104 based on the position of the target object 204 in the input image 104. An interesting portion of the target object 204, such as a face, is located within the region of interest. The feature extraction subsystem 210 may perform feature extraction on the portion of the input image 104 located in the region of interest to obtain the intermediate image features 214.

[0049] The feature extraction subsystem 210 determines a target image feature 224 for characterizing the target object based on a set of intermediate image features 214. As an example, the intermediate image features 214 and the target image features 224 can be determined by a model such as an image encoder or an image feature mapper. For example, different models can be used to determine the intermediate image features 214 and the target image features 224, respectively. For another example, a single model can be used to determine both the intermediate image features 214 and the target image features 224, wherein the intermediate image features 214 can be output by a hidden layer of the model, and the target image features 224 can be output by an output layer of the model.

[0050] The feature extraction subsystem 210 determines the target text features 212 for characterizing the image generation target based on the input text 102 and a set of intermediate image features 214. For example, the feature extraction subsystem 210 can determine the text feature representation of a set of intermediate image features 214 and the text feature representation of at least a portion of the input text 102. The feature extraction subsystem 210 then generates the target text features 212 based on the determined above text feature representations. As an example, the text feature representation can be a vectorized representation of any suitable type. Similarly, the intermediate image features 214, the target image features 224, and the target text features 212 can all be vectorized representations of any suitable type.

[0051] like Figure 2 As shown, the target image feature 224 and the target text feature 212 can be determined based on a set of intermediate image features 214. In some embodiments, the target image feature 224 can be determined based on one of the intermediate image features 214, and the target text feature 212 can be determined based on the remaining intermediate image features except the intermediate image feature, such as will be referred to below. Figure 3 Descriptive.

[0052] When the target text features 212 and the target image features 224 are determined, the image generation subsystem 220 may generate the output image 112 based on the target text features 212 and the target image features 224. For example, the image generation subsystem 220 may use a trained machine learning model to generate the output image 112. As an example, the machine learning model used may include a diffusion model.

[0053] In this way, an output image 112 that meets the user's expectations can be generated based on both the input image 104 and the input text 102 input by the user. In particular, in the process of determining the target text feature 212 used to characterize the image generation target, not only the user's input text 102 is used, but also a group of intermediate image features 214 related to the input image 104 are taken into account. In this way, the target text feature 212 can reflect the relevant information of both the input image 104 and the input text 102. The output image generated by combining such a target text feature 212 with the target image feature 224 can better match the image generation target expected by the user.

[0054] In addition, in the embodiments of the present disclosure, fine-tuning during testing can be avoided, and image generation can be achieved by only requiring the user to input a single image. In this way, fast and easy image generation can be achieved, for example, an image can be generated in just a few seconds.

[0055] Example architecture for image generation

[0056] Combination of the above Figure 2 Describes an example architecture for image generation. Next, we will combine Figure 3 A more detailed example architecture 300 for image generation is described. The architecture 300 may be implemented, for example, in Figure 1 At the electronic device 110. Similar to the architecture 200, the architecture 300 includes a feature extraction subsystem 210 for image feature extraction and text feature extraction and an image generation subsystem 220 for generating an image. The various components and models in the architecture 300 can be implemented by hardware, software, firmware, or any combination thereof.

[0057] like Figure 3 As shown, the input image 104 is input to the feature extraction subsystem 210. The input image 104 includes a target object, such as a target person or a portion of a target person. In some embodiments, the feature extraction subsystem 210 may pre-process the input image 104 to obtain a processed image 304. For example, the feature extraction subsystem 210 may determine a region of interest of the input image 104 based on the position of the target object in the input image 104. The region of interest may be a region where the target object is located. The region of interest may also be a region where an interested portion of the target object is located.

[0058] exist Figure 3 In the example of , the region of interest is the region where the face of the target person is located. In the embodiment where the region of interest is the face of the target person, the feature extraction subsystem 210 may use face detection technology to identify and locate the face in the input image 104. For example, face detection may be performed based on one or more predetermined facial feature points. In this way, it is possible to ensure that subsequent feature extraction is concentrated in the region of interest.

[0059] Additionally or alternatively, in some embodiments, after determining the region of interest, the scope of the region of interest can be appropriately adjusted. For example, the scope of the region of interest including the face region can be expanded according to a predetermined ratio parameter. The predetermined ratio parameter can be, for example, 1.3 or other values. By adjusting the scope of the region of interest, the region of interest can include images such as hair and part of the human body. In this way, it can be ensured that the features of the target object, such as details of facial features (e.g., expression, hairstyle, hair color, etc.), are included in the region of interest, thereby avoiding feature omissions.

[0060] In some embodiments, the feature extraction subsystem 210 may extract a region of interest from the input image 104. By removing portions of non-target objects from the extracted region of interest, a processed image 304 is obtained. For example, the feature extraction subsystem 210 may remove non-face elements from the region of interest. As an example, a mask may be used to process the region of interest to remove non-face elements such as background, accessories, etc. In this way, the accuracy and efficiency of subsequent feature extraction can be ensured.

[0061] In some embodiments, the feature extraction subsystem 210 further scales the processed image 304 to obtain a processed image 304 of a desired size. For example, the desired size may be an image size required by a model subsequently used to generate the output image 112. The desired size may be preset or adjusted by a user or the feature extraction subsystem 210. The embodiments of the present disclosure are not limited in this respect.

[0062] After obtaining the processed image 304, the feature extraction subsystem 210 can generate a set of intermediate image features 214 based on the processed image 304. As an example, the feature extraction subsystem 210 can generate a set of intermediate image features 214 based on the processed image 304 using a predetermined number of network layers in the image encoder 310. The output of each of these predetermined number of network layers can be determined as an intermediate image feature. For example, the output features of the output layer in the image encoder 310 and the output features of m (m is an integer greater than or equal to 1) hidden layers close to the output layer can be selected as a set of intermediate image features 214. In this way, the representation ability of image feature representation (such as image labeling) and the editorability of the model can be enhanced, and these image features capture spatial and semantic information at different levels of abstraction and concreteness.

[0063] The image encoder 310 may be a CLIP image encoder or other types of image encoders. The image encoder 310 may be the same type as the image encoder used in the subsequent image generation subsystem 220. By using the CLIP image encoder, it is not necessary to train it, thereby simplifying the feature extraction process.

[0064] Additionally, in some embodiments, the feature extraction subsystem 210 may also obtain output image features 314 of the output layer of the image encoder 310. The feature extraction subsystem 210 may then use an image feature mapper 330 to convert the output image features 314 into target image features 224. As an example, the image feature mapper 330 may be implemented using a multi-layer perceptron and is also referred to as an image adapter. The target image features 224 may also be referred to as target image tags.

[0065] In some embodiments, the image encoder 310 does not need to be trained. The image feature mapper 330 can be trained. By combining the image encoder 310 and the image feature mapper 330, the feature extraction process can be simplified.

[0066] Alternatively, in some embodiments, the output image features 314 of the image encoder 310 may be determined as the target image features 224. That is, in such embodiments, the image feature mapper 330 may be omitted. The scope of the present disclosure is not limited in this regard.

[0067] In some embodiments, the performance of the feature extraction subsystem 210 is affected by the capability of the subsequent text encoder 340. The semantic information in the text space is often abstract, which places higher demands on the text feature representation capability (i.e., the tag representation capability). Therefore, by incorporating the intermediate image features 214 in the image space as an auxiliary into the model, it is more specific for the model to understand new concepts and helps to effectively learn pseudo tags in the text space.

[0068] Specifically, the feature extraction subsystem 210 may determine the target text features 212 based on a set of intermediate image features 214 and at least a portion of the input text 102. Figure 3 In the example of FIG. 1 , the input text 102 is “A photo of S*”. It should be understood that the input text is merely exemplary and not restrictive. The input text 102 may include any appropriate content.

[0069] In some embodiments, the feature extraction subsystem 210 can use a text feature mapper 320 (also known as a text perception machine) to convert a set of intermediate image features 214 into a first set of text feature representations 322. For example, a portion of the intermediate image features or all of the intermediate image features in the set of intermediate image features 214 can be converted into a first set of text feature representations 322. In this document, the first set of text feature representations 322 is also referred to as pseudo-label embeddings or pseudo-label embedding vectors. As an example only and without any limitation, the text feature mapper 320 can convert the intermediate image features of the m hidden layers from the image encoder 310 into word embeddings to obtain a first set of text feature representations 322. The first set of text feature representations 322 can be represented as S*=S1,...,S m , which can be represented as a pseudo-word "S*", such as Figure 3 In this way, image features that are difficult to be properly described by existing words can be embedded into the text space, thereby obtaining a feature description of the image information of the target object (eg, a face).

[0070] In some embodiments, the text feature mapper 320 can be a model composed of two or more multi-layer perceptron (MLP) layers with non-linear activation functions. The text feature mapper 320 can be trained or fine-tuned. Each MLP layer can convert the intermediate image features from a network layer of the image encoder 310 into a text feature representation, such as a word embedding. By adopting a text feature mapper 320 with an MLP layer, the text feature mapper 320 can be made lightweight and can quickly obtain a text feature representation.

[0071] The feature extraction subsystem 210 may determine a second set of text feature representations 324 based on at least a portion of the input text 102. For example, the feature extraction subsystem 210 may determine a first subtext representing a target object in the input text 102 and a second subtext other than the first subtext in the input text 102. In the example where the input text 102 is “A photo of S*”, the first subtext is “S*” and the second subtext is “A photo of”.

[0072] The feature extraction subsystem 210 can then generate the target text features 212 based on the first set of text feature representations 322 and the second set of text feature representations 324. The target text features 212 can also be referred to as target text tags. As an example, the feature extraction subsystem 210 can generate the target text features 212 based on the first set of text feature representations 322 and the second set of text feature representations 324 using a text encoder 340.

[0073] In some embodiments, the text encoder 340 can be a pre-trained model and can be used directly without training. As an example, the text encoder 340 can be a CLIP text encoder. In some embodiments, by obtaining the intermediate image features 214 by the CLIP image encoder and generating the target text features 212 by the CLIP text encoder, the image features can be converted to the text space more efficiently and accurately.

[0074] By obtaining a set of intermediate image features 214 using the image encoder 310 and mapping these features using the text feature mapper 320, the target text features 212 generated thereby can capture important image clues or visual clues related to the image of the target object. Using these image clues related to the image of the target object, an output image 112 that meets the expectations can be generated later.

[0075] In some embodiments, the image feature mapper 330 and the text feature mapper 320 may adopt the same structural design, so that the target image feature 224 and the target text feature 212 adopt the same or similar vectorized representation, which can facilitate subsequent image generation.

[0076] The above describes the feature extraction process of the feature extraction subsystem 210. As mentioned above, some modules in the feature extraction subsystem 210, such as the image encoder 310 and the text encoder 340, do not need to be trained or fine-tuned. In this way, the training of the architecture 300 can be simplified.

[0077] Next, the image generation subsystem 220 may generate the output image 112 based on the extracted features. In some embodiments, the image generation subsystem 220 may generate the output image 112 from the initial noisy image using the diffusion model 350, conditioned on the target image features 224 and the target text features 212. In this document, the target image features 224 are also referred to as image conditions of the diffusion model 350, and the target text features 212 are also referred to as text conditions of the diffusion model 350.

[0078] In some embodiments, the initial noisy image may include a white noise image. By using a white noise image, the editability and richness of the output image 112 can be improved. Additionally or alternatively, in some embodiments, the initial noisy image may include an input image 104 with noise applied, and the applied noise is higher than a threshold noise level. The threshold noise level may be pre-set or dynamically set by a user or the image generation subsystem 220. By using an input image with noise applied, the output image 112 can be made closer to or maintain the posture, perspective, etc. of the target object in the input image 104.

[0079] As will be described in detail below, the process of generating the output image 112 using the diffusion model 350 may include multiple denoising steps, for example, a total of T denoising steps, where T is an integer greater than or equal to 1. These denoising steps are executed in a decreasing order of index, for example, first executing the Tth step, then the T-1 step, the T-2 step, and so on until the 1st step. The earlier the denoising step is executed, the earlier the noise level is processed. Therefore, in this embodiment, the input image can be introduced at the early stage of denoising to ensure that the noise level applied to the input image is high, for example, higher than a threshold. That is, the input image applied can be introduced in a denoising step close to the T step.

[0080] In some embodiments, the target image features 224 and the target text features 212 may be input (also referred to as injected) into the diffusion model 350 at different ratios. Alternatively, the target image features 224 and the target text features 212 may be input into the diffusion model 350 as a whole at the same ratio. The ratio of the target image features 224 and the target text features 212 input into the diffusion model 350 may be preset, dynamically adjusted, or adjusted according to user input. The embodiments of the present disclosure are not limited in this respect.

[0081] In some embodiments, the diffusion model 350 and the image feature mapper 330 and the text feature mapper 340 may be trained based on the loss value 360. The loss value 360 ​​may include a loss associated with the target object and a loss associated with the image reconstruction of the diffusion model 350. Figure 4 and Figure 5 A more detailed example of diffusion model 350 generating output image 112 is described further.

[0082] According to the above reference Figure 3 In the embodiment of the present disclosure, the user does not need to input multiple images. This can advantageously achieve fast and efficient image generation.

[0083] Example Diffusion Model

[0084] Diffusion model, also known as Denoising Diffusion Probabilistic Model, is a type of generative model. The mechanism of common generative models (e.g., Generative Adversarial Network (GAN), Variational Autoencoder (VAE)) is mainly to gradually add information by giving an input of "restriction" (such as type, style, etc.), and finally obtain the generated result (e.g., image, speech, etc.). Different from common generative models, diffusion models gradually "sample" special distributions from noise information (e.g., Gaussian noise) according to certain conditions, and finally obtain the generated result as the number of "sampling" rounds increases. In other words, the generation process of the diffusion model is to extract the required data from the noise through multiple iterations, and as the number of iterations increases, the quality of the generated data becomes better and better. The input of the diffusion model includes Gaussian noise, and the output is the data to be generated.

[0085] In general, the modeling of the diffusion model includes a forward diffusion process with continuous noise addition and a reverse diffusion process with continuous noise removal. The forward diffusion process can be expressed as a Markov forward process. The desired data generation process corresponds to the reverse process, for example, the desired data is obtained by continuously reversing the Markov forward process. In this paper, the diffusion model is used to generate images.

[0086] In some embodiments, the diffusion model can use an attention mechanism to generate images. In some embodiments, the diffusion model can include a text encoder (denoted as c θ ), and the text encoder can be used to map the input text y to the intermediate representation c θ(y). The intermediate representation is then adopted in one or more intermediate layers of the denoising sub-network through an attention mechanism. As an example, the attention mechanism can be a cross-attention mechanism. Formula (1) shows an example of an attention mechanism adopted in an intermediate layer of the denoising sub-network:

[0087]

[0088] Where Attn(Q, K, V) represents the obtained attention value, Represents the query feature, K = W K ·c θ (y) represents the bond characteristics, C = W V ·c θ (y) represents the value feature, z t represents the noise latent representation in the t-th step (t is an integer greater than or equal to 1 and less than or equal to T) (i.e., the noise latent representation in the t-th iterative denoising step), represents the hidden state in the denoising subnetwork at the tth step, T is the total time or the total number of iterative denoising steps, d′ represents the scaling factor used for the attention mechanism, and Softmax() represents the normalized exponential function.

[0089] In this paper, the diffusion model using the denoising subnetwork structure is also called the latent diffusion model. The training goal of the latent diffusion model is to predict the noise added to the latent representation of the image. The process can be expressed as formula (2):

[0090]

[0091] in, represents the denoising network, ∈ represents the unscaled noise sample, ε(x) represents the use of encoder ε to map the image x from the pixel space to the latent space, and c θ (y) Input text y using text encoder c θ The intermediate representation mapped to, ∈ θ (z t ,t,c θ (y)) represents the time t corresponding to z t and the intermediate representation c θ (y) noise, L D represents the loss value in the diffusion process of the diffusion model (ie, the loss value of the diffusion model for image reconstruction). In some schemes, the image can be generated by optimizing the loss value of the diffusion model in the diffusion process.

[0092] The above describes a general example of using the attention mechanism to generate an image using the diffusion model. In an embodiment of the present disclosure, the diffusion model 350 generates the output image 112 from the initial noisy image using the attention mechanism, taking the target image features 224 and the target text features 212 as conditions.

[0093] Figure 4 An example architecture of a diffusion model 350 for image generation according to an example embodiment of the present disclosure is shown. It should be understood that Figure 4 The architecture of the diffusion model shown is exemplary only. As shown, the diffusion model 350 includes an autoencoder, which can be represented as where ε represents the encoder, and Denotes a decoder. The diffusion model 350 includes a denoising (also referred to as noise reduction) network (denoted as ).

[0094] In some embodiments, the autoencoder Trained on a large-scale image dataset for image compression. Specifically, the encoder ε is used to map the image x from the pixel space to a low-dimensional latent space (also called hidden space), that is, to obtain the latent representation z = ε (x). The decoder is used to reconstruct the latent representation z back into an image with high fidelity in order to achieve

[0095] In some embodiments, the denoising network A plurality of denoising subnetworks may be included, for example, denoising subnetwork 450, denoising subnetwork 455, and so on. Each denoising subnetwork may be used for a denoising step. For example, a denoising subnetwork architecture may be used to implement the denoising subnetwork. is trained to perform the back diffusion process in the latent space. Using a denoising network to perform the back diffusion process can reduce the consumption of computing resources and storage resources. Figure 4 The conditions (in the embodiment of the present disclosure, the target image features 224 and the target text features 212) are injected into the denoising network

[0096] As described above, using the diffusion model 350, the attention mechanism is used to generate the output image 112 from the initial noisy image 352, with the target image feature 224 and the target text feature 212 as conditions. In some embodiments, generating the output image 112 from the initial noisy image includes multiple denoising steps. Each denoising step uses a denoising subnetwork. For example, assume that the multiple denoising steps include T iterative denoising steps. Figure 4In the example of , in the T-th denoising step, the denoising sub-network 450 is used, and in the T-1-th denoising step, the denoising sub-network 455 is used. Each denoising sub-network may include one or more attention units, such as the attention unit 460.

[0097] In a given denoising step t in the plurality of denoising steps, by performing the denoising operation on the image feature 352z to be denoised in the given denoising step t t The attention mechanism is applied to the target image feature 224 to determine the visual attention feature. If the given denoising step t is the first step of the denoising process, the image feature to be denoised 352 is the feature of the initial noisy image, otherwise the image feature to be denoised 352 may be the output of the previous denoising step t+1. t The attention mechanism is applied to the target text feature 212 to determine the text attention feature. Based on the visual attention feature and the text attention feature, the output image feature 353z of the given denoising step t is generated t-1 , as the input of the next denoising step t-1 of a given denoising step.

[0098] Example Attention Unit

[0099] In a given denoising step t, the attention unit 500 may be used to determine the visual attention features and the textual attention features, for example, to achieve Figure 4 The attention unit 500 can use the first conversion unit to convert the target text feature 212 into a first feature (also represented by K T ), which can be used as a key in the attention mechanism. The attention unit 500 can use the second conversion unit to convert the target text feature 212 into a second feature (also represented as V T ), which can be used as a value in the attention mechanism. The attention unit 500 can use the third conversion unit to convert the image feature to be denoised into a third feature (also represented as Q), which can be used as a query in the attention mechanism. The attention unit 500 can then determine the text attention feature based on the first feature, the second feature, and the third feature. In some embodiments, at least one of the first conversion unit, the second conversion unit, and the third conversion unit includes a first subunit obtained by pre-training and a second subunit obtained by fine-tuning.

[0100] Similarly, the attention unit 500 can use, for example, the fourth conversion unit to convert the denoised image feature into a fourth feature (also referred to as Q), which can be used as a query in the attention mechanism. The attention unit 500 can use the fifth conversion unit to convert the target image feature 224 into a fifth feature (also referred to as K S), which can be used as a key in the attention mechanism. The attention unit 500 can use the sixth conversion unit to convert the target image feature 224 into a sixth feature (also referred to as V S ), which can be used as a value in the attention mechanism. The attention unit 500 can then determine the visual attention feature based on the fourth feature, the fifth feature, and the sixth feature. In some embodiments, at least one of the fifth conversion unit and the sixth conversion unit can be obtained by fine-tuning.

[0101] Figure 5 A schematic diagram of an attention unit 500 is shown. As shown, the attention unit 500 includes a text attention subunit 510 and a visual attention subunit 520. In some embodiments, a hidden state 530 may be input to at least a portion of the text attention subunit 510 and the visual attention subunit 520. Depending on the position of the attention unit 500 in the denoising subnetwork, the hidden state 530 may be a to-be-denoised image feature of the corresponding denoising step or may be the output of the previous attention unit.

[0102] In some embodiments, a fine-tunable solution can be adopted for the text attention unit 510. For example, the text attention unit 510 can include a base part and a fine-tunable part. Exemplarily, the model parameters of the base part can be determined by pre-training. Then, the model parameters of the fine-tunable part can be determined in the fine-tuning stage. In the inference process, the attention mechanism can be applied based on the model parameters obtained by pre-training and the model parameters obtained by fine-tuning.

[0103] exist Figure 5 In the example of , the fine-tunable subunit can be adopted by the text attention subunit 510 to calculate the text attention feature. Specifically, the text attention subunit 510 can use the first conversion unit (including subunit 540 and subunit 545) to adopt K T =W k T p+αΔW k T p is used to calculate the first feature used as the key, where α represents the weight and p represents the target text feature 212. Similarly, the text attention subunit 510 adopts the second conversion unit (including subunit 550 and subunit 555) to adopt V T =W v T p+αΔW v Tp is used to calculate the second feature used as the value, where α represents the weight and p represents the target text feature 212. The text attention subunit 510 can also use the third conversion unit (including subunit 560 and subunit 565) to calculate the third feature used as the query heat. Based on the first feature, the second feature and the third feature, the text attention feature Attn(Q, K T , V T ). For example, ΔW in the above calculation k T and ΔW v T Denote the low-rank decomposition ΔW = BA, where A and B include fine-tunable parameters. Figure 5 In the example shown, subunits 540, 550 and 560 may be obtained through fine-tuning, and subunits 545, 555 and 565 may be obtained through pre-training. In this way, the amount of training may be reduced and the training efficiency may be improved.

[0104] In some embodiments, the visual attention subunit 520 determines the visual attention feature by applying an attention mechanism to the image feature to be denoised and the target image feature 224 in a given denoising step. As an example, the visual attention subunit 520 can use the fourth conversion unit 570 to determine the fourth feature used as a query in The hidden state in the denoising subnetwork of the denoising step t is shown in Table 1. The visual attention subunit 520 can use the fifth conversion unit 580 to adopt K S =W k S f determines the fifth feature used as a key, and uses the sixth conversion unit 590 to adopt V S =W v S f determines the sixth feature used as the value, where f represents the acquired target image feature 224. Based on the fourth, fifth and sixth features, the visual attention feature Attn (Q, K S , V S ). In some embodiments, the fifth conversion unit 580 and the sixth conversion unit 590 may be obtained through fine-tuning, and the fourth conversion unit 570 may be obtained through pre-training.

[0105] In some embodiments, based on the visual attention feature Attn(Q, K S , V S ) and text attention feature Attn(Q, K T , V T ), generating the output image features of a given denoising step as the input of the next denoising step of the given denoising step. As an example, the text attention features Attn(Q, K) determined by the text attention subunit 510 can beT , V T ) and the visual attention feature Attn(Q, K S , V S ) are weighted summed to generate the output image features for a given denoising step.

[0106] In some embodiments, the diffusion model 350 may obtain a first weight for the image feature and a second weight for the text feature. The visual attention feature and the text attention feature may be weighted by the first weight and the second weight, respectively. For example, the following formula may be used to determine:

[0107] O = γAttn(Q, K T , V T )+σAttn(Q,K S , V S ) (3)

[0108] Where O represents the feature output by the current attention unit, Attn(Q, K T , V T ) represents the text attention feature, Attn(Q, K S , V S ) represents the visual attention feature, γ and σ represent the second weight and the first weight (also known as the scaling factor), respectively. During the inference process, the values ​​of γ and σ can be fixed, dynamic, or determined according to user input. The relative size of γ and σ can indicate whether the generated output image is more consistent with the input text or the input image.

[0109] In some embodiments, during the training of the diffusion model, the value of the first weight and the value of the second weight are determined according to random numbers. For example, a random seed may be introduced during the process of determining the output image features. The random seed may be obtained from a uniform distribution. In the example of introducing random seeds, the output image features can be obtained as follows:

[0110]

[0111] Where r1 and r2 are thresholds of random seeds, which thresholds can be preset or dynamically configured, for example. Using a random approach can prevent overfitting. In this way, a random fusion strategy can be used for each multimodal branch of the attention unit 500 to determine the output image features.

[0112] Additionally or alternatively, in some embodiments, during the reasoning process, the first weight and the second weight may be pre-set or set by the user. For example, the user may set different weights as required. In an example where the user expects to obtain an output image 112 that is more consistent with the input text 102, the second weight may be set larger, while the first weight may be set smaller. For another example, in an example where the user expects to obtain an output image 112 that is more consistent with the input text 102, the first weight may be set larger, while the second weight may be set smaller. The user may also adjust the first weight and the second weight according to other factors. The embodiments of the present disclosure are not limited in this respect.

[0113] Several examples of denoising steps of the diffusion model 350 are described above. In some embodiments, during the inference process of the diffusion model 350, random noise, such as random Gaussian noise z T is sampled, and through a series of T iterative denoising steps, z T is converted to z′0. Then, using the decoder To generate the output image 112, that is, to generate the output image

[0114] In some embodiments, diffusion model 350 determines output image 112 by optimizing a loss (eg, minimizing a loss value).

[0115] In some embodiments, the first set of text feature representations 322 (also referred to as pseudo-label embeddings p ) obtained by the text feature mapper 320 is f ) and the feature V of the target image feature S Specifically, the following formulas (5) and (6) can be used to regularize p f and V S Regularization:

[0116]

[0117]

[0118] Where Mean() represents the average function. and They represent the regularized p f and V S .

[0119] In some embodiments, a loss value associated with the target object is determined. For example, a loss value associated with the face or image of the target object (also referred to as face image preservation loss). As an example, the loss value associated with the target object may be determined using the following formula:

[0120]

[0121] Wherein, f() represents a domain-specific feature extractor, x represents a reference image, and x' is a denoised image corresponding to a text input 102 such as "A photo of S*". represents the similarity function. As an example, It can be used to calculate the cosine similarity between features extracted from the face region in the image.

[0122] Taking into account the above loss value associated with the target object, the total training loss of the diffusion model 350 can be obtained as follows:

[0123]

[0124] in represents the total loss value, represents the loss value of the diffusion model 350 during the diffusion process, λ face represents the weight, λ rt and λ rv are hyperparameters that control the sparsity level for the corresponding features. In some examples, It can be obtained from the above formula (2). The diffusion model 350 can generate the output image 112 by optimizing the loss, for example, minimizing the loss value.

[0125] By combining the loss value of the target object with the loss value of the image reconstruction of the diffusion model, the generated output image can include more accurate image features of the target object. In this way, the generated image can be more in line with the user's expectations.

[0126] Example Training

[0127] Combination of the above Figures 4 to 5 Several examples of generating output images 112 using diffusion model 350 are described. In some embodiments, diffusion model 350 can be trained in a forward diffusion process. For example, diffusion model 350 can be continuously fine-tuned during image generation, such as in diffusion process 440.

[0128] In some embodiments, in the training step (or fine-tuning step) of the diffusion model 350, an input sample image and an input sample text including a sample object are obtained. Based on the input sample image and the input sample text, an output sample image associated with the input sample image and the input sample text is generated using the diffusion model 350. A first object feature representing the sample object is extracted from the input sample image. A second object feature representing the sample object is extracted from the output sample image. In some embodiments, the diffusion model 350 can be updated based on the difference between the first object feature and the second object feature. As an example, the second subunit obtained by fine-tuning in the attention unit 500 can be updated based on the difference between the first object feature and the second object feature, for example Figure 5 In addition, the fifth conversion unit 580 and the sixth conversion unit 590 may be updated based on the difference between the first object feature and the second object feature.

[0129] In this way, the diffusion model 350 can be continuously fine-tuned and updated, and the sub-units or conversion units in the attention unit can be updated to obtain a more accurate output image 112. As a result, the efficiency and accuracy of image generation can be improved, thereby further improving the user experience.

[0130] Example application scenario

[0131] Combination of the above Figures 2 to 5 Several examples of generating output images according to the image generation scheme of the present disclosure are described. In order to more clearly understand the embodiments of the present disclosure, an example application scenario is described below. The electronic device 110 obtains an input image, such as a static image input by a user, or a frame in a video. Alternatively, the electronic device 110 can also obtain an input image captured by an image acquisition device, for example, the electronic device 110 can obtain an input image taken by a user using a camera. The input image includes a target object, such as the face of a target person or the face of the user himself. In addition, the electronic device 110 obtains input text. Exemplarily, the input text can be input by the user, or acquired by a text acquisition device. The input text can describe the scene, environment, state or style of the desired output image, etc.

[0132] Then, through the above reference Figure 2 or Figure 3 The described example process may generate an output image of a target object in a scene, environment, or state described by an input text, or generate an output image of a target object in a style described by an input text.

[0133] It should be understood that if the forms (e.g., posture, expression, hairstyle, etc.) of the target objects in different input images are different, the forms of the target objects in the generated output images may also be different. By adopting this solution, various output images can be generated for different input images and according to different input texts.

[0134] The image generation scheme disclosed in the present invention is not only based on the image features of the input image, but also based on the text features associated with both the input text and the input image, so as to meet the various different scenes, environments, states, styles, etc. described by the input text. In addition, since in some embodiments, the text features are based on the features of the region of interest corresponding to the target object, such as the face, in the input image, the generated image can retain the features of the target object face, such as expression, posture, hairstyle, hair color, etc., so as to obtain a satisfactory output image. Such an image generation scheme can meet the different personalized needs of users, thereby improving the user experience.

[0135] Additionally, in some embodiments, the user can also set or adjust parameters in the image generation process, such as weights or other parameters for text attention features and visual attention features. In this way, the obtained output image can be made more in line with the user's expectations.

[0136] In addition, in some embodiments using diffusion models and attention mechanisms, units or subunits in the model can be continuously optimized, thereby continuously improving the efficiency and accuracy of the model. The disclosed solution can achieve fast, efficient and accurate image generation.

[0137] Example Process

[0138] Figure 6 FIG. 6 is a flowchart of a process 600 for image generation according to some embodiments of the present disclosure. The process 600 may be implemented in Figure 1 The electronic device 110 is located at the electronic device 110.

[0139] In block 610 , the electronic device 110 obtains a set of intermediate image features for characterizing a target object based on an input image including the target object.

[0140] In some embodiments, the electronic device 110 may determine a region of interest in the input image based on a position of the target object in the input image; extract the region of interest from the input image; obtain a processed image by removing a portion of the non-target object from the extracted region of interest; and generate a set of intermediate image features based on the processed image. As an example, the electronic device 110 may generate a set of intermediate image features based on the processed image using a predetermined number of network layers in an image encoder.

[0141] At block 620, the electronic device 110 determines a target image feature for characterizing the target object based on the set of intermediate image features. As an example, in an embodiment where a set of intermediate image features is generated using a predetermined number of network layers in an image encoder, the electronic device 110 may obtain output image features of an output layer of the image encoder. The electronic device 110 may then use an image feature mapper to convert the output image features into target image features.

[0142] At block 630, the electronic device 110 determines target text features for characterizing an image generation target based on the input text and the set of intermediate image features. In some embodiments, the electronic device 110 may use a text feature mapper to convert a set of intermediate image features into a first set of text feature representations; determine a second set of text feature representations based on at least a portion of the input text; and generate target text features based on the first set of text feature representations and the second set of text feature representations.

[0143] In some embodiments, the electronic device 110 may determine a first subtext representing the target object in the input text and a second subtext other than the first subtext in the input text. The electronic device 110 may further determine a second set of text feature representations based on the second subtext.

[0144] At block 640, the electronic device 110 generates an output image associated with the input image and the input text based on the target image features and the target text features. In some embodiments, the electronic device 110 may generate an output image from an initial noisy image using a diffusion model, conditioned on the target image features and the target text features.

[0145] In some embodiments, the initial noisy image includes at least one of the following: a white noise image, or the input image with noise applied, the applied noise being higher than a threshold noise level.

[0146] In some embodiments, generating an output image from an initial noisy image includes multiple denoising steps. A given denoising step among the multiple denoising steps includes: determining visual attention features by applying an attention mechanism to the image features to be denoised and the target image features in the given denoising step. The given denoising step also includes determining text attention features by applying an attention mechanism to the image features to be denoised and the target text features. The given denoising step also includes generating output image features of the given denoising step based on the visual attention features and the text attention features as input to the next denoising step of the given denoising step.

[0147] In some embodiments, the text attention feature can be determined by the following steps: using a first conversion unit to convert the target text feature into a first feature; using a second conversion unit to convert the target text feature into a second feature; using a third conversion unit to convert the image feature to be denoised into a third feature; and determining the text attention feature based on the first feature, the second feature and the third feature. In some embodiments, at least one of the first conversion unit, the second conversion unit and the third conversion unit includes a first subunit obtained by pre-training and a second subunit obtained by fine-tuning.

[0148] In some embodiments, the electronic device 110 can obtain the output image feature by weighting the visual attention feature and the text attention feature with the first weight and the second weight respectively. In some embodiments, during the training process of the diffusion model, the value of the first weight and the value of the second weight are determined according to random numbers.

[0149] In some embodiments, the training steps of the diffusion model include: obtaining an input sample image and an input sample text including a sample object; based on the input sample image and the input sample text, using the diffusion model, generating an output sample image associated with the input sample image and the input sample text; extracting a first object feature representing the sample object from the input sample image; extracting a second object feature representing the sample object from the output sample image; and updating the second subunit based on the difference between the first object feature and the second object feature.

[0150] Example devices and equipment

[0151] Figure 7 A schematic structural block diagram of an apparatus 700 for image generation according to some embodiments of the present disclosure is shown. The apparatus 700 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 700 may be implemented by hardware, software, firmware or any combination thereof.

[0152] As shown in the figure, the device 700 includes an intermediate feature acquisition module 710, which is configured to acquire a set of intermediate image features for characterizing the target object based on an input image including the target object. The device 700 also includes an image feature determination module 720, which is configured to determine target image features for characterizing the target object based on a set of intermediate image features. The device 700 also includes a text feature determination module 730, which is configured to determine target text features for characterizing the image generation target based on the input text and a set of intermediate image features. The device 700 also includes an image generation module 740, which is configured to generate an output image associated with the input image and the input text based on the target image features and the target text features.

[0153] In some embodiments, the intermediate feature acquisition module 710 is further configured to determine a region of interest in the input image based on the position of the target object in the input image; extract the region of interest from the input image; obtain a processed image by removing a portion of the non-target object from the extracted region of interest; and generate a set of intermediate image features based on the processed image. In some embodiments, the intermediate feature acquisition module 710 can be further configured to generate a set of intermediate image features based on the processed image using a predetermined number of network layers in the image encoder.

[0154] In some embodiments, the image feature determination module 720 is further configured to obtain output image features of an output layer of the image encoder; and convert the output image features into target image features using an image feature mapper.

[0155] In some embodiments, the text feature determination module 730 can be further configured to utilize a text feature mapper to convert a set of intermediate image features into a first set of text feature representations; determine a second set of text feature representations based on at least a portion of the input text; and generate target text features based on the first set of text feature representations and the second set of text feature representations.

[0156] In some embodiments, the text feature determination module 730 may be further configured to determine a first subtext in the input text representing the target object and a second subtext in the input text other than the first subtext; and determine a second set of text feature representations based on the second subtext.

[0157] In some embodiments, the image generation module 740 is further configured to generate an output image from an initial noisy image using a diffusion model, conditioned on a target image feature and a target text feature. In some embodiments, the initial noisy image includes at least one of the following: a white noise image, or the input image with noise applied, the applied noise being higher than a threshold noise level.

[0158] In some embodiments, generating an output image from an initial noisy image includes multiple denoising steps. A given denoising step among the multiple denoising steps includes: determining visual attention features by applying an attention mechanism to the image features to be denoised and the target image features in the given denoising step. The given denoising step also includes determining text attention features by applying an attention mechanism to the image features to be denoised and the target text features. The given denoising step also includes generating output image features of the given denoising step based on the visual attention features and the text attention features as input to the next denoising step of the given denoising step.

[0159] In some embodiments, the apparatus 700 may further include a text attention feature determination module configured to: convert the target text feature into a first feature using a first conversion unit; convert the target text feature into a second feature using a second conversion unit; convert the image feature to be denoised into a third feature using a third conversion unit; and determine the text attention feature based on the first feature, the second feature, and the third feature. In some embodiments, at least one of the first conversion unit, the second conversion unit, and the third conversion unit includes a first subunit obtained by pre-training and a second subunit obtained by fine-tuning.

[0160] In some embodiments, the apparatus 700 may further include an output image feature module configured to obtain an output image feature by weighting the visual attention feature and the text attention feature with a first weight and a second weight, respectively. In some embodiments, during the training process of the diffusion model, the value of the first weight and the value of the second weight are determined according to a random number.

[0161] In some embodiments, the device 700 may also include a diffusion model training module, which is configured to obtain an input sample image and an input sample text including a sample object; based on the input sample image and the input sample text, using a diffusion model, generate an output sample image associated with the input sample image and the input sample text; extract a first object feature representing the sample object from the input sample image; extract a second object feature representing the sample object from the output sample image; and update the second subunit based on the difference between the first object feature and the second object feature.

[0162] Figure 8 A block diagram of an electronic device 800 is shown in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 8 The electronic device 800 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 8 The electronic device 800 shown can be used to implement Figure 1 An electronic device 110.

[0163] like Figure 8 As shown, the electronic device 800 is in the form of a general electronic device. The components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

[0164] The electronic device 800 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 830 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 800.

[0165] The electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 8 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0166] The communication unit 840 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0167] The input device 850 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 860 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 800 may also communicate with one or more external devices (not shown) through the communication unit 840 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 800, or communicate with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0168] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0169] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0170] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0171] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0172] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0173] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating an image, comprising: Based on an input image including a target object, obtaining a set of intermediate image features for characterizing the target object; Determining target image features for characterizing the target object based on the set of intermediate image features; Determining target text features for characterizing an image generation target based on the input text and the set of intermediate image features; as well as Based on the target image feature and the target text feature, an output image associated with the input image and the input text is generated.

2. The method according to claim 1, wherein determining the target text feature comprises: Using a text feature mapper, converting the set of intermediate image features into a first set of text feature representations; Determining a second set of text feature representations based on at least a portion of the input text; as well as The target text features are generated based on the first set of text feature representations and the second set of text feature representations.

3. The method of claim 2, wherein determining the second set of text feature representations comprises: Determine a first subtext in the input text representing the target object and a second subtext in the input text except the first subtext; as well as Based on the second subtext, the second set of text feature representations is determined.

4. The method according to claim 1, wherein obtaining a set of intermediate image features for characterizing the target object comprises: Determining a region of interest in the input image based on a position of the target object in the input image; extracting the region of interest from the input image; Obtaining a processed image by removing a portion other than the target object from the extracted region of interest; as well as The set of intermediate image features is generated based on the processed image.

5. The method of claim 4, wherein generating the set of intermediate image features based on the processed image comprises: Based on the processed image, the set of intermediate image features is generated using a predetermined number of network layers in an image encoder.

6. The method according to claim 5, wherein determining the target image feature comprises: Obtaining output image features of an output layer of the image encoder; as well as The output image features are converted into the target image features using an image feature mapper.

7. The method of claim 1 , wherein generating the output image comprises: The output image is generated from an initial noisy image using a diffusion model and taking the target image features and the target text features as conditions.

8. The method according to claim 7, wherein the initial noisy image comprises at least one of the following: White noise image, or The input image is applied with noise, the applied noise being above a threshold noise level.

9. The method of claim 7, wherein generating the output image from the initial noisy image comprises a plurality of denoising steps, and a given denoising step of the plurality of denoising steps comprises: Determining visual attention features by applying an attention mechanism to the features of the image to be denoised in the given denoising step and the features of the target image; Determine text attention features by applying an attention mechanism to the features of the image to be denoised and the features of the target text; as well as Based on the visual attention features and the textual attention features, output image features of the given denoising step are generated as input of the next denoising step of the given denoising step.

10. The method of claim 9, wherein determining the text attention feature comprises: Using a first conversion unit to convert the target text feature into a first feature; Using a second conversion unit to convert the target text feature into a second feature; Using a third conversion unit, the image feature to be denoised is converted into a third feature; and Based on the first feature, the second feature and the third feature, the text attention feature is determined, wherein At least one of the first conversion unit, the second conversion unit, and the third conversion unit includes a first sub-unit obtained through pre-training and a second sub-unit obtained through fine-tuning.

11. The method of claim 9, wherein generating the output image features of the given denoising step comprises: The output image feature is obtained by weighting the visual attention feature and the text attention feature with a first weight and a second weight respectively. 12 . The method according to claim 10 , wherein in the training process of the diffusion model, the value of the first weight and the value of the second weight are determined according to random numbers.

13. The method according to claim 10, wherein the step of training the diffusion model comprises: Obtaining an input sample image and an input sample text including a sample object; Based on the input sample image and the input sample text, using the diffusion model, generating an output sample image associated with the input sample image and the input sample text; Extracting a first object feature representing the sample object from the input sample image; extracting a second object feature representing the sample object from the output sample image; as well as The second subunit is updated based on a difference between the first object feature and the second object feature.

14. A device for image generation, comprising: An intermediate feature acquisition module, configured to acquire a set of intermediate image features for characterizing the target object based on an input image including the target object; an image feature determination module, configured to determine a target image feature for characterizing the target object based on the set of intermediate image features; A text feature determination module, configured to determine a target text feature for characterizing an image generation target based on the input text and the set of intermediate image features; as well as The image generation module is configured to generate an output image associated with the input image and the input text based on the target image feature and the target text feature.

15. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 13 when executed by the at least one processing unit.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 13.

Images