Image generation method and apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

An image generation method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The method comprises: acquiring preset noise, and acquiring an initial object image and an image description text of a preset object, the image description text being used for describing a target attribute of the preset object; performing first encoding on the initial object image to obtain an image encoding feature; performing second encoding on the image description text to obtain a text encoding feature; and on the basis of the image encoding feature, the text encoding feature and the preset noise, using a diffusion model to generate a target object image, the target object image comprising the preset object corresponding to the target attribute.

Description

Image generation method and device, electronic device, computer-readable storage medium, and computer program product

[0001] Cross-reference to Related Applications

[0002] Embodiments of the present application are based on and claim priority from Chinese Patent Application No. 202411051690.3, filed on July 30, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of artificial intelligence, and in particular to an image generation method and device, an electronic device, a computer-readable storage medium, and a computer program product. BACKGROUND

[0004] With the development of artificial intelligence, image generation is also an important application field of artificial intelligence. In the related art, when image generation is performed based on a diffusion model, the generation of an image is controlled based on a text description, so that the quality of the generated image is not high. SUMMARY

[0005] Embodiments of the present application provide an image generation method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the image generation effect.

[0006] The technical solution of the embodiments of the present application is as follows:

[0007] The embodiments of the present application provide an image generation method applied to an electronic device, comprising:

[0008] obtaining a preset noise, and obtaining an initial object image and an image description text of a preset object, the image description text being used to describe a target attribute of the preset object;

[0009] performing first encoding on the initial object image to obtain image encoding features;

[0010] performing second encoding on the image description text to obtain text encoding features;

[0011] generating a target object image using a diffusion model based on the image encoding features, the text encoding features, and the preset noise, the target object image comprising the preset object corresponding to the target attribute.

[0012] The embodiments of the present application also provide an image generation device, comprising:

[0013] The acquisition module is configured to acquire preset noise, and acquire an initial object image of a preset object and image description text used for describing a target attribute of the preset object.

[0014] The encoding module is configured to perform first encoding on the initial object image to obtain image encoding features, and perform second encoding on the image description text to obtain text encoding features.

[0015] The generation module is configured to generate a target object image using a diffusion model based on the image encoding features, the text encoding features, and the preset noise, the target object image including the preset object corresponding to the target attribute.

[0016] Embodiments of the present application also provide an electronic device, comprising:

[0017] The memory is configured to store computer executable instructions.

[0018] The processor is configured to execute the computer executable instructions stored in the memory to implement the image generation method provided by the embodiments of the present application.

[0019] Embodiments of the present application also provide a computer readable storage medium storing computer executable instructions or computer programs, the computer executable instructions or computer programs being executed by a processor to implement the image generation method provided by the embodiments of the present application.

[0020] Embodiments of the present application also provide a computer program product comprising computer executable instructions or computer programs, the computer executable instructions or computer programs being executed by a processor to implement the image generation method provided by the embodiments of the present application.

[0021] Embodiments of the present application have the following beneficial effects:

[0022] The target attribute of the preset object is described by the image description text, the initial object image is first encoded to obtain image encoding features, the image description text is second encoded to obtain text encoding features, and a target object image is generated using a diffusion model based on the image encoding features, the text encoding features, and the preset noise. In this way, since the image encoding features retain the features of the image domain of the initial object image, the consistency of the generated target object image with the preset object in the initial object image is ensured, and the text encoding features guide the diffusion model to generate an image consistent with the description of the image description text by describing the target attribute. Therefore, by cooperation of the image encoding features and the text encoding features, the image generation process of the diffusion model is controlled, so that a target object image that is consistent with the description of the image description text and ensures the authenticity is generated, and the image generation effect for the object is improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Fig. 1 is a schematic diagram of an architecture of an image generation system according to an embodiment of the present application;

[0024] Fig. 2 is a schematic diagram of an electronic device according to an embodiment of the present application;

[0025] Fig. 3A is a first flowchart of an image generation method according to an embodiment of the present application;

[0026] Fig. 3B is a second flowchart of an image generation method according to an embodiment of the present application;

[0027] Fig. 3C is a third flowchart of an image generation method according to an embodiment of the present application;

[0028] Fig. 3D is a fourth flowchart of an image generation method according to an embodiment of the present application;

[0029] Fig. 4 is a schematic diagram of an image generation architecture according to an embodiment of the present application;

[0030] Fig. 5 is a schematic diagram of a U-shaped network according to an embodiment of the present application;

[0031] Fig. 6 is a schematic diagram of a cross-attention processing layer according to an embodiment of the present application;

[0032] Fig. 7 is a schematic diagram of a self-attention processing layer according to an embodiment of the present application;

[0033] Fig. 8 is a schematic diagram of training of an image generation model according to an embodiment of the present application;

[0034] Figs. 9A-9D are schematic diagrams of applications of an image generation method according to an embodiment of the present application.

[0035] It should be noted that the above-mentioned “first”, “second” are only used to distinguish different schemes, and do not represent the degree of superiority or priority in the implementation process. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by a person of ordinary skill in the art without making creative labor fall within the scope of protection of the present application.

[0037] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0038] In the following description, the terms "first\second\third" are merely to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that the "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0039] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by one skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0041] The relevant data collection process in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carry out subsequent data use and processing behavior.

[0042] Before further detailing the embodiments of the present application, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0043] 1) Client: an application program running in an electronic device for providing various services, such as a client supporting image generation.

[0044] 2) In response to: used to represent the conditions or states on which the operations performed depend, when the dependent conditions or states are met, one or more operations performed can be real-time or have a set delay; in the absence of special instructions, there is no restriction on the execution order of multiple operations performed.

[0045] 3) Diffusion Model: A generative model that first introduces noise into the diffusion model and then generates images by denoising. Through multiple iterations over a period of time, the diffusion model learns to generate new images each time given some noisy input. The diffusion model mentioned in the embodiments of the present application is similar to the image generation structure in the Stable Diffusion model. Here, the picture is converted into features, and the converted features are input into the image generation structure, so that the image generation structure has the ability to generate other similar pictures with the features of the picture.

[0046] 4) Image Encoding Features: Also known as image embedding (Image Embedding), which can be extracted by contrastive language-image pre-training (CLIP) model and similar models. Image Embedding can be understood as using a model to convert an image into a meaningful vector representation. This process extracts the visual features of the image, which helps to achieve more advanced image understanding and generation tasks.

[0047] 5) Text Encoding Features: Also known as text embedding (Text Embedding), which can also be extracted by CLIP and other models. Text Embedding can be understood as using a model to convert a piece of text into a meaningful vector representation. This process extracts the expression features of the text, which helps to achieve more advanced image understanding and generation tasks.

[0048] 6) Variational Autoencoders (VAE): A generative model that includes an encoder (Encoder) and a decoder (Decoder). The encoder is used to compress the input high-dimensional image data into a low-dimensional latent vector. This process can be regarded as the "encoding" of the image; the decoder performs the opposite operation, decoding the latent vector back to the high-dimensional image space, trying to restore the original image as much as possible. Among them, the Stable Diffusion model generates images based on the diffusion process, and the VAE in the Stable Diffusion model is used to improve the efficiency and quality of image generation. Specifically, VAE can map noise or text vectors in the latent space to a more easily processed form before the Stable Diffusion model generates images, or apply specific filter effects and other post-processing operations on the generated images.

[0049] The embodiment of the present application provides an image generation method and device, electronic equipment, computer readable storage medium and computer program product, which can improve the image generation effect. Next, based on the above description of the terms and terms involved in the embodiment of the present application, the embodiment of the present application is described in detail.

[0050] The image generation system provided by the embodiment of the present application is described below. Referring to FIG. 1, FIG. 1 is an architecture schematic diagram of the image generation system provided by the embodiment of the present application. In order to realize a supporting example application, the image generation system 100 includes a server 200, a network 300, and a terminal 400. Among them, the terminal 400 connects the server 200 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two, and uses wireless or wired links to realize data transmission.

[0051] In some embodiments, the terminal 400 (for example, running a client supporting image generation) sends an image generation request to the server 200 in response to an image generation instruction, the image generation request carries an object image used for image generation and an image description text used for describing an object in the target object image to be generated; the server 200 receives the image generation request sent by the terminal 400; in response to the image generation request, the initial object image and the image description text are obtained from the image generation request; the initial object image is first encoded to obtain image encoding features, and the image description text is second encoded to obtain text encoding features; based on the image encoding features, the text encoding features and the preset noise, a diffusion model is used to generate a target object image; the target object image is returned to the terminal 400; the terminal 400 receives the target object image returned by the server 200; and the target object image is displayed.

[0052] In some embodiments, the terminal 400 (for example, running a client supporting image generation) sends an image generation request to the server 200 in response to an image generation instruction, the image generation request carries an object image used for image generation and an image description text used for describing an object in the target object image to be generated; the server 200 receives the image generation request sent by the terminal 400; in response to the image generation request, the initial object image and the image description text are obtained from the image generation request; the initial object image is first encoded to obtain image encoding features, and the image description text is second encoded to obtain text encoding features; based on the image encoding features, the text encoding features and the preset noise, a diffusion model is used to generate a target object image; the target object image is returned to the terminal 400; the terminal 400 receives the target object image returned by the server 200; and the target object image is displayed.

[0053] The image generation method provided by the embodiment of the present application can be applied to various scenes requiring image generation, such as creative design scenes, game development scenes, etc., which are illustrated below.

[0054] 1) Creative design scenario, for example, the terminal receives an initial object image (such as an image including a future city) and image description text (such as "cyberpunk style future city") input by a user, initiates an image generation request to the server, and the server generates a high-definition cyberpunk style future city based on the image generation request, but the terminal provides the user with secondary editing or direct use.

[0055] 2) Game development, for example, the terminal sends an initial object image (such as an image including a volcano) and image description text (such as "medieval dragon knight and volcano scene") submitted by a designer to a cloud collaboration platform, the server of the collaboration platform generates a concept map of "medieval dragon knight and volcano scene", and synchronizes the concept map to the terminals of team members to support real-time feedback and iteration.

[0056] 3) Business and marketing, for example, the server generates multiple versions of "summer beach drink" pictures based on an initial object image (such as an image including a beverage can) and image description text (such as "summer beach drink") submitted by a user, and stores the pictures in the cloud. Marketing personnel browse the generated "summer beach drink" pictures on the cloud webpage through the terminal, select the pictures, and download them to the local for distribution.

[0057] The image generation method provided by the embodiments of the present application is implemented by an electronic device, for example, it can be implemented by a terminal alone, or by a server alone, or by a terminal and a server cooperatively. The electronic device can be various types of terminals or servers. As an example, the server (for example, server 200) can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal (for example, terminal 400) can be a notebook computer, a tablet computer, a desktop computer, a smart phone, a smart voice interaction device (such as a smart speaker), a smart home appliance (such as a smart TV), a smart watch, a vehicle-mounted terminal, a wearable device, a virtual reality (VR) device, a flying device, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and the embodiments of the present application do not limit this.

[0058] In some embodiments, a terminal or a server can implement the image generation method provided by the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer programs can be native programs in an operating system or software modules; can be native applications (APPs), i.e., programs that need to be installed in an operating system to run, such as APPs supporting image generation; or can be applets that can be embedded into any APP, i.e., programs that only need to be downloaded into a browser environment to run. In summary, the above computer-executable instructions can be any form of instructions, and the above computer programs can be any form of application programs, modules or plug-ins.

[0059] The electronic device implementing the image generation method provided by the embodiments of the present application is described below. Referring to FIG. 2, FIG. 2 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. The electronic device 500 provided by the embodiments of the present application can be a terminal or a server. As shown in FIG. 2, the electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between the components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all the buses are marked as the bus system 540 in FIG. 2.

[0060] The processor 510 can be an integrated circuit chip having a signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0061] The user interface 530 includes one or more output devices 531 enabling presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0062] The memory 550 can be removable, non-removable, or a combination thereof. The memory 550 can include one or more storage devices physically located in proximity to the processor 510. The memory 550 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. Non-volatile memory can be read only memory (ROM), and volatile memory can be random access memory (RAM). The memory 550 described in embodiments of the present application is intended to include any suitable type of memory.

[0063] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are exemplarily illustrated below.

[0064] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks.

[0065] The network communication module 552 is used to communicate with other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), etc.

[0066] The presentation module 553 is used to enable the presentation of information via one or more output devices 531 associated with the user interface 530 (e.g., a display screen, a speaker, etc.), such as a user interface for operating peripheral devices and displaying content and information.

[0067] The input processing module 554 is used to detect and interpret one or more user inputs or interactions from one or more input devices 532.

[0068] In some embodiments, the image generation apparatus provided by the embodiments of the present application can be implemented in a software manner, and FIG. 2 shows an image generation apparatus 555 stored in the memory 550, which can be software in the form of programs and plug-ins, etc., including the following software modules: an acquisition module 5551, an encoding module 5552, and a generation module 5553. These modules are logical, and thus can be combined or further split according to the functions to be implemented, and the functions of each module will be described below.

[0069] The image generation method provided by the embodiments of the present application is described below. As described above, the image generation method provided by the embodiments of the present application is implemented by an electronic device, for example, can be implemented by a server or a terminal alone, or by a server and a terminal cooperatively. Therefore, the execution subject of each step will not be repeated below. Referring to FIG. 3A, FIG. 3A is a flowchart of the image generation method provided by the embodiments of the present application, and the image generation method provided by the embodiments of the present application includes the following steps.

[0070] In step 101, a preset noise is obtained, and an initial object image and an image description text of a preset object are obtained.

[0071] The image description text is used to describe the target attribute of the preset object, that is, to describe the preset object in the target object image to be generated. For example, the image description text is “cyberpunk style future city”, and the image description text describes that the initial object in the target object image to be generated is “cyberpunk style future city”; the image description text is “summer beach drink”, and the image description text describes that the preset object in the target object image to be generated is “summer beach drink”.

[0072] Here, the preset noise can be generated. For example, the preset noise can be obtained by sampling from a target distribution; the target distribution includes a plurality of random data conforming to a specific data distribution type (such as normal distribution (Gaussian distribution), standard normal distribution, uniform distribution), and the random data conforming to the specific data distribution type can be generated based on a random data generation algorithm. For example, the preset noise can be Gaussian noise, Poisson noise, salt and pepper noise, Gaussian white noise, etc. Taking the preset noise as Gaussian noise as an example, random data conforming to Gaussian distribution can be generated by a random data generation algorithm, and then a target random number conforming to Gaussian distribution can be obtained by sampling from the random data conforming to Gaussian distribution, and the target random number conforming to Gaussian distribution is taken as the preset noise.

[0073] Here, the embodiments of the present application also obtain an initial object image, which is used for image generation and can be understood as a reference image for generating a target object image. The object image includes a preset object, which can be a virtual object, such as a virtual person, a virtual animal, a virtual object, a virtual animation character, a virtual game character, etc., or a real object, such as a real person or object. The preset object can be a two-dimensional object or a three-dimensional object.

[0074] In some embodiments, the initial object image can be obtained by performing the following steps: obtaining a candidate object image including the preset object; determining a target region in the candidate object image different from the region where the object is located; and covering the target region in the candidate object image to obtain the initial object image.

[0075] For example, object region recognition can be performed on the candidate object image to determine the region where the object in the candidate object image is located, and then a target region different from the region where the preset object is located in the candidate object image is determined, and finally the target region in the candidate object image is covered to obtain the initial object image. In this way, the interference of useless information in the initial object image can be reduced, thereby improving the image generation effect of generating the target object image (including the preset object) based on the initial object image.

[0076] Here, the image description text is also used for the generation of the target object image, and like the initial object image, the image description text is also reference information for generating the target object image. The target object image includes the preset object. The image description text is used to describe the target attributes of the preset object, for example, the image description text can describe the target attributes of the background where the preset object is located, etc., that is, the image description text can describe any target attribute related to the preset object but not the preset object itself. For example, the image description text can be: a person (i.e., the preset object) is sitting in a coffee shop at noon; a person (i.e., the preset object) walks onto the balcony when the morning light illuminates the room; a person walks along the riverbank in the golden sunset, etc.; for example, the image description text can describe the clothing, posture, action, accessories, etc. of the preset object, that is, the image description text can describe any target attribute of the preset object itself. For example, the image description text can be: a person (i.e., the preset object) is dancing; a person (i.e., the preset object) is wearing a red coat, etc. In actual applications, the image description text can be set by the user according to the needs, or it can be automatically generated by a text generation model (such as a pre-trained large model) according to the user's needs.

[0077] Step 102: performing first encoding on the initial object image to obtain image encoding features, and performing second encoding on the image description text to obtain text encoding features.

[0078] The image encoding features are low-dimensional dense vector representations obtained by feature extraction and compression of the initial object image input by a neural network (i.e., features in the image domain), which retains the key visual information (such as shape, texture, spatial structure, etc.) of the initial object image while filtering out redundant details, providing visual condition input for subsequent image generation; the text encoding features are vector representations obtained by semantic encoding of the image description text by a natural language processing model (i.e., features in the text domain), which captures high-level semantic information (such as object categories, attributes, relationships, etc.) of the image description text, used to guide the generation process to maintain semantic consistency with the image description text. These two kinds of encoding features together constitute the basis of multi-modal conditional generation, and through cross-modal interaction, fine control of the preset noise is achieved.

[0079] Here, the initial object image is image encoded to obtain an image encoding feature, which can be implemented by a visual encoder. At the same time, the image description text can also be text encoded to obtain a text encoding feature, which can be implemented by a text encoder.

[0080] Step 103: generating a target object image based on the image encoding feature, the text encoding feature, and the preset noise using a diffusion model, the target object image including a preset object corresponding to a target attribute.

[0081] Among them, since the image encoding feature retains the features of the image domain of the initial object image, it ensures the consistency of the generated target object image with the preset object in the initial object image, and the text encoding feature guides the diffusion model to generate an image consistent with the image description text description by describing the target attribute, so by the cooperation of the image encoding feature and the text encoding feature, these multi-modal information, the image generation process of the diffusion model is controlled, so as to generate a target object image that conforms to the image description text description and ensures the authenticity, and improve the image generation effect for the object.

[0082] In some embodiments, step 103 can be implemented in the following way: extracting an object feature of the preset object from the initial object image; generating a target object image based on the object feature, the image encoding feature, the text encoding feature, and the preset noise using a diffusion model.

[0083] The object feature refers to visual information with distinguishing and representative characteristics extracted from a preset object (such as an object, a person, a scene, etc.) in an initial object image, and is usually focused on semantic or structural features of the preset object itself, including but not limited to the following dimensions: morphological features (such as contour, shape, structural relationship), surface features (such as color, texture, material), semantic features (such as category attribute, component composition, functional meaning), spatial features (such as position, scale, orientation). The features of these dimensions are encoded into numerical vectors or high-level representations through computer vision techniques (such as convolutional neural networks, attention mechanisms, etc.), and the core purpose is to decouple and quantify the essential visual characteristics of the preset object, so that it has distinguishability and interpretability in the feature space. For example, the features extracted from an image of a cat may include "pointy ears", "hair texture", "crouching posture", and other key information. The image encoding feature is an embedding vector obtained by learning the overall representation of the initial object image through a neural network. The image encoding feature not only integrates the core information of the object feature, but also contains context information such as image background, texture, and lighting, forming a more comprehensive and multi-level feature representation suitable for subsequent image generation tasks. In short, the object feature is a basic component of the image encoding feature, and the image encoding feature is a deep integration and abstract representation of the object feature and its associated visual information.

[0084] In some embodiments, the image encoding feature can include the object feature.

[0085] Here, the object feature of the preset object is extracted, which can be extracted by a pre-trained feature extraction model. The object feature extraction network provided by the embodiments of the present application can be constructed based on a Unet network, which includes a down-sampling network and an up-sampling network. Based on this, in some embodiments, the object feature includes a first object feature and a second object feature, and the object feature of the object in the object image can be extracted by performing the following steps: performing third encoding on the initial object image to obtain an object encoding feature; based on the object encoding feature, performing down-sampling on the initial object image to obtain the first object feature; and based on the object encoding feature, performing up-sampling on the first object feature to obtain the second object feature.

[0086] Downsampling is a way of compressing the object encoding feature by reducing the data resolution or dimension. Pooling or strided convolution can be used to reduce the amount of calculation while retaining key information, resulting in a first object feature with higher semantic representation. Upsampling is a technique that restores or expands the spatial dimension of the low-resolution first object feature, combines the detailed information of the object encoding feature, and generates a second object feature with higher resolution and more detailed information. In summary, downsampling focuses on information condensation (such as reducing spatial size and increasing channel number), and extracting abstract features; upsampling focuses on information reconstruction (such as increasing spatial size and reducing channel number), and restoring detailed structure.

[0087] For example, the initial object image is first third encoded to obtain the object encoding feature, and the third encoding process can be implemented by a visual encoder. Then, based on the object encoding feature, the initial object image is downsampled to obtain the first object feature, i.e., the downsample process of the initial object image is controlled based on the object encoding feature to improve the downsample effect of the initial object image. Finally, based on the object encoding feature, the first object feature is upsampled to obtain the second object feature, i.e., the upsample process of the first object feature is controlled based on the object encoding feature to improve the upsample effect of the object image, thereby improving the expression ability of the extracted object feature for the object in the object image, which is beneficial to the subsequent image generation process based on the object feature.

[0088] In this way, the embodiments of the present application introduce a multi-scale object feature extraction mechanism (downsampling to generate the first object feature and upsampling to generate the second object feature), which captures the high-level semantic information (such as overall structure and category attribute) of the initial object image through downsampling, combines the local detailed features (such as texture and edge) recovered by upsampling, forms complementary multi-scale representation, enhances the structural rationality and detail richness of the generated image, and significantly improves the precision and controllability of image generation.

[0089] In some embodiments, based on object encoding features, the initial object image can be downsampled to obtain a first object feature by performing the following steps: performing a first convolution on the initial object image to obtain a first convolution feature; performing self-attention processing on the first convolution feature to obtain a first processing result; and performing cross-attention processing on the object encoding feature and the first processing result to obtain the second object feature. Correspondingly, based on the object encoding features, the first object feature can be upsampled to obtain a second object feature by performing the following steps: performing a second convolution on the first object feature to obtain a second convolution feature; performing self-attention processing on the second convolution feature to obtain a second processing result; and performing cross-attention processing on the object encoding feature and the second processing result to obtain the second object feature.

[0090] Self-attention processing refers to modeling the dependencies between elements within input features (such as first or second convolutional features). By calculating the similarity weights between each position in the input feature and other positions, it dynamically aggregates global contextual information, thereby enhancing the representational power of the output features. For example, in image processing, self-attention can capture long-distance spatial or channel associations, making the model focus more on salient regions relevant to the task (such as image generation). Cross-attention processing, on the other hand, establishes interaction relationships between different feature modalities or levels (such as object-encoded features and the first processing result, or object-encoded features and the second processing result), guiding information fusion through a query-key-value mechanism. For example, during downsampling or upsampling, object-encoded features are used as queries to perform key-value retrieval on the self-attention processing results (such as the first and second processing results), thereby dynamically aligning high-level semantic information with local detail features to generate more discriminative output features (such as the first and second object features). In summary, self-attention processing focuses on modeling global relationships within a single feature, while cross-attention processing emphasizes cross-modal or cross-level information interaction between multiple features.

[0091] For example, the down-sampling network includes a first convolutional network (for down-sampling processing), a self-attention (Self Attention) processing layer, and a cross-attention (Cross Attention) processing layer, based on which the down-sampling process includes: first performing first convolutional processing on the initial object image to obtain first convolutional features; then performing self-attention processing on the first convolutional features to obtain a first processing result; and finally performing cross-attention processing on the object encoding features and the first processing result to obtain first object features. In this way, the cross-attention processing layer of the down-sampling network increases the control of the object encoding features of the initial object image, which can improve the down-sampling effect on the initial object image.

[0092] Similarly, the up-sampling network includes a second convolutional network (for up-sampling processing), a self-attention (Self Attention) processing layer, and a cross-attention (Cross Attention) processing layer, based on which the up-sampling process includes: first performing second convolutional processing on the first object features to obtain second convolutional features; then performing self-attention processing on the second convolutional features to obtain a second processing result; and finally performing cross-attention processing on the object encoding features and the second processing result to obtain second object features. In this way, the cross-attention processing layer of the up-sampling network also increases the control of the object encoding features of the initial object image, which can improve the up-sampling effect on the initial object image, and further improve the expression ability of the extracted object features for the objects in the initial object image, which is beneficial for the subsequent image generation process based on the object features.

[0093] In this way, the embodiment of the present application constructs a hierarchical attention feature pyramid, which provides a more robust multi-granularity conditional input for the generation process of the target object image. After capturing local features through convolutional processing, the long-range dependency relationship (such as global structure or texture association) is modeled by combining the self-attention mechanism, and then the dynamic fusion of object encoding features and processing results is realized through cross-attention, while considering the consistency of local details and global semantics, to realize fine feature extraction; in the up-sampling and down-sampling processes, the object encoding features are used as a unified medium to bridge the image, text, and noise feature spaces through cross-attention, to ensure that the generation result strictly follows the text description and preserves the key attributes (such as style and pose) of the initial object image, to realize cross-modal feature alignment; the phased down-sampling-up-sampling structure (combined with the attention mechanism) effectively alleviates the information loss problem in traditional single-scale feature extraction, especially in complex scenes, which can more stably generate high-resolution and detailed target object images.

[0094] In some embodiments, the downsampling process is implemented by M layers of downsampling layers, M being an integer greater than 1; based on this, based on the object coding feature, the object image can be downsampled to obtain the first object feature by performing the following steps: based on the object coding feature, the object image is downsampled by the first layer of the M layers of downsampling layers to obtain the layer downsampling result of the first layer of the M layers of downsampling layers; based on the object coding feature, the layer downsampling result of the i-1 layer of the M layers of downsampling layers is downsampled by the i layer of the M layers of downsampling layers to obtain the layer downsampling result of the i layer of the M layers of downsampling layers; i is iterated to obtain the layer downsampling result of the M layer of the M layers of downsampling layers, i being an integer greater than 1 and i being less than or equal to M; and the layer downsampling result of each layer of the M layers of downsampling layers is taken as the first object feature.

[0095] Here, the downsampling network includes M layers of downsampling layers, the downsampling process being implemented by the M layers of downsampling layers, each layer of the downsampling layers including a first convolutional network (for downsampling processing), a self-attention (Self Attention) processing layer and a cross-attention (Cross Attention) processing layer.

[0096] Based on this, first, based on the object coding feature, the object image is downsampled by the first layer of the M layers of downsampling layers to obtain the layer downsampling result of the first layer of the M layers of downsampling layers, wherein the layer downsampling process of the first layer of the M layers of downsampling layers can refer to the above-mentioned downsampling process, and the implementation logic is the same. For the i layer of the M layers of downsampling layers, based on the object coding feature, the layer downsampling result of the i-1 layer of the M layers of downsampling layers is downsampled to obtain the layer downsampling result of the i layer of the M layers of downsampling layers, wherein the layer downsampling process of the i layer of the M layers of downsampling layers can refer to the above-mentioned downsampling process, and the implementation logic is also the same. By iterating i, the layer downsampling result of the M layer of the M layers of downsampling layers can be obtained. In this way, the layer downsampling result of each layer of the M layers of downsampling layers is taken as the above-mentioned first object feature. It should be noted that i can start from 1 and iterate to M, and i is incremented by 1 each time during the iteration.

[0097] In this way, based on the object coding feature, the object image is subjected to M times of cascaded downsampling processing by the M layers of downsampling layers, M first object features are obtained, the downsampling effect for the object image is improved, more effective features can be extracted, and the expression ability of the finally obtained object feature is improved.

[0098] In some embodiments, the up-sampling process is implemented by M layers of up-sampling layers, the i-th layer of the M layers of up-sampling layers corresponds to the i-th layer of the M layers of down-sampling layers; based on this, based on the object encoding feature, the first object feature can be up-sampled to obtain the second object feature by performing the following steps: through the M-th layer of the M layers of up-sampling layers, based on the object encoding feature, the first object feature is up-sampled to obtain the layer up-sampling result of the M-th layer of up-sampling layers; the layer down-sampling result of the i-th layer of down-sampling layers and the layer up-sampling result of the i+1-th layer of up-sampling layers are spliced to obtain the splicing result of the i-th layer of up-sampling layers, and the splicing result of the i-th layer of up-sampling layers is up-sampled based on the object encoding feature through the i-th layer of up-sampling layers to obtain the layer up-sampling result of the i-th layer of up-sampling layers; traverse i to obtain the layer up-sampling result of the first layer of up-sampling layers; the layer up-sampling result of each layer of up-sampling layers is taken as the second object feature.

[0099] Here, the up-sampling network includes M layers of up-sampling layers, each up-sampling layer includes a second convolutional network (for up-sampling processing), a self-attention (Self Attention) processing layer, and a cross-attention (Cross Attention) processing layer. The i-th layer of the M layers of up-sampling layers corresponds to the i-th layer of the M layers of down-sampling layers, and the first layer of up-sampling layers corresponds to the first layer of down-sampling layers, that is: the second layer of up-sampling layers corresponds to the second layer of down-sampling layers, the third layer of up-sampling layers corresponds to the third layer of down-sampling layers,..., the M-1-th layer of up-sampling layers corresponds to the M-1-th layer of down-sampling layers, and the M-th layer of up-sampling layers corresponds to the M-th layer of down-sampling layers.

[0100] Based on this, first, through the M-th layer of the M layers of up-sampling layers, based on the object encoding feature, the first object feature is up-sampled to obtain the layer up-sampling result of the M-th layer of up-sampling layers, wherein the layer up-sampling process of the M-th layer of up-sampling layers can refer to the above up-sampling process, and the implementation logic is the same. For the i-th layer of up-sampling layers, first, the layer down-sampling result of the i-th layer of down-sampling layers is obtained, then the layer down-sampling result of the i-th layer of down-sampling layers and the layer up-sampling result of the i+1-th layer of up-sampling layers are spliced to obtain the splicing result of the i-th layer of up-sampling layers, and the splicing process can include adding, multiplying, etc. processing of the two; further, based on the object encoding feature, the splicing result of the i-th layer of up-sampling layers is up-sampled through the i-th layer of up-sampling layers to obtain the layer up-sampling result of the i-th layer of up-sampling layers, wherein the layer up-sampling process of the i-th layer of up-sampling layers can refer to the above up-sampling process, and the implementation logic is the same. By traversing i, the layer up-sampling result of the first layer of up-sampling layers can be obtained. In this way, the layer up-sampling result of each layer of up-sampling layers is taken as the above-mentioned second object feature. It should be noted that i here can start from M and traverse to 1, and i decreases by 1 each time during the traversal.

[0101] In this way, the first object feature is subjected to M times of cascaded up-sampling processing by the M up-sampling layers in combination with the object encoded feature, M second object features are obtained, the up-sampling effect for the first object feature is improved, more effective features can be extracted, and the expression capability of the finally obtained object feature is improved.

[0102] Here, the diffusion model is used to generate the target object image based on the object feature, the image encoded feature, the text encoded feature, and the preset noise. In this way, the generation of the target object image including the object is controlled through the cooperation of the object feature, the image encoded feature, and the text encoded feature, it is ensured that the target object image has the object feature, the image encoded feature, and the text encoded feature, and the image generation effect for the preset object is improved.

[0103] In some embodiments, the generation of the target object image based on the object feature, the image encoded feature, the text encoded feature, and the preset noise using the diffusion model can be implemented in the following manner: using the diffusion model to perform the following processing: denoising the preset noise based on the object feature, the image encoded feature, and the text encoded feature to obtain denoised features; and image reconstruction of the denoised features to obtain the target object image.

[0104] wherein, denoising refers to a process of extracting effective features by gradually eliminating random interference components in the preset noise based on multi-modal conditional information such as the object feature, the image encoded feature, and the text encoded feature; and image reconstruction is a process of obtaining a target object image that is consistent with the description of the image description text and has visual rationality by decoding the denoised features. The two stages together constitute the core process of conditional image generation, wherein denoising is used for feature purification, and image reconstruction is used for pixel-level synthesis, and finally end-to-end generation from noise to high-quality image is realized.

[0105] Here, the process of image generation actually includes the denoising process of the preset noise. That is, the preset noise is denoised based on the object feature, the image encoded feature, and the text encoded feature to obtain denoised features; and then the denoised features are subjected to image reconstruction to obtain the target object image. In actual application, the process of image reconstruction can be realized by decoding, that is, the denoised features are subjected to decoding processing to obtain the target object image. For example, the denoised features can be subjected to decoding processing by the decoder of the VAE model to obtain the target object image.

[0106] Thus, the embodiments of the present application significantly improve the quality and controllability of the generation effect by explicitly dividing the image generation process into two stages of denoising and image reconstruction. The beneficial effects mainly include: 1) first gradually remove noise by guiding through multi-modal conditions (object features, image encoding features, and text encoding features), retain semantic consistency, and then restore details through the reconstruction stage, making the generation process more stable and achieving stage-by-stage optimization; 2) the denoising stage can targetly fuse different modal features to accurately control the generated content, while the reconstruction stage focuses on improving visual quality, and the two stages have clear division of labor.

[0107] In some embodiments, the denoising process includes T times of denoising processing, T being an integer greater than 1; based on this, based on the object features, the image encoding features, and the text encoding features, the preset noise can be denoised by performing the following steps to obtain the denoised features: based on the object features, the image encoding features, and the text encoding features, the preset noise is denoised for the first time to obtain the intermediate denoised features of the first denoising output; based on the object features, the image encoding features, and the text encoding features, the intermediate denoised features of the t-1th denoising output are denoised for the tth time to obtain the intermediate denoised features of the tth denoising output, t being an integer greater than 1 and t being less than or equal to T; traverse t to obtain the intermediate denoised features of the Tth denoising output, and take the intermediate denoised features of the Tth denoising output as the denoised features. It should be noted that t here can start from 1 and end at T, and t is incremented by 1 in the traversal process.

[0108] It should be noted that the denoising process is a reverse diffusion process in the diffusion model processing, that is, the preset noise is denoised for T times to gradually remove noise and obtain denoised features. The T times of denoising can be understood as T time steps of denoising, and each time step completes one denoising. The process of each denoising can be implemented by using the denoising logic provided by the embodiments of the present application. In this way, the denoising effect of the preset noise can be improved to generate the target object image for the object and improve the image generation effect for the object.

[0109] Thus, the embodiments of the present application significantly improve the precision and stability of image generation through the progressive multi-round denoising mechanism. The beneficial effects mainly include: 1) gradually refine the features through T times of iterative denoising (T>1), so that the generation process can eliminate noise and retain key semantic information at different levels, avoiding the problem of detail loss caused by single denoising; 2) in each denoising iteration, the object features, the image encoding features, and the text encoding features (i.e., multi-modal conditions) are recombined to ensure that the multi-modal conditions can effectively guide the generation direction under different noise intensities; 3) the step-by-step intermediate denoised feature output provides a debuggable intermediate state for the generation process, which not only facilitates the viewing of generation quality, but also reduces the error accumulation risk of single denoising, especially suitable for stable generation of high-resolution images.

[0110] In some embodiments, based on the object feature, the image encoding feature and the text encoding feature, the preset noise can be denoised to obtain a denoised feature by performing the following steps: based on the object feature, the image encoding feature and the text encoding feature, downsampling the preset noise to obtain a noise downsampling result; based on the object feature, the image encoding feature and the text encoding feature, upsampling the noise downsampling result to obtain the denoised feature.

[0111] In the downsampling of the preset noise, the principle of downsampling is similar to the above-mentioned principle of downsampling, and in the upsampling of the noise downsampling result, the principle of upsampling is similar to the above-mentioned principle of upsampling, which will not be repeated here.

[0112] Here, the denoising process can be realized by an image generation network (Pretrained Unet), which can also be constructed based on the Unet network. The image generation network also includes a downsampling network and an upsampling network. Based on this, based on the object feature, the image encoding feature and the text encoding feature, the preset noise is downsampled to obtain a noise downsampling result, that is, the object feature, the image encoding feature and the text encoding feature control the downsampling process of the preset noise, and improve the downsampling effect of the preset noise. Finally, based on the object feature, the image encoding feature and the text encoding feature, the noise downsampling result is upsampled to obtain the denoised feature, that is, the object feature, the image encoding feature and the text encoding feature control the upsampling process of the noise downsampling result, and improve the upsampling effect of the noise downsampling result, thereby improving the accuracy of the obtained denoised feature and improving the image effect of the target object image reconstructed based on the denoised feature.

[0113] In this way, the embodiments of the present application significantly improve the quality and efficiency of image generation by constructing a multi-scale hierarchical denoising architecture, and the beneficial effects mainly include: 1) separating noise and effective features in a low-dimensional space through downsampling operation, retaining key semantic information while filtering noise; 2) feature processing in the reduced dimension space greatly reduces the computational complexity, and the subsequent upsampling process accurately restores the detailed features through cross-layer connection, achieving a balance between computing resources and generation quality; 3) cross-modal fusion enhancement: object features, image encoding and text encoding features are fused in the downsampling stage and the upsampling stage, ensuring that multi-modal conditional information can effectively guide the denoising direction in different scale feature spaces, and finally generating high-quality images that conform to the description of the image description text and ensure visual coherence.

[0114] In some embodiments, referring to FIG. 3B, based on the object feature, the image encoding feature and the text encoding feature, the preset noise can be down-sampled to obtain a noise down-sampling result by performing the following steps: step 201, performing first convolution processing on the preset noise to obtain a first noise feature; step 202, splicing the first noise feature and the object feature to obtain a first splicing result, and performing self-attention processing on the first splicing result to obtain a third processing result; step 203, performing cross-attention processing on the image encoding feature and the third processing result to obtain a first image processing result, and performing cross-attention processing on the text encoding feature and the third processing result to obtain a first text processing result; and step 204, determining the noise down-sampling result based on the first image processing result and the first text processing result.

[0115] The principle of the self-attention processing of step 202 is similar to the principle of the self-attention processing described above, and the principle of the cross-attention processing of step 203 is similar to the principle of the cross-attention processing described above, which will not be described here.

[0116] Here, the down-sampling network includes M layers of down-sampling layers, and each down-sampling layer includes a first convolution network (for down-sampling processing), a self-attention processing layer (Self Attention) and a cross-attention processing layer (Cross Attention). Based on this, the down-sampling process of the preset noise includes:

[0117] For step 201, a first convolution network can be used to perform first convolution processing on the preset noise to obtain a first noise feature. The first convolution network is used to perform down-sampling processing on the preset noise.

[0118] Here, the object feature described above can be added to each self-attention processing layer of the image generation network. Since the object feature extraction network also includes a down-sampling network and an up-sampling network, the first object feature output by the down-sampling network of the object feature extraction network can be added to the self-attention processing layer of the down-sampling network of the image generation network. That is, for step 202, first, the first noise feature and the object feature (such as the first object feature) are spliced to obtain a first splicing result, and then the first splicing result is subjected to self-attention processing to obtain a third processing result. In actual applications, the down-sampling network of the object feature extraction network and the down-sampling network of the image generation network can be the same, and the first object feature output by each layer of the down-sampling network of the object feature extraction network can be input to the corresponding layer of the down-sampling network of the image generation network for splicing.

[0119] For step 203, cross-attention processing is performed on the image coding features and the third processing result to obtain a first image processing result, and cross-attention processing is performed on the text coding features and the third processing result to obtain a first text processing result. For step 204, the first image processing result and the first text processing result can be fused to obtain a noise down-sampling result, and the fusion process can be addition, multiplication, splicing, etc.

[0120] In this way, the embodiments of the present application increase the control of object features, image coding features and text coding features in the denoising process, and can improve the generation effect of the target object image for the object. By splicing the noise features and the object features in step 202 and then performing self-attention processing, the association between the noise space and the object features is effectively established, and the extraction of key features is strengthened. By using image coding features and text coding features to interact with the processing result respectively in step 203, the visual details and semantic consistency are ensured in the down-sampling process. By dynamically adjusting the contribution weight of different modal features through the double cross-attention mechanism, the accurate separation of noise and effective features is realized, which provides a purer low-dimensional feature representation for subsequent up-sampling, and finally improves the structural accuracy and detail fidelity of the generated image.

[0121] In some embodiments, referring to FIG. 3C, based on the object features, the image coding features and the text coding features, the noise down-sampling result can be up-sampled to obtain denoising features by performing the following steps: step 301, performing second convolution processing on the noise down-sampling result to obtain second noise features; step 302, splicing the second noise features and the object features to obtain a second splicing result, and performing self-attention processing on the second splicing result to obtain a fourth processing result; step 303, performing cross-attention processing on the image coding features and the fourth processing result to obtain a second image processing result, and performing cross-attention processing on the text coding features and the fourth processing result to obtain a second text processing result; step 304, determining the denoising features based on the second image processing result and the second text processing result.

[0122] The principle of the self-attention processing of step 302 is similar to the principle of the above-mentioned self-attention processing, and the principle of the cross-attention processing of step 303 is similar to the principle of the above-mentioned cross-attention processing, which will not be described here.

[0123] Here, the up-sampling network includes M up-sampling layers, and each up-sampling layer includes a second convolution network (for up-sampling processing), a self-attention processing layer (Self Attention) and a cross-attention processing layer (Cross Attention). Based on this, the up-sampling process for the noise down-sampling result includes:

[0124] For step 301, a second convolutional network can be used to perform a second convolutional processing on the noise down-sampling result to obtain a second noise feature, and the second convolutional network is used to perform an up-sampling processing on the noise down-sampling result.

[0125] Here, the object feature mentioned above can be added in each self-attention processing layer of the image generation network. Since the object feature extraction network also includes a down-sampling network and an up-sampling network, the second object feature output by the up-sampling network of the object feature extraction network can be added in the self-attention processing layer of the up-sampling network of the image generation network. That is, for step 302, first, the second noise feature and the object feature (such as the second object feature) are spliced to obtain a second splicing result, and the second splicing result is subjected to self-attention processing to obtain a fourth processing result. In actual application, the up-sampling network of the object feature extraction network and the up-sampling network of the image generation network can be the same, and the second object feature output by each up-sampling layer in the up-sampling network of the object feature extraction network can be input into the corresponding up-sampling layer in the up-sampling network of the image generation network for splicing.

[0126] For step 303, cross-attention processing is performed on the image encoding feature and the fourth processing result to obtain a second image processing result, and cross-attention processing is performed on the text encoding feature and the fourth processing result to obtain a second text processing result. For step 304, the second image processing result and the second text processing result can be fused to obtain a de-noising feature, and the fusion process can be adding, multiplying, splicing, etc.

[0127] In this way, the control of the object feature, the image encoding feature and the text encoding feature is added in the de-noising process, which can improve the generation effect of the target object image of the object. Through step 302, the noise feature in the up-sampling process is spliced with the object feature, and through step 303, cross-attention interaction is performed with the image encoding feature and the text encoding feature respectively, realizing multi-level fusion of visual features and semantic features; through the self-attention in step 302, the long-range dependency relationship between features is effectively modeled, and through the cross-attention in step 303, it is ensured that the up-sampling process strictly follows the description of the image description text and maintains consistency with the input image; through the double attention mechanism (self-attention and cross-attention), the feature reconstruction process is finely regulated, so that the final de-noising feature not only retains key details, but also meets the expected semantic requirements, providing high-quality input for subsequent image reconstruction.

[0128] In some embodiments, the following steps can also be performed: obtaining a demand object image, the demand object image being used to represent a generation demand of the preset object in the target object image; and extracting a demand object feature of the preset object in the demand object image. Based on this, the generation of the target object image using the diffusion model based on the image encoding feature, the text encoding feature, and the preset noise can be implemented in the following manner: the generation of the target object image using the diffusion model based on the demand object feature, the object feature, the image encoding feature, the text encoding feature, and the preset noise.

[0129] The demand object feature refers to a visual feature with semantic representation extracted from the demand object image and used to guide the generation of the target image. The essence is a high-dimensional vector representation that can reflect the user's generation intention (such as style, structure, or attribute requirements) encoded by a neural network. Specifically, it can include two types of key information: 1) explicit demand, visual elements directly analyzed through image content (such as object contour, color distribution, texture pattern); and 2) implicit demand, deep semantics obtained through reasoning (such as artistic style, emotional tendency, functional constraint).

[0130] Here, the demand object image can be set by the user according to the demand. The demand object image can represent the generation demand of the object in the target object image, which can include specific poses, specific actions, specific backgrounds, and other characteristics of the object in the target object image. Thus, when generating the image, the demand object feature can be extracted from the demand object image, for example, by using a controlnet to extract the demand object feature from the demand object image. Then, the diffusion model is used to generate the preset noise based on the demand object feature, the object feature, the image encoding feature, and the text encoding feature, to obtain the target object image.

[0131] In this way, the embodiments of the present application realize fine-grained control of the generation process by fusing the demand object feature with the object feature, the image encoding feature, and the text encoding feature, ensuring that the output target object image not only meets the text description but also satisfies the user's visual demand (for example, migrating the oil painting style of the demand object image to the generation result).

[0132] In some embodiments, the demand object feature of the object in the demand object image can be extracted by performing the following steps: extracting at least one of the following features of the demand object image: a depth map feature of the demand object image, an object pose feature of the preset object in the demand object image, and an object contour feature of the preset object in the demand object image.

[0133] The depth map feature is a geometric representation generated by estimating the spatial distance (depth information) of each pixel point in the required object image, and is used to control the three-dimensional structure and scene level of the target object in the generation process. The object pose feature is the key point / skeletal structure data of the object or person extracted from the required object image, such as the motion pose or spatial orientation of the object, which can accurately guide the action layout of the generated object. The object contour feature is a two-dimensional shape representation obtained by edge detection or segmentation, which defines the shape boundary of the object and ensures the topological consistency between the generated result and the input requirement. The three kinds of structured features together constitute a multi-dimensional control signal, which respectively constrains the generation process from the spatial geometry (depth), dynamic pose (key point) and static shape (contour) three levels, and realizes the pixel-level controllable image synthesis.

[0134] In some embodiments, the process of image generation is implemented through an image generation model, as shown in FIG. 3D, which can be trained by performing the following steps: step 401, obtaining a first image sample, a second image sample and a description text sample, and adding noise to the second image sample to obtain a noisy image sample, wherein the first image sample and the second image sample both include an object sample, and the description text sample is used to describe the attributes of the object sample, and the region in the first image sample different from the region where the object sample is located is covered; step 402, extracting the first sample feature of the object sample in the first image sample through the object feature extraction network of the initial image generation model; step 403, performing first encoding on the first image sample to obtain first encoding features, and performing second encoding on the description text sample to obtain second encoding features; step 404, performing image generation on the noisy image sample based on the first sample feature, the first encoding feature and the second encoding feature through the image generation network of the initial image generation model to obtain a generated image; and step 405, training the initial image generation model based on the difference between the second image sample and the generated image to obtain the image generation model.

[0135] For step 401, a first image sample, a second image sample and a description text sample are obtained, and noise is added to the second image sample to obtain a noisy image sample. The first image sample and the second image sample both include an object sample, and the description text sample is used to describe the object sample in the generated image to be generated, and the region in the first image sample different from the region where the object sample is located is covered. When adding noise, noise can be generated first, which can be Gaussian noise, Poisson noise, etc. The noisy image sample is obtained by adding noise to the second image sample.

[0136] For step 402, the first sample feature of the object sample in the first image sample is extracted through the object feature extraction network of the initial image generation model. The process of step 402 is similar to that of extracting the object feature, which will not be described here.

[0137] For step 403, the first image sample is image encoded to obtain a first encoded feature, and the description text sample is text encoded to obtain a second encoded feature. The step 403 is similar to the step 102, and will not be repeated here.

[0138] For step 404, the noise image sample is image generated based on the first sample feature, the first encoded feature and the second encoded feature through the image generation network of the initial image generation model to obtain a generated image. For example, the noise image sample can be de-noised based on the first sample feature, the first encoded feature and the second encoded feature through the initial image generation model to obtain a de-noised image feature, and then the de-noised image feature is decoded to reconstruct the image to obtain the generated image. The step 404 is similar to the step 103, and will not be repeated here.

[0139] For step 405, the difference between the second image sample and the generated image is obtained, and then the value of the loss function of the initial image generation model is determined based on the difference, so that the model parameters of the initial image generation model are updated based on the value of the loss function to obtain the image generation model.

[0140] By applying the above embodiments of the present application, the target attributes of the preset object are described by the image description text, the initial object image is first encoded to obtain image encoding features, and the image description text is second encoded to obtain text encoding features, and the diffusion model is used to generate the target object image based on the image encoding features, the text encoding features and the preset noise. In this way, since the image encoding features retain the features of the image domain of the initial object image, the consistency of the generated target object image with the preset object in the initial object image is ensured, and the text encoding features guide the diffusion model to generate an image that conforms to the description of the image description text, so that the image generation process of the diffusion model is controlled through the cooperation of the multi-modal information of the image encoding features and the text encoding features, thereby generating a target object image that conforms to the description of the image description text and ensures the authenticity, and improving the image generation effect for the object.

[0141] The following describes an exemplary application of the embodiments of the present application in an actual application scenario. First, the nouns involved are described, including: 1) object (such as human body) features (ID), each object image (i.e. the initial object image described above) is unique, and the object ID reflects the appearance features of the object in the object image, including the object facial features (ID), the object's clothing, accessories, shoes, hairstyle, etc.; 2) object facial ID, which refers to the object facial features, including the shape and features of the facial features.

[0142] The image generation model in the related art can only have image generation capability of a single object ID, and if image generation capability of other object IDs is needed, the image generation model needs to be fine-tuned by images of the other object IDs, that is, independent training needs to be performed for each different object ID, and the training time is also relatively long.

[0143] Based on this, the embodiment of the present application can achieve that in the test stage, the target object ID can be generated without fine-tuning again based on the diffusion model after fine-tuning the image generation model by a certain number of object ID data in the early stage, and the generation result can strictly keep consistent with the original object ID. The pre-fine-tuning model method proposed in the embodiment of the present application is a training method for enabling the diffusion model to have human ID understanding and keeping ability. The method uses object image data as the input of the "main feature keeping network" (i.e., the object feature extraction network mentioned above) designed additionally, so as to obtain object features, and uses the object features as the generation condition of the diffusion model, and the diffusion model can guide the generation of the target object image using the generation condition.

[0144] Next, the image generation method provided by the embodiment of the present application is described. Referring to FIG. 4, FIG. 4 is a schematic diagram of an image generation architecture for image generation provided by the embodiment of the present application. Here, the image generation architecture for image generation includes a visual encoder 1 (Visual Encoder1), a visual encoder 2 (Visual Encoder2), a text encoder (Text Encoder), an object (or main body) feature keeping network (Reference Unet), and an image generation network (Pretrained Unet). Among them, the main body feature keeping network and the image generation network are both constructed based on the Unet network in the Stable Diffusion model. The structure of the Unet network is shown in FIG. 5, which is a structural schematic diagram of a U-shaped network 501 provided by the embodiment of the present application. The Unet network includes a down-sampling network and an up-sampling network. The down-sampling network includes M layers of down-sampling layers, and each down-sampling layer includes a first convolutional network (for down-sampling processing), a self-attention (Self Attention) processing layer, and a cross-attention (Cross Attention) processing layer. The up-sampling network includes M layers of up-sampling layers, and each up-sampling layer includes a second convolutional network (for up-sampling processing), a self-attention (Self Attention) processing layer, and a cross-attention (Cross Attention) processing layer. The following are described respectively.

[0145] For the object image (Mask object image) and the image description text (Text prompt) used for image generation, one branch is to perform text encoding on the image description text through a text encoder (Text Encoder) to obtain text encoding features (Text Embedding), where the Text Encoder can be any image-text matching encoder, such as ViT-L / 14, ViT-B / 32, etc. Another branch is to perform image encoding on the object image through an image encoder 1 (Visual Encoder1) to obtain image encoding features (Image Embedding), where the Image Encoder can also be any image-text matching encoder, such as ViT-L / 14, ViT-B / 32, etc. It should be noted that the area in the object image different from the area where the object is located is covered, and the object is the main body in the object image, which can be a real person, a virtual person (such as an animation character, a game character), a virtual object, etc.

[0146] Continuing, the obtained Text Embedding and Image Embedding are jointly input into a cross attention (Cross Attention) processing layer of an image generation network (Pretrained Unet), and the working mode of the cross attention processing layer is as shown in FIG. 6 (FIG. 6 is a processing flow diagram of a cross attention processing layer 601 provided by an embodiment of the present application), that is, cross attention processing is performed with the Text Embedding as a condition vector (Cross Attention for text condition) to obtain a text hidden state; cross attention processing is performed with the Image Embedding as a condition vector (Cross Attention for image condition) to obtain an image hidden state; and the text hidden state and the image hidden state are added to obtain a hidden state output by the cross attention processing layer.

[0147] Continuing, the subject feature preserving network receives two inputs, including: an object image (Mask object image) as the Unet input of the subject feature preserving network, and an Image embedding obtained by image encoding the object image based on the image encoder 2 (Visual ncoder2) as the Unet condition vector input of the subject feature preserving network. It should be noted that the hidden vector (embedding) of each Self Attention input in the subject feature preserving network is saved, and the saved hidden vector is used for each Self Attention processing in the image generation network. For example, as shown in FIG. 7 (FIG. 7 is a processing flow diagram of the self-attention processing layer 701 provided in the embodiment of the present application), in the Self Attention of the image generation network, the hidden vector from the subject feature preserving network (embedding from Reference Unet) is used as the K variable and the V variable in the Self Attention, is concatenated (Concat) with the K variable and the V variable of the Self Attention of the image generation network, and then the hidden vector from the image generation network (embedding from Pretrained Unet) is used as the Q variable, and the self-attention processing is performed based on the concatenation result.

[0148] Based on the above, when generating an image based on the image generation architecture, in the actual use stage (inference stage), an object image with the background region cut out and an image description text describing the object in the object image (for example, which can describe the background environment of the object, the action or posture desired by the object, etc.) are obtained; a preset noise (such as Gaussian noise) is generated; the object image and the Image embedding of the object image are processed by the Reference Unet to obtain a hidden vector; the Text Embedding of the image description text, the Image Embedding of the object image, and the hidden vector output by the Reference Unet are combined to denoise the preset noise to obtain a denoised feature; and the denoised feature is decoded to obtain a generated target object image (Object-consistency Generated Image).

[0149] Next, the training process of the image generation model provided in the embodiment of the present application is described. The embodiment of the present application mainly trains a diffusion model controlled by Image Embedding and Text Embedding. The specific training process includes:

[0150] 1) Collect a batch of object image samples, group them according to object ID, and each object ID corresponds to at least two object image samples with different backgrounds or poses. The data size is 100,000 or more object IDs. Here, the object image samples are preprocessed, which can include: labeling the object frame position in the object image sample, and the mask image (used to extract the object image except the background), and grouping the object image samples according to the object ID. In addition, each object image sample needs to be described by a Bootstrapping Language-Image Pretraining (Blip) or other language model, including the object's pose, action, and background content. The description text sample is obtained.

[0151] 2) Referring to FIG. 8, which is a training diagram of an image generation model provided by an embodiment of the present application. For a group of object ID object image samples, one object image sample is selected as the object condition image (i.e., the first image sample described above), denoted as I cond , and another object image sample is selected as the target generated image (i.e., the second image sample described above), denoted as I tgt . Here, I cond is obtained by separately extracting the object part from the mask image obtained in the pre-data preparation process, i.e., the Masked object image part in FIG. 8. Masked object image will be sent to Pretrained Unet through Visual Encoder. Similarly, the description text sample obtained in the preprocessing part will be sent to Pretrained Unet through Text Encoder to obtain Text Embedding.

[0152] 3) Masked object image will also pass through VAE model to obtain the input of the subject feature preservation network.

[0153] 4) I tgt in the same group of object IDs will be input into Pretrained Unet for single-step noise addition (i.e., adding noise) and noise prediction through Pretrained Unet to obtain predicted noise; then calculate the loss. For example, the denoising result obtained by using DDIM Scheduler to remove noise can be compared with the original input I tgtThe L2 loss is calculated, and the prediction noise and the added noise can also be used to calculate the L2 loss; then the weight vectors of the main body feature preservation network and the Cross Attention part are iteratively optimized, wherein the Cross Attention part is an additional part due to the additional Visual Encoder in the Pretrained Unet. The overall training logic is the same as that of the diffusion model. The added noise is predicted through diffusion and sampling, and the loss is compared with the actual added noise, and the model is iteratively optimized. The training process is actually to randomly sample the noisy image at the t-th step, input the noisy picture and the step number t, and the model predicts the noise ε. The model training target is that the error between the predicted noise and the actual added noise is as small as possible. Here, it can be understood that the single-step result of the Diffusion network z(t-1) gradually denoising is z(t), and later z(t) will become the new z(t-1) to continue to denoise to become the new z(t). In the training stage, it is a single-step iteration, and the iteration step length in the prediction stage (inference stage) can be 25-50 steps of iteration.

[0154] The application scenario of the image generation method provided by the embodiments of the present application includes:

[0155] 1) As shown in FIG. 9A, a human image generation scene under a target human ID. As shown in FIG. 9A, part (1) of FIG. 9A is an object image, part (2) of FIG. 9A is two kinds of image description texts, and part (3) of FIG. 9A is a target object image corresponding to each image description text.

[0156] 2) As shown in FIG. 9B, a multi-panel story comic generation scene under the same human ID. Part (1) of FIG. 9B is an object image, part (2) of FIG. 9B is four different image description texts, and part (3) of FIG. 9B is a target object image corresponding to each image description text, which has different scenes and actions, and the patterns of the clothes of the characters are completely consistent.

[0157] 3) As shown in FIG. 9C, a four-panel comic or comic serialization generation scene of a two-dimensional or 3D character. Part (1) of FIG. 9C is an object image, part (2) of FIG. 9C is four different image description texts, and part (3) of FIG. 9C is a target object image corresponding to each image description text, which is a product schematic of a story comic or a story serialization form under different styles of human IDs.

[0158] As shown in FIG. 9D, it is a generation scene of a four-frame comic or a comic strip of a two-dimensional or 3D character. Part (1) of FIG. 9D is an object image, part (2) of FIG. 9D is four different image description texts, and part (3) of FIG. 9D is a target object image corresponding to each image description text, which is a product schematic diagram in the form of a story comic or a story strip.

[0159] 4) Using the human body ID to maintain the consistency of the generated subject to make a story video. Using the above image results, combined with the display of music and subtitles, a comic video or a story strip video with coherence is generated.

[0160] In the embodiments of the present application, the image generation model does not need to use a specific style Stable Diffusion model to maintain the object ID of any style, which benefits from the excellent characteristics of the subject feature preservation network proposed in the present application; of course, the embodiments of the present application can still be compatible with different styles of Stable Diffusion derivative models or Low-Rank Adaptation (LoRA); for controlnet, the corresponding controlnet model can be mounted on the already trained image generation network here, so as to obtain the input of additional control signals, such as including depth map, skeletal pose map, face deflection angle control, etc. to control the generation result; the Stable Diffsuion framework used in the present application can be extended to all Stable Diffusion family, including but not limited to Stable Diffusion2.0, Stable Diffusion XL.

[0161] By applying the above embodiments of the present application, 1) both text features and object ID features are considered in the input, and the two features can be mixed to achieve double control; 2) without using a specific style Stable Diffusion model, the object ID of any style can be maintained, and the generated image based on the embodiments of the present application can strictly correspond to the object image in terms of face ID, clothing of the character (especially the texture and pattern on the clothes), accessories, shoes, and hairstyle of the character; 3) the embodiments of the present application can also be combined with various Stable Diffusion derivative models, LoRA, Adapter, controlnet, to achieve more control (such as including depth map, human skeletal pose map, face deflection angle control, etc.) while ensuring the object ID.

[0162] It should be noted that the face in the drawings provided in the embodiments of the present application is generated based on artificial intelligence technology, not a real face.

[0163] The following continues to illustrate an exemplary structure of the image generation apparatus 555 provided by the embodiments of the present application, which is implemented as a software module. In some embodiments, as shown in FIG. 2, the software module stored in the image generation apparatus 555 of the memory 550 can include: an acquisition module 5551 configured to acquire a preset noise, and acquire an initial object image of a preset object and image description text used to describe a target attribute of the preset object; an encoding module 5552 configured to perform first encoding on the initial object image to obtain image encoding features, and perform second encoding on the image description text to obtain text encoding features; and a generation module 5553 configured to generate a target object image based on the image encoding features, the text encoding features, and the preset noise using a diffusion model, the target object image including the preset object corresponding to the target attribute.

[0164] In some embodiments, the generation module 5553 is further configured to extract an object feature of the preset object from the initial object image; and generate a target object image based on the object feature, the image encoding features, the text encoding features, and the preset noise using a diffusion model.

[0165] In some embodiments, the object feature includes a first object feature and a second object feature, and the generation module 5553 is further configured to perform third encoding on the initial object image to obtain object encoding features; perform down-sampling on the initial object image based on the object encoding features to obtain the first object feature; and perform up-sampling on the first object feature based on the object encoding features to obtain the second object feature.

[0166] In some embodiments, the generation module 5553 is further configured to perform first convolution processing on the initial object image to obtain first convolution features; perform self-attention processing on the first convolution features to obtain a first processing result; and perform cross-attention processing on the object encoding features and the first processing result to obtain the first object feature.

[0167] In some embodiments, the generation module 5553 is further configured to perform second convolution processing on the first object feature to obtain second convolution features; perform self-attention processing on the second convolution features to obtain a second processing result; and perform cross-attention processing on the object encoding features and the second processing result to obtain the second object feature.

[0168] In some embodiments, the downsampling process is implemented by M layers of downsampling layers, where M is an integer greater than 1; the generation module 5553 is further configured to, by a first layer of the M layers of downsampling layers, perform downsampling on the initial object image based on the object encoding feature to obtain a layer downsampling result of the first layer of downsampling layers; by an i-th layer of the M layers of downsampling layers, perform downsampling on a layer downsampling result of an (i-1)-th layer of downsampling layers based on the object encoding feature to obtain a layer downsampling result of the i-th layer of downsampling layers; iterate through the i to obtain a layer downsampling result of an M-th layer of downsampling layers, where i is an integer greater than 1 and i is less than or equal to M; and take the layer downsampling result of each layer of downsampling layers as the first object feature.

[0169] In some embodiments, the upsampling process is implemented by M layers of upsampling layers, where a first layer of the M layers of upsampling layers corresponds to a first layer of the M layers of downsampling layers; the generation module 5553 is further configured to, by an M-th layer of the M layers of upsampling layers, perform upsampling on the first object feature based on the object encoding feature to obtain a layer upsampling result of the M-th layer of upsampling layers; splice the layer downsampling result of the i-th layer of downsampling layers and a layer upsampling result of an (i+1)-th layer of upsampling layers to obtain a splicing result of the i-th layer of upsampling layers, and by the i-th layer of upsampling layers, perform upsampling on the splicing result of the i-th layer of upsampling layers based on the object encoding feature to obtain a layer upsampling result of the i-th layer of upsampling layers; iterate through the i to obtain a layer upsampling result of a first layer of upsampling layers; and take the layer upsampling result of each layer of upsampling layers as the second object feature.

[0170] In some embodiments, the generation module 5553 is further configured to perform the following processing using a diffusion model: based on the object feature, the image encoding feature, and the text encoding feature, denoising the preset noise to obtain a denoised feature; and performing image reconstruction on the denoised feature to obtain the target object image.

[0171] In some embodiments, the generation module 5553 is further configured to, based on the object feature, the image encoding feature, and the text encoding feature, perform downsampling on the preset noise to obtain a noise downsampling result; and based on the object feature, the image encoding feature, and the text encoding feature, perform upsampling on the noise downsampling result to obtain the denoised feature.

[0172] In some embodiments, the generation module 5553 is further configured to perform first convolution processing on the preset noise to obtain a first noise feature; concatenate the first noise feature and the object feature to obtain a first concatenation result, and perform self-attention processing on the first concatenation result to obtain a third processing result; perform cross-attention processing on the image encoding feature and the third processing result to obtain a first image processing result, and perform cross-attention processing on the text encoding feature and the third processing result to obtain a first text processing result; and determine the noise down-sampling result based on the first image processing result and the first text processing result.

[0173] In some embodiments, the generation module 5553 is further configured to perform second convolution processing on the noise down-sampling result to obtain a second noise feature; concatenate the second noise feature and the object feature to obtain a second concatenation result, and perform self-attention processing on the second concatenation result to obtain a fourth processing result; perform cross-attention processing on the image encoding feature and the fourth processing result to obtain a second image processing result, and perform cross-attention processing on the text encoding feature and the fourth processing result to obtain a second text processing result; and determine the de-noising feature based on the second image processing result and the second text processing result.

[0174] In some embodiments, the de-noising process includes T times of de-noising processing, where T is an integer greater than 1; the generation module 5553 is further configured to perform the first de-noising on the preset noise based on the object feature, the image encoding feature, and the text encoding feature to obtain an intermediate de-noising feature of the first de-noising output; perform the t-th de-noising on the intermediate de-noising feature of the (t-1)-th de-noising output based on the object feature, the image encoding feature, and the text encoding feature to obtain an intermediate de-noising feature of the t-th de-noising output, where t is an integer greater than 1 and t is less than or equal to T; traverse the t to obtain an intermediate de-noising feature of the T-th de-noising output, and take the intermediate de-noising feature of the T-th de-noising output as the de-noising feature.

[0175] In some embodiments, the image encoding feature includes an object feature of the preset object.

[0176] In some embodiments, the generation module 5553 is further configured to obtain a demand object image, where the demand object image is used to represent a generation demand of a preset object in the target object image; and extract a demand object feature of the preset object in the demand object image.

[0177] In some embodiments, the generation module 5553 is further configured to generate the target object image based on the demand object feature, the image encoding feature, the text encoding feature, and the preset noise using a diffusion model.

[0178] In some embodiments, the generation module 5553 is further configured to extract at least one of the following features of the demand object image: a depth map feature of the demand object image, an object pose feature of a preset object in the demand object image, and an object contour feature of the preset object in the demand object image.

[0179] In some embodiments, the acquisition module 5551 is further configured to acquire a candidate object image including the preset object; determine a target region in the candidate object image different from a region where the preset object is located; and mask the target region in the candidate object image to obtain the initial object image.

[0180] In some embodiments, the image generation process is implemented through an image generation model, and the generation module 5553 is further configured to acquire a first image sample, a second image sample, and a description text sample, and add noise to the second image sample to obtain a noise image sample, wherein the first image sample and the second image sample each include an object sample, the description text sample is used to describe the attributes of the object sample, and a region in the first image sample different from a region where the object sample is located is masked; extract a first sample feature of the object sample in the first image sample through an object feature extraction network of an initial image generation model; perform first encoding on the first image sample to obtain a first encoding feature, and perform second encoding on the description text sample to obtain a second encoding feature; perform image generation on the noise image sample based on the first sample feature, the first encoding feature, and the second encoding feature through an image generation network of the initial image generation model to obtain the generated image; and train the initial image generation model based on the difference between the second image sample and the generated image to obtain the image generation model.

[0181] It should be noted that the description of the device embodiments in the present application is similar to the description of the above-mentioned method embodiments, and has similar beneficial effects as the method embodiments, which will not be repeated here. For technical details not described in the image generation device provided by the embodiments of the present application, they can be understood based on the description of the technical details in the above-mentioned method embodiments.

[0182] The embodiment of the present application further provides a computer program product, which comprises computer executable instructions or a computer program stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions or the computer program from the computer readable storage medium, and executes the computer executable instructions or the computer program, so that the electronic device executes the image generation method provided by the embodiment of the present application.

[0183] The embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by the processor, the processor will execute the image generation method provided by the embodiment of the present application.

[0184] In some embodiments, the computer readable storage medium can be RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM memory, etc. It can also be various devices comprising one or any combination of the above storage.

[0185] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0186] As an example, the computer executable instructions can but not necessarily correspond to files in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code parts).

[0187] As an example, the computer executable instructions can be deployed to execute on one electronic device, or on multiple electronic devices located in one place, or on multiple electronic devices distributed in multiple places and interconnected through a communication network.

[0188] It can be understood that in the embodiment of the present application, the data related to user information and the like are involved, and when the embodiment of the present application is applied to specific products or technologies, the user's permission or consent needs to be obtained, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the relevant region.

[0189] The above merely provides an example of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. An image generation method, applied to an electronic device, the method comprising: Obtain preset noise, and obtain the initial object image and image description text of the preset object, wherein the image description text is used to describe the target attributes of the preset object; The initial object image is first encoded to obtain image encoding features; The image description text is then subjected to a second encoding to obtain text encoding features; Based on the image encoding features, the text encoding features, and the preset noise, a diffusion model is used to generate a target object image, the target object image including the preset object corresponding to the target attribute.

2. The method as described in claim 1, wherein, The step of generating a target object image using a diffusion model based on the image encoding features, the text encoding features, and the preset noise includes: Extract the object features of the preset object from the initial object image; Based on the object features, the image encoding features, the text encoding features, and the preset noise, a diffusion model is used to generate a target object image.

3. The method as described in claim 2, wherein, The object features include a first object feature and a second object feature. Extracting the object features of the preset object from the initial object image includes: The initial object image is subjected to a third encoding to obtain object encoding features; Based on the object encoding features, the initial object image is downsampled to obtain the first object features; Based on the object encoding features, the first object features are upsampled to obtain the second object features.

4. The method of claim 3, wherein, The step of downsampling the initial object image based on the object encoding features to obtain the first object features includes: The initial object image is subjected to a first convolution process to obtain the first convolution feature; The first convolutional feature is subjected to self-attention processing to obtain the first processing result; The object encoding features and the first processing result are subjected to cross-attention processing to obtain the first object features.

5. The method as described in any one of claims 3-4, wherein, The step of upsampling the first object feature based on the object encoding feature to obtain the second object feature includes: The first object features are subjected to a second convolution process to obtain the second convolution features; The second convolutional feature is subjected to self-attention processing to obtain the second processing result; The object encoding features and the second processing result are subjected to cross-attention processing to obtain the second object features.

6. The method according to any one of claims 3-5, wherein, The downsampling process is implemented through M downsampling layers, where M is an integer greater than 1; the downsampling of the initial object image based on the object encoding features to obtain the first object features includes: By using the first downsampling layer in the M-layer downsampling layer, the initial object image is downsampled based on the object encoding features to obtain the downsampling result of the first downsampling layer; By using the i-th downsampling layer in the M-layer downsampling layer, and based on the object encoding features, the downsampling result of the (i-1)-th downsampling layer is downsampled to obtain the downsampling result of the i-th downsampling layer; By iterating through i, the downsampling result of the Mth downsampling layer is obtained, where i is an integer greater than 1 and i is less than or equal to M; The downsampling result of each downsampling layer is used as the first object feature.

7. The method of claim 6, wherein, The upsampling process is implemented through M upsampling layers, where the i-th upsampling layer corresponds to the i-th downsampling layer. The step of upsampling the first object feature based on the object encoding feature to obtain the second object feature includes: By using the Mth upsampling layer in the Mth upsampling layer, the first object feature is upsampled based on the object encoding feature to obtain the layer upsampling result of the Mth upsampling layer; The downsampling result of the i-th downsampling layer is concatenated with the upsampling result of the (i+1)-th upsampling layer to obtain the concatenated result of the i-th upsampling layer. Based on the object encoding features, the concatenated result of the i-th upsampling layer is upsampled to obtain the upsampling result of the i-th upsampling layer. By iterating through i, the upsampling result of the first upsampling layer is obtained; The upsampling results of each upsampling layer are used as the second object feature.

8. The method according to any one of claims 2-7, wherein, The step of generating a target object image using a diffusion model based on the object features, the image encoding features, the text encoding features, and the preset noise includes: Perform the following processing using the diffusion model: Based on the object features, the image encoding features, and the text encoding features, the preset noise is denoised to obtain denoised features; The denoised features are used to reconstruct the image to obtain the target object image.

9. The method of claim 8, wherein, The step of denoising the preset noise based on the object features, the image encoding features, and the text encoding features to obtain denoised features includes: Based on the object features, the image encoding features, and the text encoding features, the preset noise is downsampled to obtain a noise downsampling result. Based on the object features, the image encoding features, and the text encoding features, the noise downsampling result is upsampled to obtain the denoising features.

10. The method of claim 9, wherein, The step of downsampling the preset noise based on the object features, the image encoding features, and the text encoding features to obtain a noise downsampling result includes: The preset noise is subjected to a first convolution process to obtain a first noise feature; The first noise feature and the object feature are concatenated to obtain a first concatenation result, and self-attention processing is applied to the first concatenation result to obtain a third processing result; Cross-attention processing is performed on the image encoding features and the third processing result to obtain a first image processing result, and cross-attention processing is performed on the text encoding features and the third processing result to obtain a first text processing result; Based on the first image processing result and the first text processing result, the noise downsampling result is determined.

11. The method as described in any one of claims 9-10, wherein, The step of upsampling the noise downsampling result based on the object features, the image encoding features, and the text encoding features to obtain the denoising features includes: The noise downsampling result is subjected to a second convolution process to obtain a second noise feature; The second noise feature and the object feature are concatenated to obtain a second concatenation result, and self-attention processing is applied to the second concatenation result to obtain a fourth processing result; Cross-attention processing is performed on the image encoding features and the fourth processing result to obtain a second image processing result, and cross-attention processing is performed on the text encoding features and the fourth processing result to obtain a second text processing result; The denoising features are determined based on the second image processing result and the second text processing result.

12. The method as described in any one of claims 8-11, wherein, The denoising process includes T denoising operations, where T is an integer greater than 1; the denoising of the preset noise based on the object features, the image encoding features, and the text encoding features to obtain denoised features includes: Based on the object features, the image encoding features, and the text encoding features, the preset noise is denoised for the first time to obtain the intermediate denoised features of the first denoised output; Based on the object features, the image encoding features, and the text encoding features, the intermediate denoising features of the (t-1)th denoising output are denoised for the tth time to obtain the intermediate denoising features of the tth denoising output, where t is an integer greater than 1 and t is less than or equal to T; Iterate through t to obtain the intermediate denoising features of the Tth denoising output, and use the intermediate denoising features of the Tth denoising output as the denoising features.

13. The method according to any one of claims 1-12, wherein, The image encoding features include the object features of the preset object.

14. The method according to any one of claims 1-13, wherein, The method further includes: Obtain a requirement object image, which represents the generation requirement of a preset object in the target object image; Extract the requirements object features of the preset objects from the requirements object image.

15. The method of claim 14, wherein, The step of generating a target object image using a diffusion model based on the image encoding features, the text encoding features, and the preset noise includes: Based on the characteristics of the target object, the image encoding characteristics, the text encoding characteristics, and the preset noise, a diffusion model is used to generate the target object image.

16. The method as described in any one of claims 14-15, wherein, The step of extracting the demand object features of the preset object in the demand object image includes: Extract at least one of the following features from the image of the target object: The depth map features of the required object image, the object pose features of the preset object in the required object image, and the object contour features of the preset object in the required object image.

17. The method according to any one of claims 1-16, wherein, The initial object image is obtained in the following way: Obtain candidate object images including the preset object; Determine the target region in the candidate object image that is different from the region where the preset object is located; The target region in the candidate object image is masked to obtain the initial object image.

18. The method as described in any one of claims 1-17, wherein, The diffusion model includes an image generation model, and the method further includes: A first image sample, a second image sample, and a descriptive text sample are obtained, and noise is added to the second image sample to obtain a noisy image sample. The first image sample and the second image sample both include object samples, and the descriptive text sample is used to describe the attributes of the object samples. The regions in the first image sample that are different from the regions where the object samples are located are covered. The object feature extraction network of the initial image generation model is used to extract the first sample features of the object samples in the first image sample; The first image sample is encoded in a first way to obtain a first encoded feature, and the descriptive text sample is encoded in a second way to obtain a second encoded feature; The image generation network of the initial image generation model generates an image from the noisy image sample based on the first sample features, the first coding features, and the second coding features, thereby obtaining the generated image. Based on the difference between the second image sample and the generated image, the initial image generation model is trained to obtain the image generation model.

19. An image generation apparatus, the apparatus comprising: The acquisition module is configured to acquire preset noise and acquire an initial object image and image description text of a preset object, wherein the image description text is used to describe the target attributes of the preset object; The encoding module is configured to perform a first encoding on the initial object image to obtain image encoding features, and to perform a second encoding on the image description text to obtain text encoding features; The generation module is configured to generate a target object image using a diffusion model based on the image encoding features, the text encoding features, and the preset noise. The target object image includes the preset object corresponding to the target attribute.

20. An electronic device, the electronic device comprising: Memory, configured to store computer-executable instructions; The processor, configured to execute computer-executable instructions stored in the memory, implements the image generation method according to any one of claims 1 to 18.

21. A computer-readable storage medium storing computer-executable instructions or a computer program, which, when executed by a processor, implement the image generation method according to any one of claims 1 to 18.

22. A computer program product comprising computer-executable instructions or a computer program, wherein when executed by a processor, the computer-executable instructions or the computer program implement the image generation method according to any one of claims 1 to 18.

Images