Image generation method and apparatus and electronic device

Abstract

The application discloses an image generation method and device and an electronic device, relates to the technical field of image processing, and comprises the following steps: acquiring a reference image and a detailed text description corresponding to the reference image, wherein the detailed text description is used for describing a point of interest of the reference image; inverting the reference image to obtain a noise image containing feature information of the reference image; and matching the detailed text description, performing noise removal on the noise image to generate a similar image similar to the reference image. The application can enrich image data of a single scene.

Description

Image generation methods, apparatus and electronic devices Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image generation method, apparatus, and electronic device. Background Technology

[0002] In certain business scenarios, image data is often concentrated in a single scene, with a fixed perspective and different background elements. For example, target images taken under specific background, lighting conditions, or by a specific device. Although this type of image data can support the model to achieve high-precision recognition or detection in that scene, the lack of diversity in the image data causes the model to perform poorly in other similar scenarios when it is used for model training, resulting in insufficient generalization ability. Summary of the Invention

[0003] This application provides an image generation method that can solve the problem of insufficient diversity of image data in a single scene.

[0004] In a first aspect, embodiments of this application provide an image generation method, the method comprising:

[0005] Obtain a reference image and a detailed text description corresponding to the reference image, wherein the detailed text description is used to describe the focus of the reference image;

[0006] Inverting the reference image yields a noise map containing the feature information of the reference image;

[0007] Matching the detailed text description, the noise map is denoised to generate a similar image that is similar to the reference image.

[0008] Optionally, the reference image is inverted to obtain a noise map containing feature information of the reference image, including:

[0009] In multiple time steps, the reference image is iteratively denoised to obtain the noise map, wherein each round of iterative denoising includes the following steps:

[0010] Obtain the gradient vector field at a time step corresponding to the current iteration of noise addition, wherein the gradient vector field is generated by the image generation model based on the noisy image after the previous iteration of noise addition;

[0011] The gradient vector field at one time step is used to add noise to the noisy image after the previous iteration to obtain the noisy image after the current iteration.

[0012] The noise map is determined using the noisy image obtained in the last iteration.

[0013] Optionally, the reference image is iteratively denoised at multiple time steps to obtain the noise map, including:

[0014] The plurality of time steps is T time steps;

[0015] For the t-th time step, the noisy image latent vector from the (t-1)-th time step and the time step number t-1 are input into the image generation model. The output is the gradient vector field pointing towards the reference image at the t-th time step. Based on the step size difference between two adjacent time steps, the gradient vector field at the t-th time step, and the noisy image latent vector from the (t-1)-th time step, the noisy image latent vector at the t-th time step is determined. This process continues until the noisy image latent vector corresponding to the last time step in T time steps is obtained. The noise map is then determined based on the noisy image latent vector corresponding to the last time step, where t ∈ [1, T].

[0016] For the first time step, the reference image is input into the image generation model, and the gradient vector field of the first time step is output. The noisy image latent vector of the first time step is determined by the gradient vector field of the first time step and the reference image.

[0017] Optionally, the image generation model includes an image encoder, a diffusion transformer, and an image decoder, and iteratively adds noise to the reference image at multiple time steps to obtain the noise map, including:

[0018] The reference image is encoded using the image encoder to obtain the image encoding vector of the reference image;

[0019] The image encoding vector and the number of time steps are input into the diffusion transformer, and the gradient vector field of the first time step is output. The noisy image latent vector of the first time step is determined by the gradient vector field of the first time step and the reference image.

[0020] For the t-th time step, the noisy image latent vector of the (t-1)-th time step and the time step number t-1 are input into the diffusion transformer, and the gradient vector field of the t-th time step is output. Based on the step size difference between two adjacent time steps, the gradient vector field of the t-th time step, and the noisy image latent vector of the (t-1)-th time step, the noisy image latent vector of the t-th time step is determined, until the noisy image latent vector corresponding to the last time step in T time steps is obtained, where 1 < t ≤ T, and T represents the total number of time steps;

[0021] The image decoder decodes the noisy image latent vector corresponding to the last time step and outputs the corresponding noise map.

[0022] Optionally, obtain a reference image and a detailed text description corresponding to the reference image, including:

[0023] Obtain a reference image;

[0024] In response to simple text information input by the user, the simple text information is used to describe the focus of the reference image, the focus of which includes the scene information that the user is focusing on;

[0025] Based on the simple text information, semantic features are extracted from the points of interest in the reference image to obtain a detailed text description corresponding to the reference image.

[0026] Optionally, based on the simple text information, semantic features are extracted from the points of interest in the reference image to obtain a detailed text description corresponding to the reference image, including:

[0027] The simple text information and reference image are input into a multimodal large model. Guided by prompts for the noise description angle, and combined with at least one adjustment factor among chroma, brightness, illumination, and background changes, a detailed text description of the reference image corresponding to the corresponding noise description angle is generated.

[0028] Optionally, matching the detailed text description, denoising the noise map to generate a similar image to the reference image includes:

[0029] The detailed text description and the noise map are input into the image generation model. Under the guidance of the detailed text description, the noise map is denoised to generate a similar image to the reference image.

[0030] Optionally, the image generation model includes a text encoder, an image encoder, a diffusion transformer, and an image decoder, which, when matched with the detailed text description, denoise the noise map to generate a similar image to the reference image, including:

[0031] The detailed text description is semantically encoded using the text encoder to obtain the text semantic vector corresponding to the detailed text description.

[0032] The image encoder performs image encoding on the noise map to obtain the image encoding vector corresponding to the noise map;

[0033] The image encoding vector and the text semantic vector are input into the diffusion transformer. Guided by the text semantic vector, iterative denoising is performed to obtain the denoised latent vector.

[0034] The image decoder decodes the denoised latent vectors to obtain a similar image to the reference image, and the similar image matches the detailed text description.

[0035] Secondly, embodiments of this application provide an image generation apparatus, the apparatus comprising:

[0036] A semantic extraction module is used to obtain a reference image and a detailed text description corresponding to the reference image, wherein the detailed text description is used to describe the focus of the reference image;

[0037] An image inversion module is used to invert the reference image to obtain a noise map containing the feature information of the reference image;

[0038] An image generation module is used to match the detailed text description, denoise the noise image, and generate a similar image that is similar to the reference image.

[0039] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the image generation method described above.

[0040] This application obtains a noisy map containing feature information of the reference image by inverting the reference image, and then denoises the noisy map based on the detailed text description corresponding to the reference image to generate a similar image. This solves the problem of insufficient diversity in image data for a single scene, and can generate images that are similar to the reference image in terms of scene and target of interest, but with certain differences in angle, lighting, and background. Furthermore, the generated images retain information from the reference image while also incorporating information of interest to the user, enriching image data for a single scene and reducing the model's dependence on specific scenes. This method can directly infer and generate similar images without training, eliminating the need to train complex image generation models, saving computational resources and time, and reducing costs. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 is a first flowchart of the image generation method provided in an embodiment of this application;

[0043] Figure 2 is a schematic diagram of the second process of the image generation method provided in the embodiment of this application;

[0044] Figure 3 is a schematic diagram of a reference image provided in an embodiment of this application;

[0045] Figure 4 is a schematic diagram of the third process of the image generation method provided in the embodiments of this application;

[0046] Figure 5 is a schematic diagram of the generated similar image provided in the embodiment of this application;

[0047] Figure 6 is a system block diagram of the image generation apparatus provided in an embodiment of this application;

[0048] Figure 7 is a system block diagram of the electronic device provided in an embodiment of this application. Detailed Implementation

[0049] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0050] Please refer to Figure 1. This embodiment of the application provides an image generation method, which includes the following steps:

[0051] S101, Obtain the reference image and its corresponding detailed text description;

[0052] S102, Invert the reference image to obtain a noise map containing the feature information of the reference image;

[0053] S103, matching the detailed text description, denoising the noisy image, and generating a similar image to the reference image.

[0054] For example, a detailed text description is used to describe the focus of the reference image. For example, the focus includes scene information that the user is interested in. For instance, if a user provides reference image A, and the focus in reference image A is a "person fallen to the ground," a detailed text description of that focus is obtained based on reference image A.

[0055] For example, an image generation model is used to invert a reference image to a noise latent variable space to obtain a noise map containing feature information of the reference image.

[0056] For example, an image generation model, guided by a detailed text description, is used to denoise a noisy image and generate a similar image that matches the reference image and the detailed text description.

[0057] In this embodiment, the reference image is inverted to obtain a noise map containing the feature information of the reference image. The noise map is then denoised based on the detailed text description corresponding to the reference image to generate a similar image. This addresses the problem of insufficient diversity in image data for a single scene, generating images that are similar to the reference image in terms of scene and target of interest, but with differences in angle, lighting, and background variations. Furthermore, the generated images retain information from the reference image while also incorporating information relevant to the user, enriching the image data for a single scene and reducing the model's dependence on specific scenes. This method can directly infer and generate similar images without training, eliminating the need to train complex image generation models, saving computational resources and time, and reducing costs.

[0058] One embodiment of this application, as shown in FIG2, involves obtaining a reference image and a corresponding detailed text description, including the following steps:

[0059] S201, Obtain the reference image;

[0060] S202, responding to simple text information input by the user, the simple text information is used to describe the focus of the reference image, the focus of which includes scene information that the user is concerned with;

[0061] S203, based on simple text information, extract semantic features from the points of interest in the reference image to obtain a detailed text description corresponding to the reference image.

[0062] A reference image is acquired, along with simple text information input by the user. This simple text information describes the focus of attention in the reference image, representing the information in the reference image that the user is interested in. For example, the focus may include scene information that the user is interested in. For instance, for reference image A, the user could use the simple text information "person fallen" to indicate that the user is interested in the scene information of a person falling in reference image A; for reference image B, the user could use the simple text information "car on fire" to indicate that the user is interested in the scene information of a car on fire in reference image B.

[0063] By combining this simple text information and utilizing the points of interest in the reference image, semantic features are extracted from the reference image to obtain a detailed text description corresponding to the reference image. This detailed text description is used to describe the points of interest in the reference image, so as to generate a more accurate text description that meets the user's needs.

[0064] In one embodiment of this application, simple text information and a reference image are input into a multimodal large model. Guided by prompts targeting a noise description angle, and incorporating at least one adjustment factor among chroma, brightness, illumination, and background variations, a detailed text description corresponding to the reference image at the corresponding noise description angle is generated. Exemplarily, the multimodal large model can be a Qwen2.5-VL model. By using the multimodal large model and prompts, combined with simple text information about points of interest in the reference image, a more accurate and detailed text description expressing the user's concerns can be generated.

[0065] In this embodiment, by leveraging the user's attention information and a multimodal large model, a comprehensive and key information description (points of interest) is performed on the reference image, so that the generated similar image can not only retain the feature information of the reference image, but also take into account the feature information that the user is interested in.

[0066] For example, the prompt could be designed as: Describe this reference image in detail, focusing on simple textual information.

[0067] For example, as shown in Figure 3, a reference image is presented. The user's focus is on the car on fire in the image. The user inputs the simple text "car on fire," and the prompt is: "Describe this reference image in detail, focusing on the car on fire." When the simple text and reference image are input into a multimodal large model, guided by the prompt, the multimodal large model outputs a detailed text description: "A white sedan is on fire at the edge of a road. The front of the car is engulfed in orange-red flames and thick smoke. The flames are raging, and thick smoke is rising. Several other vehicles are driving normally on the road. The background is an open road, trees, and traffic signs. The daytime lighting highlights the emergency scene of the vehicle on fire."

[0068] One embodiment of this application involves inverting a reference image to obtain a noise map containing feature information of the reference image, including:

[0069] In multiple time steps, the reference image is iteratively denoised to obtain a noise map. Each round of denoising includes the following steps:

[0070] Obtain the gradient vector field at a time step corresponding to the current iteration of noise addition, where the image gradient vector field is generated by the image generation model based on the noisy image after the previous iteration of noise addition;

[0071] The gradient vector field at one time step is used to add noise to the noisy image after the previous iteration to obtain the noisy image after the current iteration.

[0072] The noise map is determined using the noisy image obtained in the last iteration.

[0073] Specifically, multiple time steps are T time steps.

[0074] For the t-th time step, the noisy image latent vector from the (t-1)-th time step and the time step number t-1 are input into the image generation model. The output is the gradient vector field pointing towards the reference image at the t-th time step. Based on the step size difference between two adjacent time steps, the gradient vector field at the t-th time step, and the noisy image latent vector from the (t-1)-th time step, the noisy image latent vector at the t-th time step is determined. This process continues until the noisy image latent vector corresponding to the last time step in T time steps is obtained. The noise map is then determined based on the noisy image latent vector corresponding to the last time step, where t ∈ [1, T].

[0075] For the first time step, the reference image is input into the image generation model, and the gradient vector field of the first time step is output. The noisy image latent vector of the first time step is determined by the gradient vector field of the first time step and the reference image.

[0076] For example, the image generation model includes a text encoder, an image encoder, a diffusion transformer (DiT), and an image decoder, which iteratively adds noise to the reference image at multiple time steps to obtain a noise map, including:

[0077] The reference image is encoded using an image encoder to obtain the image encoding vector of the reference image.

[0078] The image encoding vector and time step number are input into the diffusion transformer, which outputs the gradient vector field of the first time step. The noisy image latent vector of the first time step is determined by the gradient vector field of the first time step and the reference image.

[0079] For the t-th time step, the noisy image latent vector of the (t-1)-th time step and the time step number t-1 are input into the diffusion transformer, and the gradient vector field of the t-th time step is output. Based on the step size difference between two adjacent time steps, the gradient vector field of the t-th time step, and the noisy image latent vector of the (t-1)-th time step, the noisy image latent vector of the t-th time step is determined, until the noisy image latent vector corresponding to the last time step in T time steps is obtained, where 1 < t ≤ T, and T represents the total number of time steps;

[0080] The image decoder decodes the noisy image latent vector corresponding to the last time step and outputs the corresponding noise map.

[0081] One embodiment of this application, matching a detailed text description to denoise a noise map and generate a similar image to a reference image, includes: inputting a detailed text description and a noise map into an image generation model, and performing denoising processing on the noise map under the guidance of the detailed text description to generate a similar image to the reference image.

[0082] The principle of iterative noise addition to the above reference image will be explained below.

[0083] The process of iteratively denoising a random noise image using an image generation model is as follows: Set the number of iterations to T, that is, there are T time steps. In time step t, input the text description, random noise x(t), and time step t into the image generation model, and output the gradient vector field u(t). After calculating the output gradient vector field u(t) using the following formula (1), obtain the noisy image latent vector x(t-1). In time step t-1, input the noisy image latent vector x(t-1) and time step t-1 into the image generation model, and output the gradient vector field u(t-1). Calculate the vector x(t-2) using formula (1). Repeat this process for T iterations. Based on formula (1), x(0) can be obtained. Decode x(0) using the decoder in the image generation model to obtain the image. Formula (1) is expressed as follows:

[0084] ;

[0085] Where t represents the t-th time step, t∈[1,T] This represents the noisy image latent vector at time step t-1. Let represent the noisy image latent vector (i.e., the random noise described above) at time step t. Let represent the gradient vector field at time step t. This represents the difference between two time steps, typically taken as 1 / T. For example, T is set to 1000.

[0086] Based on the above formula (1), formula (2) can be derived in this embodiment. The noisy image latent vector at the t-th time step is determined by formula (2), which is expressed as:

[0087] .

[0088] Where t represents the t-th time step, t∈[1,T] This represents the noisy image latent vector at time step t-1. This represents the noisy image latent vector at time step t. This represents the gradient vector field pointing to the reference image at time step t. This represents the gradient vector field pointing towards the noise direction at time step t. This represents the difference between two time steps, typically taken as 1 / T. For example, T is set to 1000.

[0089] In this embodiment, the image generation model and formula (2) are used to invert the reference image and iteratively add noise to the reference image to obtain a noise map containing reference image information.

[0090] The following example illustrates the process of iteratively adding noise to a reference image to obtain a noise map, combining the image generation model and formula (2) in this embodiment.

[0091] The reference image is encoded by an image encoder to obtain the image encoding vector of the reference image, which is represented as x(0). In the first time step, x(0) and the time step number are input to the diffusion transformer, and the initial gradient vector field u(1) is output. x(0) and u(1) are substituted into formula (2) to calculate the noisy image latent vector x(1) in the first time step. In the second time step, x(1) and the time step number are input to the diffusion transformer, and the gradient vector field u(2) is output. x(1) and u(2) are substituted into formula (2) to calculate the noisy image latent vector x(2) in the second time step. In the third time step, x(2) and the time step number are input to the diffusion transformer, and the gradient vector field u(3) is output. x(2) and u(3) are substituted into formula (2) to calculate the noisy image latent vector x(3) in the third time step. And so on. In the last time step T, x(T-1) and u(T) are substituted into formula (2) to calculate the noisy image latent vector x(T). The image decoder decodes x(T) to obtain the corresponding noise map.

[0092] For example, as shown in Figure 4, the image generation model includes a text encoder, an image encoder, a diffusion transformer, and an image decoder. It matches detailed text descriptions, denoises noisy images, and generates similar images to the reference image, including:

[0093] S401, through a text encoder, semantically encodes the detailed text description to obtain the text semantic vector corresponding to the detailed text description;

[0094] S402. The image encoder performs image encoding on the noise map to obtain the image encoding vector corresponding to the noise map;

[0095] S403, input the image encoding vector and text semantic vector into the diffusion transformer, and perform iterative denoising based on the guidance of the text semantic vector to obtain the denoised latent vector;

[0096] S404: The image decoder decodes the denoised latent vectors to obtain a similar image that is similar to the reference image, and the similar image matches the detailed text description.

[0097] In this embodiment, an image generation model is used to progressively denoise the noisy image to reconstruct a similar image to the reference image. This similar image matches the detailed text description of the user's points of interest. The generated image retains the information of the reference image while also taking into account the information of interest to the user, thus enriching the image data of a single scene. Figure 5 shows a schematic diagram of the similar image to the reference image generated in this embodiment.

[0098] Please refer to Figure 6. This application provides an image generation apparatus, which includes:

[0099] The semantic extraction module 601 is used to obtain a reference image and a detailed text description corresponding to the reference image, wherein the detailed text description is used to describe the points of interest in the reference image;

[0100] The image inversion module 602 is used to invert the reference image to obtain a noise map containing the feature information of the reference image;

[0101] Image generation module 603 is used to match detailed text descriptions, denoise the noisy image, and generate a similar image that is similar to the reference image.

[0102] Optionally, the image inversion module 602 is specifically used for:

[0103] In multiple time steps, the reference image is iteratively denoised to obtain a noise map. Each round of denoising includes the following steps:

[0104] Obtain the gradient vector field at one time step corresponding to the current iteration of noise addition, where the gradient vector field is generated by the image generation model based on the noisy image after the previous iteration of noise addition;

[0105] The gradient vector field at one time step is used to add noise to the noisy image after the previous iteration to obtain the noisy image after the current iteration.

[0106] The noise map is determined using the noisy image obtained in the last iteration.

[0107] Optionally, the image inversion module 602 is specifically used for:

[0108] Multiple time steps are T time steps;

[0109] For the t-th time step, the noisy image latent vector from the (t-1)-th time step and the time step number t-1 are input into the image generation model. The output is the gradient vector field pointing towards the reference image at the t-th time step. Based on the step size difference between two adjacent time steps, the gradient vector field at the t-th time step, and the noisy image latent vector from the (t-1)-th time step, the noisy image latent vector at the t-th time step is determined. This process continues until the noisy image latent vector corresponding to the last time step in T time steps is obtained. The noise map is then determined based on the noisy image latent vector corresponding to the last time step, where t ∈ [1, T].

[0110] For the first time step, the reference image is input into the image generation model, and the gradient vector field of the first time step is output. The noisy image latent vector of the first time step is determined by the sum of the first time step and the reference image.

[0111] Optionally, the image generation model includes an image encoder, a diffusion transformer, and an image decoder, and the image inversion module 602 is specifically used for:

[0112] The reference image is encoded using an image encoder to obtain the image encoding vector of the reference image.

[0113] The image encoding vector and the first time step are input into the diffusion transformer, and the gradient vector field of the first time step is output. The noisy image latent vector of the first time step is determined by the gradient vector field of the first time step and the reference image.

[0114] For the t-th time step, the noisy image latent vector of the (t-1)-th time step and the time step number t-1 are input into the diffusion transformer, and the gradient vector field of the t-th time step is output. Based on the step size difference between two adjacent time steps, the gradient vector field of the t-th time step, and the noisy image latent vector of the (t-1)-th time step, the noisy image latent vector of the t-th time step is determined, until the noisy image latent vector corresponding to the last time step in T time steps is obtained.

[0115] The image decoder decodes the noisy image latent vector corresponding to the last time step and outputs the corresponding noise map.

[0116] Optionally, the semantic extraction module 601 is specifically used for:

[0117] Obtain a reference image;

[0118] Responding to simple text input from the user, the simple text information describes the focus of the reference image, including scene information that the user is concerned with;

[0119] Based on simple textual information, semantic features are extracted from the points of interest in the reference image to obtain a detailed textual description of the reference image.

[0120] Optionally, the semantic extraction module 601 is specifically used for:

[0121] The simple text information and reference image are input into a multimodal large model. Guided by prompts for the noise description angle, and combined with at least one adjustment factor among chroma, brightness, illumination, and background changes, a detailed text description corresponding to the reference image at the corresponding noise description angle is generated.

[0122] Optionally, the image generation module 603 is specifically used for:

[0123] Detailed text descriptions and noise maps are input into the image generation model. Guided by the detailed text descriptions, the noise maps are denoised to generate similar images that are similar to the reference image.

[0124] Optionally, the image generation model includes a text encoder, an image encoder, a diffusion transformer, and an image decoder. The image generation module 603 is specifically used for:

[0125] By using a text encoder, the detailed text description is semantically encoded to obtain the text semantic vector corresponding to the detailed text description;

[0126] The image encoder performs image encoding on the noise map to obtain the image encoding vector corresponding to the noise map;

[0127] The image encoding vector and the text semantic vector are input into the diffusion transformer. Guided by the text semantic vector, iterative denoising is performed to obtain the denoised latent vector.

[0128] The denoised latent vectors are decoded by an image decoder to obtain a similar image that is similar to the reference image, and the similar image matches the detailed text description.

[0129] As shown in Figure 7, an electronic device provided in this application includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704. The processor 701, communication interface 702, and memory 703 communicate with each other through the communication bus 704. In implementation, the processor 701, communication interface 702, and memory 703 may also communicate with each other using other connection methods besides the communication bus 704.

[0130] The memory 703 can be used to store computer programs, which may include instructions and data to implement the steps of any of the image generation methods described above. In this embodiment, the memory 703 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical memory, and registers. The memory may include a hard disk and / or RAM.

[0131] Processor 701 can be a general-purpose processor, which can be a processor that performs specific steps and / or operations by reading and executing a computer program (e.g., a computer program) stored in memory (e.g., a memory module). The general-purpose processor may use data stored in memory during the execution of said steps and / or operations. The general-purpose processor can be, for example, but not limited to, a central processing unit (CPU). Furthermore, processor 701 can also be a special-purpose processor, which can be a processor specifically designed to perform specific steps and / or operations. Special-purpose processors can be, for example, but not limited to, ASICs and FPGAs. Additionally, processor 701 can also be a combination of multiple processors, such as a multi-core processor.

[0132] Communication interface 702 may include input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the network device, as well as interfaces for interconnecting the network device with other devices (e.g., network devices). The communication network may be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Communication interface 702 may be a module, circuit, transceiver, or any device capable of enabling communication.

[0133] In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in processor 701. The software modules can reside in readily available storage media in the art, such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (ROM), electrically erasable programmable memory (EPR), or registers. This storage medium is located in memory 703, and processor 701 reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0134] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0135] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0137] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0138] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image generation method, characterized in that, The method includes: acquiring a reference image; responding to simple text information input by a user, the simple text information describing the focus of the reference image, the focus including scene information of interest to the user; inputting the simple text information and the reference image into a multimodal large model, and, guided by prompts for noise description angles, generating a detailed text description of the reference image corresponding to a corresponding noise description angle, incorporating at least one adjustment factor among chroma, brightness, illumination, and background changes, wherein the detailed text description describes the focus of the reference image; inverting the reference image to obtain a noise map containing feature information of the reference image; and matching it to... The detailed text description describes denoising the noise map to generate a similar image to the reference image. The process of inverting the reference image to obtain a noise map containing feature information of the reference image includes: iteratively adding noise to the reference image at multiple time steps to obtain the noise map. Each iteration of noise addition includes the following steps: obtaining the gradient vector field of a time step corresponding to the current iteration, wherein the gradient vector field is generated by the image generation model based on the noisy image after the previous iteration; and using the gradient vector field of a time step to add noise to the noisy image after the previous iteration to obtain the noisy image after the current iteration.

2. The image generation method as described in claim 1, characterized in that, In multiple time steps, the reference image is iteratively denoised to obtain the noise map, including: the multiple time steps are T time steps; for the t-th time step, the noisy image latent vector of the (t-1)-th time step and the time step number t-1 are input into the image generation model, and the gradient vector field pointing to the reference image of the t-th time step is output. Based on the step size difference between two adjacent time steps, the gradient vector field of the t-th time step, and the noisy image latent vector of the (t-1)-th time step, the noisy image latent vector of the t-th time step is determined, until the noisy image latent vector corresponding to the last time step in the T time steps is obtained, and the noise map is determined based on the noisy image latent vector corresponding to the last time step, where t ∈ [1,T], where, for the first time step, the reference image is input into the image generation model, the gradient vector field of the first time step is output, and the noisy image latent vector of the first time step is determined by the gradient vector field of the first time step and the reference image.

3. The image generation method as described in claim 1, characterized in that, The image generation model includes an image encoder, a diffusion transformer, and an image decoder. Iteratively adding noise to the reference image at multiple time steps to obtain the noise map includes: encoding the reference image using the image encoder to obtain an image encoding vector; inputting the image encoding vector and the time step number into the diffusion transformer to output the gradient vector field of the first time step; using the gradient vector field of the first time step and the reference image to determine the noisy image latent vector of the first time step; for the t-th time step, adding noise to the reference image at the (t-1)-th time step... The noisy image latent vector and time step t-1 are input into the diffusion transformer, which outputs the gradient vector field at time step t. Based on the step size difference between two adjacent time steps, the gradient vector field at time step t, and the noisy image latent vector at time step t-1, the noisy image latent vector at time step t is determined, until the noisy image latent vector corresponding to the last time step in T time steps is obtained, where 1 < t ≤ T, and T represents the total number of time steps. The noisy image latent vector corresponding to the last time step is decoded by the image decoder, and the corresponding noise map is output.

4. The image generation method as described in claim 1, characterized in that, Matching the detailed text description to denoise the noise map and generate a similar image to the reference image includes: inputting the detailed text description and the noise map into an image generation model, and, under the guidance of the detailed text description, performing denoising processing on the noise map to generate a similar image to the reference image.

5. The image generation method as described in claim 4, characterized in that, The image generation model includes a text encoder, an image encoder, a diffusion transformer, and an image decoder. Matching the detailed text description, it denoises the noise map to generate a similar image to the reference image. This includes: semantically encoding the detailed text description using the text encoder to obtain a text semantic vector corresponding to the detailed text description; image encoding the noise map using the image encoder to obtain an image encoded vector corresponding to the noise map; inputting the image encoded vector and the text semantic vector into the diffusion transformer, and performing iterative denoising based on the text semantic vector to obtain a denoised latent vector; and decoding the denoised latent vector using the image decoder to generate a similar image to the reference image, wherein the similar image matches the detailed text description.

6. An image generation apparatus, characterized in that, The device includes: a semantic extraction module for acquiring a reference image; responding to simple text information input by a user, the simple text information describing the focus of the reference image, the focus including scene information of interest to the user; inputting the simple text information and the reference image into a multimodal large model, and, guided by prompts for noise description angles, generating a detailed text description of the reference image corresponding to the corresponding noise description angle, combined with at least one adjustment factor among chroma, brightness, illumination, and background changes, wherein the detailed text description describes the focus of the reference image; and an image inversion module for inverting the reference image to obtain a noise map containing feature information of the reference image. The image generation module is used to match the detailed text description, denoise the noise map, and generate a similar image to the reference image. The inversion of the reference image to obtain a noise map containing feature information of the reference image includes: iteratively adding noise to the reference image at multiple time steps to obtain the noise map. Each round of iterative noise addition includes the following steps: obtaining the gradient vector field of one time step corresponding to the current round of iterative noise addition, wherein the gradient vector field is generated by the image generation model based on the noisy image after the previous round of iterative noise addition; using the gradient vector field of one time step, adding noise to the noisy image after the previous round of iterative noise addition to obtain the noisy image after the current round of iterative noise addition.

7. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the image generation method according to any one of claims 1 to 5.

Images