Image generation method and apparatus, electronic device, storage medium, and computer program product

Abstract

This application provides an image generation method, apparatus, electronic device, storage medium, and computer program product, relating to the technical field of image processing. The method includes: responding to user request information, generating a background image based on spatial dimension requirements for a target image; wherein the spatial dimension requirements are derived from the user request information; and generating a foreground image by combining the spatial dimension requirements and pose dimension requirements for the background image to generate the target image. The solution in this application embodiment can improve the quality of the target image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image generation method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] In related technologies, the Stable Diffusion model is now widely used in the field of Artificial Intelligence (AI) image generation. Industry-leading examples such as NovelAI are based on implementations of the Stable Diffusion model. Because the code and paper for Stable Diffusion are open-source, it is easier for creators to develop more plugins and implement more applications. Currently, the Stable Diffusion model performs very well when generating images with a resolution less than 768*768 pixels. However, when generating very large images (images with a width and height of 768 pixels or more), due to model and algorithm characteristics, the generated images may contain abnormal image blocks such as multiple human figures, multiple heads, or too many objects, resulting in lower quality of the generated very large images. Summary of the Invention

[0003] The embodiments of this application provide an image generation method, apparatus, electronic device, storage medium, and computer program product that can improve the quality of generated ultra-large images.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides an image generation method, including:

[0006] In response to user request information, a background image is generated based on the spatial dimension requirements of the target image; wherein, the spatial dimension requirements are derived from the user request information.

[0007] For the background image, a foreground is generated by combining the spatial dimension requirements and the pose dimension requirements to determine the target image; wherein, the pose dimension requirements are derived from the user requirement information.

[0008] In the above scheme, the spatial dimension requirements include: background and foreground requirements in the hierarchical dimension, and requirements for each sub-region of the target image in the planar dimension; the pose dimension requirements include: pose requirements for each object in the foreground; the step of generating a background image in response to user request information based on the spatial dimension requirements of the target image includes:

[0009] In response to the user's request information, the background image is generated by combining the background requirements and the requirements of each of the sub-regions.

[0010] The step of generating a foreground based on the background image, combined with the spatial dimension requirements and pose dimension requirements, to determine the target image includes:

[0011] Based on the background image, the foreground is generated by combining the foreground requirements, the requirements of each sub-region, and the pose requirements, thereby generating the target image.

[0012] In the above solution, the step of responding to the user's request information and generating the background image by combining the background requirements and the requirements of each of the sub-regions includes:

[0013] In response to the user's request information, a white noise image is generated;

[0014] The white noise image is denoised using each of the first network layers in the background image generation model to determine the background image.

[0015] The output of each of the first network layers is constrained by a first matrix and a second matrix; the first matrix is ​​used to characterize the requirements of each of the sub-regions, and the second matrix is ​​used to characterize the background requirements; the first matrix and the second matrix are derived from the user requirement information.

[0016] In the above scheme, the step of using each first network layer in the background image generation model to denoise the white noise image and determine the background image includes:

[0017] The white noise image is compressed using the background image generation model to form a first image feature;

[0018] The first image feature is sliced ​​to form multiple sub-image features; wherein, the sub-image features have a one-to-one correspondence with the sub-region requirements;

[0019] By combining the constraints of the first matrix and the second matrix, the background image is generated by denoising multiple sub-image features using each of the first network layers.

[0020] The method in the above scheme further includes:

[0021] Based on a preset language model, the user's needs information is processed to determine the prompt information and background prompt information for each sub-area;

[0022] The first matrix corresponding to each sub-region is determined by transforming the prompt information of each sub-region, and the second matrix is ​​determined by transforming the background prompt information.

[0023] In the above scheme, the step of generating the foreground based on the background image, combined with the foreground requirements, the requirements of each sub-region, and the pose requirements, to determine the target image includes:

[0024] The background image is compressed using a foreground image generation model to form a second image feature;

[0025] The target image is determined by redrawing the foreground of the second feature map using each of the second network layers in the foreground generation model.

[0026] The output of each second network layer is constrained by a first matrix, a third matrix, and a fourth matrix; the third matrix is ​​used to characterize the foreground requirement, and the fourth matrix is ​​used to characterize the pose requirement; the third matrix and the fourth matrix are derived from the user requirement information.

[0027] The method in the above scheme further includes:

[0028] Based on a preset language model, the user's demand information is processed to determine foreground and posture prompts.

[0029] The foreground cue information is transformed to determine the third matrix, and the pose cue information is transformed to determine the fourth matrix.

[0030] This application also provides an image generation apparatus, including:

[0031] An image generation unit is used to respond to user request information and generate a background image based on the spatial dimension requirements of the target image; wherein, the spatial dimension requirements are derived from the user request information.

[0032] An image generation unit is used to generate a foreground image based on the background image, combining the spatial dimension requirements and the pose dimension requirements, to determine the target image; wherein the pose dimension requirements are derived from the user requirement information.

[0033] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps in the above-described method.

[0034] This application also provides a computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps in the above-described method.

[0035] This application also provides a computer program product, including a computer program, characterized in that the computer program implements the steps in the above method when executed by a processor.

[0036] In this embodiment, responding to user request information, a background image is generated based on the spatial dimension requirements of the target image; wherein the spatial dimension requirements are derived from the user request information. For the background image, a foreground image is generated by combining the spatial dimension requirements and the pose dimension requirements; wherein the pose dimension requirements are derived from the user request information. Because this embodiment considers both spatial and pose dimension requirements during the target image generation process, it is more comprehensive and avoids situations where poor image quality is caused by certain factors during image generation. Furthermore, in this embodiment, the background image is generated first, and then the target image is generated based on the background image. This step-by-step generation process avoids the shortcomings of image model algorithms in generating only a single background or foreground, reduces the probability of image anomalies, and thus improves the quality of the target image. Attached Figure Description

[0037] Figure 1 A schematic diagram of an optional process for an image generation method provided in an embodiment of this application;

[0038] Figure 2 A schematic diagram of an optional process for an image generation method provided in an embodiment of this application;

[0039] Figure 3 A schematic diagram of an optional process for an image generation method provided in an embodiment of this application;

[0040] Figure 4 A schematic diagram of an optional process for an image generation method provided in an embodiment of this application;

[0041] Figure 5 A schematic diagram of an optional process for an image generation method provided in an embodiment of this application;

[0042] Figure 6 A schematic diagram of an optional process for an image generation method provided in an embodiment of this application;

[0043] Figure 7 This is a schematic diagram of the structure of the image generation apparatus provided in the embodiments of this application;

[0044] Figure 8 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0047] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0049] Among related technologies, the following four image generation schemes based on Stable Diffusion are currently commonly used in the industry:

[0050] 1. Super-Resolution (SR). First, a small image is generated using Stable Diffusion, then a super-resolution algorithm is used to improve the resolution of the small image. However, super-resolution primarily adds image details and does not actually increase the image content. Furthermore, super-resolution requires a large amount of GPU memory, resulting in very high space costs when generating extremely large images.

[0051] 2. SDXL (Stable Diffusion XL). A derivative version of the Stable Diffusion model. SDXL significantly improves the quality of generated images (1024*1024) by concatenating two UNet (U-shaped network) models and using a higher number of UNet model parameters. However, SDXL uses a completely different model structure than other Stable Diffusion versions, resulting in lower generalizability compared to other SD versions. Furthermore, at resolutions higher than 1024*1024, the generated images still exhibit anomalies such as multiple human figures. Essentially, SDXL does not solve the problem of generating extremely large images using Stable Diffusion.

[0052] 3. MultiDiffusion: This algorithm slices a large image, uses a pre-trained Stable Diffusion model to generate denoised slices from each slice, and finally combines all the denoised slices to generate a complete, denoised, ultra-large image. This algorithm performs well in generating backgrounds for large images, but it produces abnormal human figures and poor results when rendering foregrounds, especially those with multiple subjects (e.g., two people).

[0053] 4. Mixture of Diffusers: This algorithm predicts noise in different image regions through a diffusion process, then mixes the predicted noise to obtain a complete image. The drawback of this algorithm is that the individual diffusion processes are not interconnected, resulting in a strong sense of fragmentation between image patches and a generally unnatural appearance in the final image. Furthermore, it has poor background compositing capabilities when generating large images, easily resulting in poor-quality backgrounds.

[0054] To address the aforementioned problems, this application provides an image generation method. Please refer to [link to relevant documentation]. Figure 1 This is an optional flowchart illustrating an image generation method provided in an embodiment of this application, which will be combined with... Figure 1 The steps shown are explained below:

[0055] S101. Responding to user request information, generate a background image based on the spatial dimension requirements for the target image; wherein, the spatial dimension requirements are derived from the user request information.

[0056] In this embodiment, the image generation device can obtain user requirement information through a human-computer interaction device. Based on the user requirement information, the hierarchical and planar dimensions of the corresponding target image are determined. The generation device can then generate a background image by combining the spatial dimension requirements (hierarchical and planar dimension requirements) with a background generation model.

[0057] The user requirement information describes the display content within the target image. The image generation device can be a terminal, mobile terminal, or server with corresponding image processing capabilities. The hierarchical dimension can include at least one of the following: foreground, midground, and background dimensions. The planar dimension can include dimensions on each sub-region of the target image. The pose dimension can include the pose dimensions of each object in the foreground. The spatial dimension requirements can include constraint matrices generated based on the user requirement information to constrain background image generation and foreground generation on the planar and hierarchical dimensions. The image generation device can use an image generation model combined with the above constraint matrices to generate the background image and the foreground based on the background image.

[0058] The hierarchical dimension requirements include: background requirements and foreground requirements; the planar dimension requirements include: the requirements of each sub-region of the target image; and the pose dimension requirements include: the pose requirements of each object in the foreground.

[0059] In this embodiment, the generation device analyzes and determines the background requirements and requirements for each sub-region of the corresponding target image based on user demand information. The generation device can generate a background image by combining the background requirements and the constraints of each sub-region requirement using a background generation model.

[0060] S102. Based on the background image, generate the foreground by combining spatial dimension requirements and pose dimension requirements to determine the target image.

[0061] In this embodiment, the generation device analyzes and determines the hierarchical, planar, and pose dimensions of the target image based on user requirement information. The generation device can use a foreground generation model combined with the hierarchical, planar, and pose dimensions to generate the foreground of the background image, thereby determining the target image; wherein the pose dimension requirement originates from the user requirement information. The pose dimension requirement can be a constraint matrix generated based on the user requirement information to constrain the poses of various objects in the foreground. The generation device can use an image generation model set to utilize this constraint matrix to generate the foreground of the background image, thereby determining the target image.

[0062] In this embodiment, the generation device analyzes and determines the foreground requirements, requirements for each sub-region, and pose requirements of the corresponding target image based on user requirement information. The generation device can generate the foreground and determine the target image by combining the constraints of the foreground requirements, requirements for each sub-region, and pose requirements using a foreground generation model.

[0063] In this embodiment, responding to user request information, a background image is generated based on the spatial dimension requirements of the target image; wherein, the spatial dimension requirements are derived from the user request information; for the background image, foreground generation is performed by combining the spatial dimension requirements and the pose dimension requirements to determine the target image. Because this embodiment considers at least two dimensions in the process of generating the target image, it is more comprehensive and avoids situations where poor image quality occurs due to the influence of certain factors during image generation. Furthermore, in this embodiment, the background image is generated first, and then the target image is generated based on the background image. This step-by-step generation process avoids the shortcomings of image model algorithms in generating a single background or foreground, reduces the probability of image anomalies, and thus improves the quality of the target image.

[0064] Please see Figure 2 This is a schematic diagram of an optional process for the image generation method provided in an embodiment of this application. Figure 1 S101 to S102 can also be implemented via S201 to S202, which will be explained in conjunction with the steps:

[0065] S201. In response to the user's request information, the background image is generated by combining the background request and the requests of each of the sub-regions.

[0066] In this embodiment, the generation device responds to user request information, extracts and determines background prompt information and sub-region prompt information for each sub-region of the target image based on the user request information, transforms the background prompt information to determine background requirements, and transforms the sub-region prompt information to determine the requirements for each sub-region. The generation device can also use the background requirements and the requirements for each sub-region as constraints for the background image generation model to generate a background image.

[0067] The background requirements and the requirements of each sub-region are derived from the user requirement information. The background requirements can be a constraint matrix generated based on the user requirement information to constrain the relevant information of the background image, and the requirements of each sub-region can be the features of each sub-region generated based on the user requirement information, and the constraint matrix corresponding to each sub-region transformed for each sub-region feature.

[0068] S202. Based on the background image, the foreground is generated by combining the foreground requirements, the requirements of each sub-region, and the pose requirements, and the target image is determined.

[0069] In this embodiment, the generation device responds to user request information, extracts and determines foreground prompt information and pose prompt information for each foreground object, transforms the foreground prompt information to determine foreground requirements, and transforms the pose prompt information to determine pose requirements. The generation device can also use the foreground requirements, pose requirements, and requirements of each sub-region as constraints for the foreground image generation model to generate a background image.

[0070] The foreground requirements and the requirements of each sub-region are derived from the user requirement information. The background requirements can be a constraint matrix generated based on the user requirement information to constrain foreground-related information.

[0071] In this embodiment, the foreground image generation model and the background image generation model can share the same UNet model. The UNet model first generates a background image based on the requirements of each sub-region and the background constraint, and then generates the foreground image by adding foreground and pose constraints, thus determining the target image. The UNet model is a fully convolutional neural network model. Its architecture is U-shaped, hence its name.

[0072] In this embodiment, responding to user requirements, a background image is generated by combining background requirements and requirements for each sub-region. Based on the background image, foreground requirements, requirements for each sub-region, and pose requirements are combined to generate the foreground image, thus determining the target image. Because this embodiment considers background requirements, requirements for each sub-region, foreground requirements, and pose requirements during the target image generation process, it is more comprehensive and avoids situations where poor image quality is caused by certain factors during image generation. Furthermore, in this embodiment, the background image is generated first, and then the target image is generated based on the background image. This step-by-step generation process avoids the shortcomings of image model algorithms in generating only a single background or foreground, reduces the probability of image anomalies, and thus improves the quality of the target image.

[0073] Please see Figure 3 This is a schematic diagram of an optional process for the image generation method provided in an embodiment of this application. Figure 2 S201 can also be implemented through S301 to S302, which will be explained in conjunction with the steps:

[0074] S301. In response to the user request information, generate a white noise image.

[0075] In this embodiment of the application, the user requirement information may include the size information of the target image, and the generating device can generate a corresponding white noise image based on the size information.

[0076] In this embodiment, the generating device can use the MultiDiffusion algorithm to generate a white noise image based on size information. In other embodiments, the generating device can also use other algorithms to generate white noise images; this embodiment does not impose specific limitations.

[0077] S302. The white noise image is denoised using each of the first network layers in the background image generation model to determine the background image; wherein the output of each of the first network layers is constrained by a first matrix and a second matrix; the first matrix is ​​used to characterize the requirements of each of the sub-regions, and the second matrix is ​​used to characterize the background requirements; the first matrix and the second matrix are derived from the user requirement information.

[0078] In this embodiment, the image generation device can extract background prompt information and sub-region prompt information corresponding to each sub-region from the user demand information in advance. The prompt information for each sub-region is transformed to determine a first matrix, and the background prompt information is transformed to determine a second matrix. Using the first and second matrices as constraints for each first network layer of the background generation model, a white noise image is input into the background image generation model, and the background image is determined after denoising processing by each first network layer.

[0079] In this embodiment, the generation device can process the user requirement information based on a preset language model to determine the prompt information and background prompt information for each sub-region. The prompt information for each sub-region is transformed to determine the first matrix corresponding to each sub-region, and the background prompt information is transformed to determine the second matrix. The preset language model may include the Clip model; in other embodiments, other language models may be used, and this embodiment does not impose specific limitations.

[0080] In this embodiment, the generation device can process user requirement information based on a preset language model, and leverage the inductive and logical reasoning capabilities of multiple network layers within the preset language model to determine corresponding background prompt information based on the background image. The generation device can also process user requirement information based on a preset language model, and leverage the inductive and logical reasoning capabilities of multiple network layers within the preset language model to determine corresponding prompt information for each sub-region based on the user requirement information.

[0081] The pre-defined language model includes the following network layers:

[0082] User layer: Users interact directly with the application and use the language model capabilities provided by the application to complete tasks.

[0083] Application layer: Includes libraries, tools and application code, builds out-of-the-box general solutions based on underlying capabilities, and also supports some configuration capabilities.

[0084] Optimization layer: Optimizes certain performance characteristics of the language model or language model system based on indicators.

[0085] Control layer: Supports common control flows in the framework, such as conditional, loop, and dispatch, enabling more complex tasks.

[0086] Prompt Constraint Layer: Applies prompt rules or structural requirements, as well as output constraints and validation.

[0087] Prompt layer: Use text to call the Application Programming Interface (API), chat manually, or other forms of interaction, with no restrictions on text input and output.

[0088] Neural network layers: These layers provide direct access to the underlying architecture of the language model, including weight parameters and encoding / decoding components. These hierarchical structures help in understanding the interactions and functions of large language models at different levels.

[0089] In this embodiment, a white noise image is generated in response to user request information. The white noise image is denoised using the first network layers of the background image generation model to determine the background image. The output of each first network layer is constrained by a first matrix and a second matrix. The first matrix represents the requirements of each sub-region, and the second matrix represents the background requirements. Both the first and second matrices originate from the user request information. Because this embodiment utilizes a background image generation model that combines background requirements and the requirements of each sub-region to generate the background image, it is more comprehensive and avoids situations where poor quality occurs due to certain factors during background image generation, thus improving the quality of the target image background image.

[0090] Please see Figure 4 This is a schematic diagram of an optional process for the image generation method provided in an embodiment of this application. Figure 3 S302 in the above can also be implemented through S401 to S403, which will be explained in conjunction with the steps:

[0091] S401. The white noise image is encoded and compressed using the background image generation model to form a first image feature.

[0092] In this embodiment of the application, the generating device can use the encoder in the background image generation model to encode and compress a white noise image to generate a first image feature.

[0093] In this embodiment, the background image generation model and the foreground image generation model of the generation device can share a single encoder. The background image generation model uses the encoder to compress the white noise image into the latent space to form the first image feature.

[0094] The background image generation model may include the MultiDiffusion algorithm model. In other embodiments, the background image generation model may also include other algorithm models. No specific limitations are made in this application embodiment.

[0095] S402. The first image feature is sliced ​​to form multiple sub-image features; wherein, the sub-image features have a one-to-one correspondence with the sub-region requirements.

[0096] In this embodiment of the application, the generating device can use a background image generation model to form multiple sub-image features from the first image feature slices. The sub-image features have a one-to-one correspondence with the sub-region requirements.

[0097] S403. Combining the constraints of the first matrix and the second matrix, the first network layers are used to denoise the multiple sub-image features to generate the background image.

[0098] In this embodiment, the generating device uses a first matrix and a second matrix as constraints for each first network layer, performs denoising processing on multiple sub-image features using each first network layer, and finally uses a decoder to decode the output features to generate a background image.

[0099] The first matrix and the second matrix serve as constraints for the first network layer, and can be used to influence the parameter matrix of each first network layer.

[0100] In this embodiment, the background image generation model and the foreground image generation model can share the same encoder and decoder.

[0101] Similarly, since the embodiments of this application utilize a background image generation model that combines background requirements and the requirements of each sub-region to generate a background image, it takes a more comprehensive approach and avoids the occurrence of poor quality due to the influence of certain factors during the background image generation process, thereby improving the quality of the target image background image.

[0102] Please see Figure 5 This is a schematic diagram of an optional process for the image generation method provided in an embodiment of this application. Figure 2 S202 in the above can also be implemented through S501 to S502, which will be explained in conjunction with the steps:

[0103] S501. The background image is encoded and compressed using a foreground image generation model to form a second image feature.

[0104] In this embodiment of the application, the generating device can use the encoder in the foreground image generation model to encode and compress the background image to generate a second image feature.

[0105] In this embodiment, the foreground image generation model and the foreground image generation model of the generation device can share a single encoder. The foreground image generation model uses the encoder to compress the background image into the latent space to form a second image feature.

[0106] The foreground image generation model may include the Mixture of Diffusers algorithm model. In other embodiments, the foreground image generation model may also include other algorithm models. No specific limitations are made in this application embodiment.

[0107] S502. The second feature map is redrawn using each of the second network layers in the foreground generation model to determine the target image; wherein the output of each of the second network layers is constrained by a first matrix, a third matrix, and a fourth matrix; the third matrix is ​​used to characterize the foreground requirement, and the fourth matrix is ​​used to characterize the pose requirement; the third matrix and the fourth matrix are derived from the user requirement information.

[0108] In this embodiment, the image generation device can extract foreground cues and pose cues from the user's requirement information in advance. The foreground cues are transformed to determine a third matrix, and the pose cues are transformed to determine a fourth matrix. Using the first, third, and fourth matrices as constraints for each first network layer of the foreground generation model, the second image features are input into the background image generation model. After foreground redrawing and outputting feature maps through each second network layer, the target image is then decoded by a decoder.

[0109] In this embodiment of the application, the generation device can process the user demand information based on a preset language model to determine foreground prompt information and posture prompt information; transform the foreground prompt information to determine the third matrix, and transform the posture prompt information to determine the fourth matrix.

[0110] Among them, the fourth matrix, as a pose constraint, can be applied to the human pose constraint when generating images of each region.

[0111] In this embodiment, the generation device can process user requirement information based on a preset language model, and leverage the inductive and logical reasoning capabilities of multiple network layers within the preset language model to determine foreground prompt information based on the foreground graph. The generation device can also process user requirement information based on a preset language model, and leverage the inductive and logical reasoning capabilities of multiple network layers within the preset language model to determine posture prompt information based on each object in the foreground.

[0112] In this embodiment, a foreground image generation model is used to encode and compress the background image, forming a second image feature. The second feature map is then redrawn using each second network layer in the foreground generation model to determine the target image. The output of each second network layer is constrained by a first matrix, a third matrix, and a fourth matrix. The third matrix represents foreground requirements, and the fourth matrix represents pose requirements. The third and fourth matrices are derived from user requirement information. Because this embodiment utilizes a foreground image generation model that combines foreground requirements, requirements for each sub-region, and pose requirements to generate the background image, it provides a more comprehensive approach, avoids interference from undesirable factors during foreground generation, and improves the quality of the target image.

[0113] Please see Figure 6 The solution of this application will now be described with reference to a specific embodiment:

[0114] Area suggestion words:

[0115] In Stable Diffusion, the cue words are input into the Clip language model to generate a cue word feature. This feature is then embedded into each layer of the UNet through an attention mechanism, ultimately achieving the effect of additional input constraints. In region-based cue words, different cue words can be input to different regions of the image, simultaneously generating an influence matrix. This matrix represents the image region affected by the cue word feature. The generated cue word feature and influence matrix are then used together as input to the UNet in the noise reduction process.

[0116] MultiDiffusion module:

[0117] We first use the MultiDiffusion algorithm to generate a background image. First, we generate a white noise image of the same size as the target image, then use an autoencoder to encode and compress the noise image into the latent space. Within the latent space, we use an algorithm to slice the image, and then input these slices into UNet for prediction and noise removal. Simultaneously, we extract features from the region cue words used as the background using a Clip model, input these cue word features into UNet, and then use a decoder to decode and generate the background image. This background image is the same size as the target image.

[0118] Mixture of Diffusers module:

[0119] Similar to the MultiDiffusion module, the Mixture of Diffusers module uses the same autoencoder, decoder, and UNet to save memory. After generating the background image, we use it as input to the autoencoder to generate the latent space. Then, we input the foreground region cue words into the Clip model for processing, generating region cue word features.

[0120] Then we will redraw the foreground area in the background image. The foreground area will undergo a new diffusion denoising process based on the background image, and a small amount of noise will also be added at the boundary between the background and foreground to improve the overall smoothness of the image.

[0121] Furthermore, OpenPose in ControlNet performs very well in constraining human pose during Stable Diffusion full-image generation. However, OpenPose currently lacks implementation for individual image blocks. Therefore, we added the OpenPose module to the Mixture of Diffusers module to achieve human pose constraints for individual image blocks, helping users better achieve portrait effects. OpenPose's input is an image plus pose constraint prompts. The original Stable Diffusion module includes OpenPose, but it constrains the entire image. We made some adjustments in the project to allow OpenPose to apply human pose constraints during image generation for each region.

[0122] Please see Figure 7 This is a schematic diagram of the structure of the image generation device provided in the embodiments of this application.

[0123] This application also provides an image generation apparatus 800, including an image generation unit 801.

[0124] Image generation unit 801 is used to respond to user demand information and generate a background image based on the spatial dimension requirements of the target image; wherein, the spatial dimension requirements are derived from the user demand information.

[0125] Image generation unit 801 is used to generate the target image by combining the spatial dimension requirements and pose dimension requirements with the background image; wherein the pose dimension requirements are derived from the user requirement information.

[0126] In this embodiment of the application, the spatial dimension requirements include: background and foreground requirements in the hierarchical dimension, and requirements for each sub-region of the target image in the planar dimension; the pose dimension requirements include: pose requirements for each object in the foreground; the image generation unit 801 in the image generation device 800 is used to respond to the user requirement information, and generate the background image by combining the background requirements and the requirements for each sub-region.

[0127] Based on the background image, the foreground is generated by combining the foreground requirements, the requirements of each sub-region, and the pose requirements, thereby generating the target image.

[0128] In this embodiment of the application, the image generation unit 801 in the image generation apparatus 800 is used to generate a white noise image in response to the user demand information;

[0129] The white noise image is denoised using each of the first network layers in the background image generation model to determine the background image.

[0130] The output of each of the first network layers is constrained by a first matrix and a second matrix; the first matrix is ​​used to characterize the requirements of each of the sub-regions, and the second matrix is ​​used to characterize the background requirements; the first matrix and the second matrix are derived from the user requirement information.

[0131] In this embodiment of the application, the image generation unit 801 in the image generation device 800 is used to encode and compress the white noise image using the background image generation model to form a first image feature;

[0132] The first image feature is sliced ​​to form multiple sub-image features; wherein, the sub-image features have a one-to-one correspondence with the sub-region requirements;

[0133] By combining the constraints of the first matrix and the second matrix, the background image is generated by denoising multiple sub-image features using each of the first network layers.

[0134] In this embodiment of the application, the image generation unit 801 in the image generation device 800 is used to process the user demand information based on a preset language model and determine the prompt information and background prompt information for each sub-region.

[0135] The first matrix corresponding to each sub-region is determined by transforming the prompt information of each sub-region, and the second matrix is ​​determined by transforming the background prompt information.

[0136] In this embodiment of the application, the image generation unit 801 in the image generation apparatus 800 is used to encode and compress the background image using a foreground image generation model to form a second image feature;

[0137] The target image is determined by redrawing the foreground of the second feature map using each of the second network layers in the foreground generation model.

[0138] The output of each second network layer is constrained by a first matrix, a third matrix, and a fourth matrix; the third matrix is ​​used to characterize the foreground requirement, and the fourth matrix is ​​used to characterize the pose requirement; the third matrix and the fourth matrix are derived from the user requirement information.

[0139] In this embodiment of the application, the image generation unit 801 in the image generation device 800 is used to process the user demand information based on a preset language model and determine foreground prompt information and posture prompt information;

[0140] The foreground cue information is transformed to determine the third matrix, and the pose cue information is transformed to determine the fourth matrix.

[0141] It should be noted that, in the embodiments of this application, if the above-described image generation method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an image generation device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0142] Correspondingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method on one side of an image generation apparatus.

[0143] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0144] It should be noted that, Figure 8 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 8As shown, this application embodiment provides an electronic device 900, including a memory 902 and a processor 901. The memory 902 stores a computer program that can run on the processor 901. When the processor 901 executes the program, it implements the steps in the above-described method, wherein;

[0145] Processor 901 typically controls the overall operation of electronic device 900.

[0146] The memory 902 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 901 and various modules in the electronic device 900. It can be implemented by flash memory or random access memory (RAM).

[0147] Correspondingly, this application also provides a computer program product, including a computer program that can be executed by the processor 901 of the electronic device 900 to complete the steps in the method of the image generation apparatus 800.

[0148] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0149] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0151] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0152] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0153] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0154] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0155] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An image generation method, characterized in that, include: In response to user request information, a background image is generated based on the spatial dimension requirements of the target image; wherein, the spatial dimension requirements are derived from the user request information. For the background image, a foreground is generated by combining the spatial dimension requirements and the pose dimension requirements to determine the target image; wherein, the pose dimension requirements are derived from the user requirement information.

2. The image generation method according to claim 1, characterized in that, The spatial dimension requirements include: background and foreground requirements in the hierarchical dimension, and requirements for each sub-region of the target image in the planar dimension; the pose dimension requirements include: pose requirements for each object in the foreground; the step of generating a background image in response to user request information, based on the spatial dimension requirements of the target image, includes: In response to the user's request information, the background image is generated by combining the background requirements and the requirements of each of the sub-regions. The step of generating a foreground based on the background image, combined with the spatial dimension requirements and pose dimension requirements, to determine the target image includes: Based on the background image, the foreground is generated by combining the foreground requirements, the requirements of each sub-region, and the pose requirements, thereby generating the target image.

3. The image generation method according to claim 2, characterized in that, The process of responding to the user's request information, combining the background requirements and the requirements of each of the sub-regions, to generate the background image includes: In response to the user's request information, a white noise image is generated; The white noise image is denoised using each of the first network layers in the background image generation model to determine the background image. The output of each of the first network layers is constrained by a first matrix and a second matrix; the first matrix is ​​used to characterize the requirements of each of the sub-regions, and the second matrix is ​​used to characterize the background requirements; the first matrix and the second matrix are derived from the user requirement information.

4. The image generation method according to claim 3, characterized in that, The step of denoising the white noise image using each first network layer in the background image generation model to determine the background image includes: The white noise image is compressed using the background image generation model to form a first image feature; The first image feature is sliced ​​to form multiple sub-image features; wherein, the sub-image features have a one-to-one correspondence with the sub-region requirements; By combining the constraints of the first matrix and the second matrix, the background image is generated by denoising multiple sub-image features using each of the first network layers.

5. The image generation method according to claim 3, characterized in that, The method further includes: Based on a preset language model, the user's needs information is processed to determine the prompt information and background prompt information for each sub-area; The first matrix corresponding to each sub-region is determined by transforming the prompt information of each sub-region, and the second matrix is ​​determined by transforming the background prompt information.

6. The image generation method according to claim 2, characterized in that, The step of generating the foreground based on the background image, combining the foreground requirements, the requirements of each sub-region, and the pose requirements, to determine the target image includes: The background image is compressed using a foreground image generation model to form a second image feature; The target image is determined by redrawing the foreground of the second feature map using each of the second network layers in the foreground generation model. The output of each second network layer is constrained by a first matrix, a third matrix, and a fourth matrix; the third matrix is ​​used to characterize the foreground requirement, and the fourth matrix is ​​used to characterize the pose requirement; the third matrix and the fourth matrix are derived from the user requirement information.

7. An image generation apparatus, characterized in that, include: An image generation unit is used to respond to user request information and generate a background image based on the spatial dimension requirements of the target image; wherein, the spatial dimension requirements are derived from the user request information. An image generation unit is used to generate a foreground image based on the background image, combining the spatial dimension requirements and the pose dimension requirements, to determine the target image; wherein the pose dimension requirements are derived from the user requirement information.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images