Image generation method and device and electronic equipment

Abstract

The invention discloses an image generation method and device and electronic equipment. The method comprises the steps of obtaining a user prompt word, performing part-of-speech analysis on the user prompt word, and determining an image analysis result corresponding to a to-be-generated image according to a part-of-speech analysis result; obtaining a background cue word, analyzing the image analysis result and the background cue word by using a layout generation model to obtain a target image feature, and performing feature coding and fusion on entity information, background information and global information of the to-be-generated image by using the layout generation model through a variational auto-encoder to obtain a target image feature; the target loss function of the layout generation model at least comprises an activation loss function and a semantic loss function; and generating an image according to the target image feature. The technical problem that an image cannot be directly generated based on a layout demand and a specified element of a user when an image generation method in related technologies processes complex layout and multi-text description is solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image generation method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of deep learning technology, the field of image generation has ushered in unprecedented innovation, especially with the application of diffusion models and autoregressive generative networks, which have significantly enhanced the ability to generate images from text. Diffusion models, by simulating the diffusion process of image data, can generate synthetic images that are highly similar to real images, while autoregressive generative networks, such as VAEs (Variational Autoencoders), achieve efficient compression and high-quality reconstruction of image information by encoding and decoding images, providing powerful algorithmic support for image generation.

[0003] In this technological wave, traditional image generation methods mostly rely on a single text description; given a descriptive text, the model generates an image that matches that description. However, this generation approach often lacks precise control over image details, especially when multiple elements need to be arranged in specific positions. In such cases, the model is prone to inaccurate spatial positioning and confusion regarding element attributes. In summary, while methods that generate layouts based on existing images or generate new images based on image features provide some guidance, they cannot directly respond to user-provided layouts and specific element descriptions, limiting the diversity and controllability of the generated images.

[0004] Furthermore, current text-to-image models often face limitations in semantic understanding and spatial localization in handling multi-text layout control tasks. Text encoders may struggle to distinguish subtle differences between various text descriptions, leading to generated images that deviate from expectations in terms of element position, pose, and color. Simultaneously, the lack of effective mechanisms to guide the model to focus on fine-grained regions of the image can result in attention leakage when generating specific entities, hindering precise control over entity generation regions and impacting the overall quality and effectiveness of image generation.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides an image generation method, apparatus, and electronic device to at least solve the technical problem that image generation methods in the related art cannot directly generate images based on user layout requirements and specified elements when dealing with complex layouts and multi-text descriptions.

[0007] According to one aspect of the embodiments of this application, an image generation method is provided, comprising: acquiring user prompt words, performing part-of-speech parsing on the user prompt words, and determining an image parsing result corresponding to the image to be generated based on the part-of-speech parsing result; acquiring background prompt words, and analyzing the image parsing result and background prompt words using a layout generation model to obtain target image features, wherein the layout generation model performs feature encoding and fusion of entity information, background information and global information of the image to be generated through a variational autoencoder, and the target loss function of the layout generation model includes at least an activation loss function for enhancing the correlation between entities and corresponding positions in the image to be generated, and a semantic loss function for enhancing the correlation between entities and corresponding attribute descriptions in the image to be generated; and generating an image based on the target image features.

[0008] Optionally, the layout generation model is trained as follows: Historical generated images are acquired; the historical generated images are processed based on background prompts and a predefined thought chain to obtain image description information. The predefined thought chain defines the image description format of the historical generated images. The image description information includes entity description information, background description information, global description information, and a segmentation mask. The entity description information includes entity text features and entity image features of the historical generated images; the background description information includes background text features and background image features of the historical generated images; the global description information includes global text features and global image features of the historical generated images; and the segmentation mask is used to distinguish entity positions in the historical generated images. The historical generated images and their corresponding image description information are combined to obtain image data pairs. The initial model is trained based on the image data pairs until a predefined number of iterations is reached, at which point training stops, resulting in the layout generation model.

[0009] Optionally, training an initial model based on image data includes: determining text features, image features, and initial noise for the image data pair; fusing text features, image features, and initial noise to obtain an input image for training the initial model; jointly encoding entity text features and entity image features in the entity description information of the input image through a cross-attention network to obtain a first encoded feature; jointly encoding background text features and background image features in the background description information of the input image through a cross-attention network to obtain a second encoded feature; and jointly encoding global text features and global image features in the global description information of the input image through a convolutional neural network to obtain a third encoded feature; fusing the first, second, and third encoded features to obtain a comprehensive feature of the input image; determining the prediction noise corresponding to the input image based on the comprehensive feature; and subtracting the prediction noise from the initial noise in the input image to obtain the output image of the initial model.

[0010] Optionally, fusing the first coding feature, the second coding feature, and the third coding feature includes: fusing the first coding feature and the second coding feature to obtain local features of the input image; determining global features of the input image based on the third coding feature; and determining comprehensive features of the input image based on the local features, the global features, and the preset fusion weights.

[0011] Optionally, the method further includes: performing a Fourier transform on the entity image features in the comprehensive features to obtain continuous position features; and performing a self-attention transform after concatenating the continuous position features with the comprehensive features.

[0012] Optionally, the target loss function is determined as follows: a base loss function for the layout generation model is determined, whereby the base loss function enhances the denoising learning ability of the layout generation model; a first attention score and a second attention score corresponding to the background description information and the global description information are determined respectively through a cross-attention mechanism, and an activation loss function for the layout generation model is determined based on the first attention score, the second attention score, and the segmentation mask; the semantic similarity between the entity text features and entity image features of the entity description information is determined, and a semantic loss function for the layout generation model is determined based on the semantic similarity; the base loss function, the activation loss function, and the semantic loss function are fused to obtain the target loss function.

[0013] Optionally, a layout generation model is used to analyze the image parsing results and background prompts to obtain target image features, including: obtaining target image description information from the image parsing results; determining the target text features and target initial noise corresponding to the target image description information, and fusing the target text features and target initial noise to obtain the target input image of the layout generation model; analyzing the target input image and background prompts using the layout generation model to obtain target prediction noise corresponding to the target input image; and obtaining target image features by subtracting the target prediction noise from the target initial noise in the target input image and repeating the model analysis process for a first preset number of times.

[0014] Optionally, the method further includes: denoising the global descriptive information in the target image features using a layout generation model, and repeatedly performing the model denoising process for a second preset number of times to obtain optimized image features; and generating an image based on the optimized image features.

[0015] According to another aspect of the embodiments of this application, an image generation apparatus is also provided, comprising: a parsing module, configured to acquire user prompt words, perform part-of-speech parsing on the user prompt words, and determine an image parsing result corresponding to the image to be generated based on the part-of-speech parsing result; an analysis module, configured to acquire background prompt words, and analyze the image parsing result and background prompt words using a layout generation model to obtain target image features, wherein the layout generation model performs feature encoding and fusion of entity information, background information and global information of the image to be generated through a variational autoencoder, and the target loss function of the layout generation model includes at least an activation loss function for enhancing the correlation between entities in the image to be generated and their corresponding positions, and a semantic loss function for enhancing the correlation between entities in the image to be generated and their corresponding attribute descriptions; and a generation module, configured to generate an image based on the target image features.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and used to execute the above-described image generation method.

[0017] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described image generation method by running the computer program.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described image generation method.

[0019] In this embodiment, user prompts are obtained and their parts-of-speech tags are parsed. Based on the parsing results, an image parsing result corresponding to the image to be generated is determined. Background prompts are obtained, and a layout generation model is used to analyze the image parsing result and the background prompts to obtain target image features. The layout generation model uses a variational autoencoder to encode and fuse entity information, background information, and global information of the image to be generated. The target loss function of the layout generation model includes at least an activation loss function to enhance the correlation between entities in the image to be generated and their corresponding positions, and a semantic loss function to enhance the correlation between entities in the image to be generated and their corresponding attribute descriptions. Based on the target image features, an image is generated, which achieves the goal of strengthening the correlation between entities in the image and their specified positions and attribute descriptions. This achieves accurate generation from layout to image, significantly improving the controllability and semantic consistency of image generation. This solves the technical problem that image generation methods in related technologies cannot directly generate images based on user layout requirements and specified elements when dealing with complex layouts and multi-text descriptions. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a hardware structure diagram of a computer terminal for implementing an image generation method according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of an image generation method according to an embodiment of this application;

[0023] Figure 3 This is a flowchart of an overall scheme for image generation according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a foreground / background decoupling enhancement network coding process according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of cross-granularity reinforcement feedback according to an embodiment of this application;

[0026] Figure 6 This is an application flowchart of a text layout control generation system according to an embodiment of this application;

[0027] Figure 7 This is a structural diagram of an image generation apparatus according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:

[0031] Diffusion Model: A deep learning model used to generate high-dimensional data (such as images and audio). It transforms the data into a Gaussian distribution by progressively adding noise, and then generates new data by learning the reverse process (i.e., denoising). In the field of image generation, the diffusion model is considered a powerful generative model because it can generate images with high resolution and rich detail.

[0032] U-Net (U-Network): A convolutional neural network architecture originally designed for image segmentation tasks. It consists of an encoder (downsampling path) and a decoder (upsampling path), fusing features from the encoder and decoder through skip connections to recover details and accurately segment objects.

[0033] VAE (Variational Autoencoder): A deep learning model used for unsupervised learning and efficient encoding of data. It generates or reconstructs images by encoding data into a low-dimensional latent space and then decoding it back into the original data space.

[0034] CLIP (Contrastive Language-Image Pre-training): A multimodal model that processes both text and images simultaneously through pre-training, enabling it to learn the correspondence between text and images.

[0035] SAM (Segment Anything Model): An image segmentation model capable of segmenting any object in an image without requiring specific category labels. In this application, SAM is used to segment foreground elements of an image, extract object masks, and provide precise location information for image elements at specified locations.

[0036] To address the issue of poor efficiency in directly generating images in related technologies, this application provides an image generation method that can be run on... Figure 1 The computer terminal shown is described below.

[0037] The image generation method embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing an image generation method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image generation method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the image generation method described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0041] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0042] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0043] In the above operating environment, this application provides an embodiment of an image generation method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] Figure 2 This is a flowchart of an image generation method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0045] Step S202: Obtain user prompt words, perform part-of-speech analysis on user prompt words, and determine the image parsing result corresponding to the image to be generated based on the part-of-speech analysis result.

[0046] Step S204: Obtain background cue words. Analyze the image parsing results and background cue words using a layout generation model to obtain target image features. The layout generation model uses a variational autoencoder to encode and fuse entity information, background information, and global information of the image to be generated. The target loss function of the layout generation model includes at least an activation loss function to enhance the correlation between entities in the image to be generated and their corresponding positions, and a semantic loss function to enhance the correlation between entities in the image to be generated and their corresponding attribute descriptions.

[0047] Step S206: Generate an image based on the features of the target image.

[0048] Through steps S202 to S206, the correlation between entities in an image and their specified locations and attribute descriptions is strengthened, thereby achieving accurate image generation from layout. This significantly improves the controllability and semantic consistency of image generation, and solves the technical problem that image generation methods in related technologies cannot directly generate images based on user layout requirements and specified elements when dealing with complex layouts and multi-text descriptions. The following is a detailed explanation.

[0049] Figure 3 This is a flowchart of an overall scheme for image generation according to an embodiment of this application, such as... Figure 3 As shown, this covers the entire process of model training and application (image generation), mainly including three key steps: building a massive unsupervised data engine, expanding multi-text layout control generation, and applying a multi-text layout control system. The following combines... Figure 3 The implementation logic of the scheme provides a detailed explanation of the training process of the layout generation model and the above-mentioned image generation method.

[0050] Step 1: Building a massive unsupervised data engine.

[0051] In this embodiment of the application, an engine capable of automatically collecting and generating large-scale multimodal training data is constructed, providing massive amounts of multimodal and structured training data for model training.

[0052] Based on this data engine, the layout generation model can be trained as follows: First, acquire historical generated images. Second, process these images according to background prompts and a predefined thought chain to obtain image description information. The predefined thought chain defines the image description format for the historical generated images. This image description information includes entity description information, background description information, global description information, and a segmentation mask. Entity description information includes entity text features and entity image features of the historical generated images. Background description information includes background text features and background image features of the historical generated images. Global description information includes global text features and global image features of the historical generated images. The segmentation mask is used to distinguish entity locations within the historical generated images. Third, combine the historical generated images and their corresponding image description information to obtain image data pairs. Fourth, train the initial model based on the image data pairs until a predefined number of iterations is reached, at which point training stops, resulting in the layout generation model.

[0053] Specifically, the training process of the layout generation model revolves around historically generated images and pre-defined thought chains. Its core purpose is to enable the model to learn to generate corresponding images from text descriptions and layout information through deep learning technology.

[0054] First, for image data, a large number of high-quality images or historically generated images can be collected as training images. Through cue word engineering (i.e., background cue words), a large language model is used to output structured image description information. To obtain a description that meets the required format, the following background cue words can be provided:

[0055] "You are a smart bounding box generator. I will provide you with a title for a photo, image, or painting. Your task is to generate bounding boxes for the objects mentioned in the title, along with background cues describing the scene."

[0056] The image size is 512×512. The coordinates of the top-left corner are [0,0]. The coordinates of the bottom-right corner are [512, 512]. The bounding boxes should not overlap or extend beyond the image boundaries. Each bounding box should be formatted as (object name, [top-left x-coordinate, top-left y-coordinate, box width, box height]) and contain only one object. You may make reasonable guesses if necessary.

[0057] Furthermore, to improve the format and content of descriptions generated by the large language model, a series of examples can be used based on a pre-defined thought chain to correct the direction of the large language model's response and make reasonable guesses and assumptions. The format of this pre-defined thought chain is as follows:

[0058] Description 1: Green bench and blue bowl; Objects: [('Green bench', [50284412, 82]), ('Blue bowl', [217244, 78, 40])].

[0059] Description 2: A dog is curled up on a bed under a blanket; Objects: [('bed', [59, 231, 394, 148]), ('dog', [210, 281, 92, 98]), ['blanket', [120, 281, 272, 98]].

[0060] Description 3: A white Disney bus is passing under the train tracks; object: [('White Disney Bus', [100204312179]), ('Train Tracks', [0, 201512, 60])].

[0061] Description 4: A cat is sitting in a computer chair; object: [('cat', [201100110160]), ('computer chair', [30250452262])].

[0062] Description 5: A black and white dog is sitting on a bench; object: [('black and white dog', [156204200150]), ('bench', [515354410, 58])].

[0063] The specific expression is as follows:

[0064]

[0065] In the formula, Representing a large language model; It can represent image input, such as historically generated images, or global image features; Indicates background cue words; This represents a global image description, i.e., global text features; Represents foreground (entity) elements, i.e., entity text features; This represents the location region corresponding to the foreground (entity) element, i.e., the entity image feature. Subsequently, a segmentation model (such as SAM) is used to segment entities in the historically generated images, taking foreground element words as input and outputting the corresponding element mask (segmentation mask). Then, the segmented main area is turned black to obtain a background image that retains only the background:

[0066]

[0067]

[0068] For the background image, i.e., the background image features, the same multimodal large language model is used, combined with the corresponding background prompt template. Image description generation is performed on the background image to obtain background text features. :

[0069]

[0070] in, Here's an example: "The black areas in the image are the foreground elements. Please ignore the black areas and describe the visible areas to generate the background description of the image."

[0071] Finally, the image data and corresponding image descriptions are combined to form the image data pairs required for training. The initial model learns the association between image features and text descriptions in these image data pairs, gradually optimizing its generation capabilities. During training, the model's objective loss function ensures that the position and attributes of entities closely correspond to the description information. Through iterative training until a preset number of iterations is reached, the final layout generation model can generate high-quality images that conform to both the overall description and precise layout based on the input layout and description, meeting the precise control requirements for image generation in creative design and image editing fields.

[0072] Step 2: Generate extended multi-text layout control.

[0073] In this embodiment, unlike traditional image generation tasks that rely solely on a single text segment for image generation, a more precise generation method is provided. A multi-text layout control generation system is constructed, and the basic text-to-image system is expanded. Users can simultaneously input multiple text segments and specify multiple different local regions for precise element generation.

[0074] The multi-text layout control generation system, with a layout generation model at its core, trains the model based on the aforementioned training data. This process includes: determining the text features, image features, and initial noise of image data pairs; fusing these features to obtain the input image for training the initial model; jointly encoding the entity text features and entity image features in the entity description information of the input image using a cross-attention network to obtain the first encoded feature; jointly encoding the background text features and background image features in the background description information of the input image using a cross-attention network to obtain the second encoded feature; and jointly encoding the global text features and global image features in the global description information of the input image using a convolutional neural network to obtain the third encoded feature; fusing the first, second, and third encoded features to obtain the comprehensive features of the input image; determining the prediction noise corresponding to the input image based on the comprehensive features; and subtracting the prediction noise from the initial noise in the input image to obtain the output image of the initial model. The specific process analysis is as follows:

[0075] 1. Determine the input image to be used for training the initial model.

[0076] The data is processed using the following pairs: image (global image features) - background image (background image features) - global description (global text features) - background image description (background text features) - segmentation mask - entity word (entity text features) - entity word location region (entity image features). The input image is used as input, and the input image is preprocessed.

[0077] For text features The CLIP multimodal perception model is used for global description of the input image. Perform text feature modeling:

[0078]

[0079] in, This indicates that the text is tokenized to obtain the token for the corresponding word.

[0080] For image features (including global and local images), an autoregressive generative network (VAE) is used to encode the images:

[0081] 1) Global Image: Compress the image information to 128*128*3 (original image size is 1024*1204*3) to obtain Then, by randomly sampling a timestamp t, Gaussian noise (i.e., the initial noise) is gradually superimposed onto it. The noise map is obtained above. .

[0082]

[0083] in, It is Gaussian noise. The coefficients for increasing the noise level are determined. To achieve image reconstruction, a Unet network is used to analyze the noisy image. Modeling is performed to learn how to superimpose on... Predicted noise distribution on and The original image is reconstructed by subtraction. By learning different noise distributions, the Unet network can reconstruct and generate different high-quality images.

[0084] 2) Local image: Entity word location region Used to crop the global image To obtain a local image representing the entity. :

[0085]

[0086] All entities and background images are encoded through a VAE network to obtain image features in the same feature space as the full image.

[0087] 2. Combined coding.

[0088] like Figure 4 As shown, the encoding process of one layer in a VAE network is illustrated. Each layer of the VAE network is divided into two parts, including foreground / background decoupling generation and foreground / background fusion enhancement.

[0089] 1) Foreground / Background Decoupling Generation Module: This module is primarily responsible for jointly encoding the entire image, foreground (entities), and background using the Unet network and cross-attention networks. To achieve text-guided image generation, different fine-grained text features are injected and fused with corresponding image features in the network through basic cross-attention connections. The fusion formula is as follows:

[0090]

[0091] in, Representing text features, Representing image features, This represents the combined features after joint encoding. This represents learnable dynamic weights.

[0092] Specifically, entity description information of the input image is processed through a cross-attention network. The entity text features and entity image features in the input image are jointly encoded to obtain the first encoded feature. The background description information of the input image is then processed through a cross-attention network. The background text features and background image features in the input image are jointly encoded to obtain the second encoded feature. The global text features and global image features in the global description information of the input image are jointly encoded using the Unet network to obtain the third encoded feature. This decoupled encoding method effectively solves the semantic confusion and spatial localization problems caused by the weak text encoder, improving the spatial localization capability of the generative model.

[0093] 2) Foreground and background fusion enhancement module: mainly responsible for effectively fusing the encoded features of the entity region with the encoded features of the background region, that is, fusing the first encoded feature, the second encoded feature and the third encoded feature, including: fusing the first encoded feature and the second encoded feature to obtain the local features of the input image; determining the global features of the input image based on the third encoded feature; and determining the comprehensive features of the input image based on the local features, the global features and the preset fusion weights.

[0094] Specifically, firstly, the first encoded feature of the entity region is adjusted to match the size of the corresponding bounding box, ensuring that the entity element can be accurately aligned to its actual position in the image. Then, the resized first encoded feature is overlaid with the background encoded feature (i.e., the second encoded feature) of the region containing the corresponding bounding box. This allows the entity element to blend more naturally into the background environment while preserving its key details, thereby improving the overall performance of the model. The specific expression is as follows:

[0095]

[0096]

[0097] in, This represents the first encoded feature corresponding to the entity region. This represents the second encoded feature corresponding to the background region. Represents local features of the input image. Represents the global features of the input image. Represents the comprehensive features of the input image. This indicates the preset fusion weights, which are the aforementioned learnable dynamic weights. They are initialized to 0 during the actual training process to ensure smooth training.

[0098] 3. Self-attention optimization.

[0099] In this embodiment, to introduce additional entity and location information, the aforementioned foreground / background decoupling network explicitly fuses location and corresponding descriptive information through a gated self-attention mechanism. Specifically, the method further includes: performing a Fourier transform on the entity image features in the comprehensive features to obtain continuous location features; and concatenating the continuous location features with the comprehensive features before performing a self-attention transform. The specific expression is as follows:

[0100]

[0101]

[0102] Among them, entity location region (i.e. entity image features) The feature representation is obtained by performing a Fourier transform on the feature representation. ; Represents the self-attention mechanism function; These represent learnable weight parameters used to control the degree of fusion. Represents the features of the original image; This represents the image feature table updated by the self-attention mechanism; Indicates the feature dimension.

[0103] In this embodiment, the target loss function is determined as follows: a base loss function for the layout generation model is determined, wherein the base loss function is used to enhance the denoising learning ability of the layout generation model; a first attention score and a second attention score corresponding to the background description information and the global description information are determined respectively through a cross-attention mechanism, and an activation loss function for the layout generation model is determined based on the first attention score, the second attention score, and the segmentation mask; the semantic similarity between the entity text features and entity image features of the entity description information is determined, and a semantic loss function for the layout generation model is determined based on the semantic similarity; the base loss function, the activation loss function, and the semantic loss function are fused to obtain the target loss function.

[0104] To improve positioning accuracy, this application also introduces a latent space reward feedback mechanism and a semantic feedback reward mechanism, such as... Figure 5 As shown, the cross-granularity reinforcement feedback mechanism significantly enhances the model's layout generation capability through optimization at different levels of abstraction, achieving high-quality and semantically accurate image generation. Overall, the objective loss function of this layout generation model mainly comprises three key parts: the basic loss function, the activation loss function, and the semantic loss function, with the specific expressions as follows:

[0105]

[0106] in, Represents the target loss function. Represents the basic loss function. This represents the activation loss function. This represents the semantic loss function.

[0107] 1) Basic loss function Primarily used to enhance the denoising learning capability of layout generation models.

[0108] The specific expression is as follows:

[0109]

[0110] in, This represents the expected value operation. Noise term Randomly drawn from the standard normal distribution N(0,1) Indicates prediction noise, This represents the image features after adding initial noise. This represents the time step in the diffusion process. Represents text features.

[0111] 2) Activation loss function To enhance the accuracy of entity word generation at specified positions, a reward feedback mechanism constructed in the latent space layer is proposed. This is achieved by optimizing the weight matrix across the attention mechanism, i.e., the attention activation graph. This allows for more precise control over the model's localization of entities in images. This mechanism not only enhances the model's perception of entity locations in the hidden layers but also introduces semantic reward feedback at the pixel level, further ensuring that the entity attributes and relationships in the generated image are consistent with the description.

[0112] The specific expression is as follows:

[0113]

[0114] In the formula, This represents the global attention activation map, corresponding to the first attention score; This represents the background attention activation map, corresponding to the second attention score; This represents the segmentation mask.

[0115]

[0116] in, It is an m*n two-dimensional matrix, where n is the number of text tokens and m is the product of the length and width of the hidden layer features of the image. That is... The value in the k-th row and j-th column represents the probability of the k-th text token appearing at the j-th position. By constraining the probability of a specific token appearing at a specific position, the accuracy of positioning can be effectively improved.

[0117] 3) Semantic loss function To further ensure the association between image attribute descriptions and entities, a semantic reward feedback mechanism at the pixel level is proposed. The accuracy of entity attributes in the generated image is measured by calculating the cosine similarity between local entity image features and corresponding entity text features. By minimizing the average cosine similarity loss of all entities, the model is guided to generate more semantically consistent images, thereby maintaining the consistency between entities and their descriptions.

[0118] The specific expression is as follows:

[0119]

[0120] In the formula, The cosine similarity between the entity image features and the corresponding entity text features is represented by N, where N represents the number of entities.

[0121] Step 3: Application of the multi-text layout control generation system.

[0122] In this embodiment of the application, the overall layout generation process is as follows: Figure 6As shown, this method can be directly applied to analyze image parsing results and background prompts using a layout generation model to obtain target image features and generate an image. This includes: acquiring target image description information from the image parsing results; determining the target text features and initial target noise corresponding to the target image description information, and fusing the target text features and initial target noise to obtain the target input image for the layout generation model; analyzing the target input image and background prompts using the layout generation model to obtain the target prediction noise corresponding to the target input image; subtracting the target prediction noise from the initial target noise in the target input image, and repeating the model analysis process for a first preset number of times to obtain the target image features. The following is combined with... Figure 6 The steps S202 to S206 above will be described in detail.

[0123] In step S202 above, the multi-text layout control generation system is mainly responsible for receiving user prompts about the image to be generated. These prompts can include descriptions of the image elements and their relative positions in the image (which can be in coordinate form), for example: a tennis ball and a tennis racket are placed between a skateboard and a beverage cup.

[0124] Subsequently, part-of-speech analysis was performed on the user prompts to identify key elements such as entities, attributes, and location information, providing accurate guidance for subsequent image generation.

[0125] In step S204 above, after obtaining the parsing results of the user prompt words and the background prompt words, a layout generation model is used to encode and fuse entity information, background information, and global description through a VAE network, converting the text description into a feature space representation to guide the model in generating target image features that match the user description. Through feature encoding and fusion, the layout generation model can learn and generate image features that match the input text description, ensuring that the generated image not only meets the user's layout requirements but also accurately reflects the attribute descriptions of the entities.

[0126] It should be noted that the application process differs from the training phase. In this phase, the input image is replaced with random Gaussian noise (i.e., the initial target noise) for random initialization. This noise, along with the target text features corresponding to the input image, is then fed into the Unet network to predict the noise and obtain the target prediction noise. The target prediction noise is then removed from the initial target noise. This process is repeated 15 times (i.e., the first preset number of times) to ensure that the generated features can meet the expected layout, ultimately yielding the target image features.

[0127] In step S206 above, the main responsibility is to convert the target image features obtained from the layout generation model into an intuitive, high-quality visual image.

[0128] Optionally, the above method further includes: denoising the global descriptive information in the target image features using a layout generation model, and repeatedly performing the model denoising process for a second preset number of times to obtain optimized image features; and generating an image based on the optimized image features.

[0129] Specifically, to further improve image quality, the generated target image features are input again into the layout generation model. It should be noted that at this stage, the layout generation model only receives the overall image description and performs denoising. This step is repeated 25 times (the second preset number) to iteratively optimize and gradually improve image quality, ultimately generating an image that meets the layout requirements and has high resolution, which is then returned as output.

[0130] Overall, in this application embodiment, a multimodal seven-tuple data engine that can be scaled to hundreds of millions without manual annotation is constructed. Based on the previous background-decoupled diffusion generation framework, a cross-granularity joint loss mechanism is proposed: in the latent space, attention mask loss is used to enhance the generation accuracy of entities at specified positions, and in the pixel space, CLIP semantic consistency loss is used to ensure that entities are strictly aligned with their attribute descriptions. Thus, for the first time, end-to-end layout and accurate image generation of multiple texts, multiple regions, and multiple attributes are realized, significantly improving the controllability, semantic consistency, and fine-grained positioning capabilities of image generation.

[0131] According to embodiments of this application, an image generation apparatus is provided. It should be noted that the image generation apparatus of this application can be used to execute the image generation method provided in the embodiments of this application. The image generation apparatus provided in the embodiments of this application will be described below.

[0132] Figure 7 This is a structural diagram of an image generation apparatus provided according to an embodiment of this application. For example... Figure 7 As shown, the device includes:

[0133] The parsing module 70 is used to obtain user prompt words, perform part-of-speech parsing on the user prompt words, and determine the image parsing result corresponding to the image to be generated based on the part-of-speech parsing result;

[0134] Analysis module 72 is used to obtain background prompt words. It uses a layout generation model to analyze the image parsing results and background prompt words to obtain target image features. The layout generation model uses a variational autoencoder to encode and fuse the entity information, background information and global information of the image to be generated. The target loss function of the layout generation model includes at least an activation loss function to enhance the correlation between entities in the image to be generated and their corresponding positions, and a semantic loss function to enhance the correlation between entities in the image to be generated and their corresponding attribute descriptions.

[0135] The generation module 74 is used to generate an image based on the features of the target image.

[0136] By using the parsing module, analysis module, and generation module in the aforementioned image generation device, the goal of strengthening the correlation between entities in an image and their specified locations and attribute descriptions is achieved. This enables accurate generation from layout to image, significantly improving the controllability and semantic consistency of image generation. Consequently, it solves the technical problem that image generation methods in related technologies cannot directly generate images based on user layout requirements and specified elements when dealing with complex layouts and multi-text descriptions.

[0137] In the image generation apparatus provided in this application embodiment, the analysis module is further configured to obtain target image description information in the image parsing result; determine the target text features and target initial noise corresponding to the target image description information, and fuse the target text features and target initial noise to obtain the target input image of the layout generation model; analyze the target input image and background prompt words using the layout generation model to obtain the target prediction noise corresponding to the target input image; and obtain the target image features by subtracting the target prediction noise from the target initial noise in the target input image and repeatedly performing the model analysis processing for a first preset number of times.

[0138] In the image generation apparatus provided in this application embodiment, the generation module is further configured to perform denoising processing on the global description information in the target image features through a layout generation model, and repeatedly perform model denoising processing for a second preset number of times to obtain optimized image features; and generate an image based on the optimized image features.

[0139] The image generation apparatus provided in this application embodiment further includes a training module 76, used to acquire historical generated images; process the historical generated images according to background prompts and a preset thought chain to obtain image description information, wherein the preset thought chain is used to define the image description format of the historical generated images, and the image description information includes entity description information, background description information, global description information, and a segmentation mask. The entity description information includes entity text features and entity image features of the historical generated images, the background description information includes background text features and background image features of the historical generated images, the global description information includes global text features and global image features of the historical generated images, and the segmentation mask is used to distinguish the entity positions in the historical generated images; combine the historical generated images and the corresponding image description information to obtain image data pairs; train an initial model based on the image data pairs until a preset number of iterations is reached and then training stops to obtain a layout generation model.

[0140] In the image generation apparatus provided in this application embodiment, the training module is further configured to determine the text features, image features, and initial noise of the image data pair, and to fuse the text features, image features, and initial noise to obtain an input image for training an initial model; to obtain a first encoded feature by jointly encoding the entity text features and entity image features in the entity description information of the input image through a cross-attention network; to obtain a second encoded feature by jointly encoding the background text features and background image features in the background description information of the input image through a cross-attention network; and to obtain a third encoded feature by jointly encoding the global text features and global image features in the global description information of the input image through a convolutional neural network; to obtain a comprehensive feature of the input image by fusing the first encoded feature, the second encoded feature, and the third encoded feature; to determine the prediction noise corresponding to the input image based on the comprehensive feature, and to obtain the output image of the initial model by subtracting the prediction noise from the initial noise in the input image.

[0141] In the image generation apparatus provided in this application embodiment, the training module is further configured to fuse the first coding feature and the second coding feature to obtain the local features of the input image; determine the global features of the input image based on the third coding feature; and determine the comprehensive features of the input image based on the local features, the global features, and the preset fusion weights.

[0142] In the image generation apparatus provided in this application embodiment, the training module is further used to perform Fourier transform on the entity image features in the comprehensive features to obtain continuous position features; and to perform self-attention transform after splicing the continuous position features and the comprehensive features.

[0143] In the image generation apparatus provided in this application embodiment, the training module is further used to determine the basic loss function of the layout generation model, wherein the basic loss function is used to enhance the denoising learning ability of the layout generation model; the first attention score and the second attention score corresponding to the background description information and the global description information are determined respectively through a cross-attention mechanism, and the activation loss function of the layout generation model is determined based on the first attention score, the second attention score and the segmentation mask; the semantic similarity between the entity text features and the entity image features of the entity description information is determined, and the semantic loss function of the layout generation model is determined based on the semantic similarity; the basic loss function, the activation loss function and the semantic loss function are fused to obtain the target loss function.

[0144] This application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and used to execute the above-described image generation method.

[0145] It should be noted that the aforementioned electronic equipment is used to perform Figure 2The image generation method shown above is also applicable to this electronic device, and will not be repeated here.

[0146] This application also provides a non-volatile storage medium including a stored computer program, wherein the device containing the non-volatile storage medium executes the above-described image generation method by running the computer program.

[0147] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The image generation method shown above is also applicable to this non-volatile storage medium, and will not be repeated here.

[0148] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described image generation method.

[0149] It should be noted that the above-mentioned computer program product is used to execute... Figure 2 The image generation method shown above is also applicable to this computer program product, and will not be repeated here.

[0150] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0151] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0155] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0156] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image generation method, characterized in that, include: Obtain user prompt words, perform part-of-speech tagging on the user prompt words, and determine the image parsing result corresponding to the image to be generated based on the part-of-speech tagging result; Background prompts are obtained, and the image parsing results and background prompts are analyzed using a layout generation model to obtain target image features. The layout generation model encodes and fuses entity information, background information, and global information of the image to be generated using a variational autoencoder. The target loss function of the layout generation model includes at least an activation loss function to enhance the correlation between entities and corresponding positions in the image to be generated, and a semantic loss function to enhance the correlation between entities and corresponding attribute descriptions in the image to be generated. An image is generated based on the features of the target image.

2. The method according to claim 1, characterized in that, The layout generation model is trained in the following way: Retrieve historical generated images; The historically generated images are processed based on the background prompts and a preset thought chain to obtain image description information. The preset thought chain is used to define the image description format of the historically generated images. The image description information includes entity description information, background description information, global description information, and a segmentation mask. The entity description information includes entity text features and entity image features of the historically generated images. The background description information includes background text features and background image features of the historically generated images. The global description information includes global text features and global image features of the historically generated images. The segmentation mask is used to distinguish the entity positions in the historically generated images. By combining the historical generated images and their corresponding image description information, image data pairs are obtained; The initial model is trained based on the image data until a preset number of iterations is reached, at which point training stops, resulting in the layout generation model.

3. The method according to claim 2, characterized in that, Training an initial model based on the image data includes: The text features, image features, and initial noise of the image data pair are determined, and the text features, image features, and initial noise are fused to obtain the input image for training the initial model; The first encoded feature is obtained by jointly encoding the entity text features and entity image features in the entity description information of the input image through a cross-attention network; the second encoded feature is obtained by jointly encoding the background text features and background image features in the background description information of the input image through the cross-attention network; and the third encoded feature is obtained by jointly encoding the global text features and global image features in the global description information of the input image through a convolutional neural network. By fusing the first coding feature, the second coding feature, and the third coding feature, a comprehensive feature of the input image is obtained; Based on the comprehensive features, the predicted noise corresponding to the input image is determined, and the predicted noise is subtracted from the initial noise in the input image to obtain the output image of the initial model.

4. The method according to claim 3, characterized in that, The fusion of the first coding feature, the second coding feature, and the third coding feature includes: By fusing the first encoded feature and the second encoded feature, local features of the input image are obtained; The global features of the input image are determined based on the third encoded feature; The comprehensive features of the input image are determined based on the local features, the global features, and the preset fusion weights.

5. The method according to claim 4, characterized in that, The method further includes: Perform a Fourier transform on the entity image features in the comprehensive features to obtain continuous position features; After splicing the continuous positional features with the comprehensive features, a self-attention transformation is performed.

6. The method according to claim 2, characterized in that, The target loss function is determined in the following way: Determine the base loss function of the layout generation model, wherein the base loss function is used to enhance the denoising learning ability of the layout generation model; The first attention score and the second attention score corresponding to the background description information and the global description information are determined by cross-attention mechanism, and the activation loss function of the layout generation model is determined based on the first attention score, the second attention score and the segmentation mask. Determine the semantic similarity between the entity text features and entity image features of the entity description information, and determine the semantic loss function of the layout generation model based on the semantic similarity; The target loss function is obtained by fusing the base loss function, the activation loss function, and the semantic loss function.

7. The method according to claim 1, characterized in that, The image parsing results and background prompts are analyzed using a layout generation model to obtain target image features, including: Obtain the target image description information from the image parsing results; The target text features and initial target noise corresponding to the target image description information are determined, and the target text features and initial target noise are fused to obtain the target input image of the layout generation model; The layout generation model is used to analyze the target input image and the background prompt words to obtain the target prediction noise corresponding to the target input image; The target image features are obtained by subtracting the target prediction noise from the target initial noise in the target input image and repeating the model analysis process for a first preset number of times.

8. The method according to claim 1, characterized in that, The method further includes: The global descriptive information in the target image features is denoised using the layout generation model, and the model denoising process is repeated a second preset number of times to obtain optimized image features. An image is generated based on the optimized image features.

9. An image generation apparatus, characterized in that, include: The parsing module is used to obtain user prompt words, perform part-of-speech parsing on the user prompt words, and determine the image parsing result corresponding to the image to be generated based on the part-of-speech parsing result; The analysis module is used to obtain background prompt words and analyze the image parsing results and background prompt words using a layout generation model to obtain target image features. The layout generation model uses a variational autoencoder to perform feature encoding and fusion of entity information, background information and global information of the image to be generated. The target loss function of the layout generation model includes at least an activation loss function to enhance the correlation between entities and corresponding positions in the image to be generated, and a semantic loss function to enhance the correlation between entities and corresponding attribute descriptions in the image to be generated. The generation module is used to generate an image based on the features of the target image.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; The processor, connected to the memory, is used to execute the image generation method according to any one of claims 1 to 8.

11. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the image generation method according to any one of claims 1 to 8 by running the computer program.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the image generation method according to any one of claims 1 to 8.

Images