Image generation method and device, computer device, and storage medium

Abstract

This application proposes an image generation method, apparatus, computer device, and storage medium, relating to the field of artificial intelligence technology. The method includes: encoding target conditional text to obtain conditional features; decomposing the conditional features to obtain a low-rank matrix and sparse coded features; obtaining a diffusion model, which includes K cascaded diffusion layers, using the k-th diffusion layer as the current diffusion layer and a preset feature map as input to the current diffusion layer, with any two diffusion layers having different output resolutions; fusing the low-rank matrix, sparse coded features, and preset feature map through the current diffusion layer to obtain the next conditional feature, ultimately obtaining the target feature map; and performing image transformation on the target feature map to obtain the target generated image. Applicable to image generation scenarios in the financial and medical fields, this application can solve the technical problem of decreased image generation efficiency due to high dimensionality of conditional embedding.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, computer vision and other technologies, and is applicable to image generation scenarios in the financial and medical fields. In particular, it relates to an image generation method and apparatus, computer equipment and storage medium. Background Technology

[0002] In the field of artificial intelligence, images that meet specific requirements can be generated based on given conditional text. For example, in the financial claims process, images of claims forms and visualizations related to the claims process can be generated. Another example is in the medical image management process, where images of the lungs showing spiculated nodules can be generated.

[0003] In related technologies, generative adversarial networks (GANs) can be used for image generation. However, existing technologies face two major challenges when handling high-dimensional conditional inputs: conditional redundancy and low computational efficiency. For example, while GAN-based frameworks can generate high-quality images, direct processing of high-dimensional conditional embeddings can lead to feature redundancy and low computational efficiency. Summary of the Invention

[0004] The main objective of this application is to provide an image generation method, apparatus, computer device, and storage medium that can solve the technical problem of decreased image generation efficiency due to high dimensionality of conditional embedding.

[0005] To achieve the above objectives, a first aspect of this application proposes an image generation method, the method comprising: Obtain the target conditional text, encode the target conditional text, and obtain conditional features; The conditional features are decomposed to obtain a low-rank matrix and sparse coding features; Obtain a diffusion model, which includes K cascaded diffusion layers. The kth diffusion layer is used as the current diffusion layer, and a preset feature map is used as the input of the current diffusion layer. The output resolutions of any two diffusion layers are different. The low-rank matrix, the sparse coding features, and the preset feature map are fused through the current diffusion layer to obtain the next conditional feature, and the next conditional feature is used as the input of the (k+1)th diffusion layer; wherein, the conditional feature output by the Kth diffusion layer in the diffusion model is used as the target feature map; The target feature map is transformed to obtain the target generated image.

[0006] To achieve the above objectives, a second aspect of this application provides an image generation apparatus, the apparatus comprising: The conditional encoding module is used to acquire target conditional text, encode the target conditional text, and obtain conditional features; The condition decomposition module is used to decompose the conditional features to obtain a low-rank matrix and sparse coding features; The model acquisition module is used to acquire a diffusion model, which includes K cascaded diffusion layers. The kth diffusion layer is used as the model acquisition module for the current diffusion layer. A preset feature map is used as the input of the current diffusion layer. The output resolutions of any two diffusion layers are different. The feature fusion module is used to fuse the low-rank matrix, the sparse coding features, and the preset feature map through the current diffusion layer to obtain the next conditional feature, and use the next conditional feature as the input of the (k+1)th diffusion layer; wherein, the conditional feature output by the Kth diffusion layer in the diffusion model is used as the target feature map; The image conversion module is used to perform image conversion on the target feature map to obtain the target generated image.

[0007] To achieve the above objectives, a third aspect of this application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the image generation method described in the first aspect.

[0008] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image generation method described in the first aspect.

[0009] The image generation method, apparatus, computer device, and storage medium proposed in this application first acquire target conditional text, encode the target conditional text to obtain conditional features; then, decompose the conditional features to obtain a low-rank matrix and sparse coded features; next, obtain a diffusion model, which includes K cascaded diffusion layers, using the k-th diffusion layer as the current diffusion layer and a preset feature map as the input of the current diffusion layer, with the output resolutions of any two diffusion layers being different; then, perform feature fusion on the low-rank matrix, sparse coded features, and preset feature map through the current diffusion layer to obtain the next conditional feature, and use the next conditional feature as the input of the (k+1)-th diffusion layer; wherein, the conditional feature output by the k-th diffusion layer in the diffusion model is used as the target feature map; finally, perform image transformation on the target feature map to obtain the target generated image. In this application, high-dimensional conditional features are dynamically compressed into compact low-rank matrices and sparse coded features while preserving semantic fidelity; and for different resolutions, the compressed conditions are fused with the diffusion process to achieve adaptive image generation under multi-resolution conditions. Therefore, this application can significantly improve computational efficiency while ensuring generation quality.

[0010] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for an image generation method according to an embodiment of this application; Figure 2 This is a flowchart of an image generation method provided in an embodiment of this application; Figure 3 yes Figure 2 The flowchart for step 202 in the document; Figure 4 yes Figure 2 The flowchart for step 204 in the document; Figure 5 yes Figure 4 The flowchart for step 404 in the middle; Figure 6 This is a schematic diagram of the structure of an image generation apparatus in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device according to one embodiment of this application; Figure 8 This is another structural schematic diagram of a computer device in one embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0014] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0016] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0017] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0018] Generative Adversarial Networks (GANs) are deep learning models consisting of a generator and a discriminator. Through adversarial training, they learn data distributions and possess powerful data generation capabilities. They can not only generate high-quality images but also demonstrate broad application potential in multimodal data processing.

[0019] Diffusion models are a class of deep generative models based on progressive noise addition and denoising. Their core idea is to transform complex data distributions into easily tractable Gaussian noise distributions, and then gradually reconstruct the data through learning the inverse process. This approach has shown outstanding performance in areas such as image generation, text-to-image processing, and multimodal generation.

[0020] Conditional image generation techniques have made significant progress in computer vision, with Generative Adversarial Networks (GANs) and diffusion models becoming mainstream methods. However, existing techniques face two major challenges when handling high-dimensional conditional inputs (such as text embeddings or complex category labels): conditional redundancy and low computational efficiency. For example, while GAN-based frameworks can generate high-quality images, direct processing of high-dimensional conditions can lead to feature redundancy and training instability; and while diffusion models perform well in generating quality, their iterative denoising process requires significant computational resources, especially when the conditional dimension is large.

[0021] To address the issue of conditional redundancy, sparse coding techniques have been introduced into generative tasks. For example, some methods have proposed a differentiable low-rank sparse coding layer for hyperspectral image restoration, demonstrating the effectiveness of sparse coding in feature compression. Similarly, some techniques combine sparse coding with GANs to improve the performance of hyperspectral demixing. However, these methods do not address the efficiency problem of the generative model itself and lack the ability to dynamically adapt to conditional semantics.

[0022] In the design of conditional generation frameworks, attention mechanisms are widely used to enhance conditional alignment; for example, some techniques achieve image super-resolution through hierarchical conditional flow. Furthermore, some methods have explored the application of diffusion models in image compression, but have not considered the sparsity optimization of conditional inputs. Recent work has proposed generative compression methods with dynamically granular adaptation, but their conditional fusion mechanisms still rely on fixed-dimensional embeddings, limiting their flexibility.

[0023] Limitations of existing technologies: 1. Redundancy and inefficient compression: Existing methods directly process high-dimensional conditions, resulting in high computational overhead and scattered semantic information; 2. Static conditional fusion: Most frameworks use fixed-dimensional conditional embedding, which cannot dynamically adjust feature resolution according to the generation stage; 3. Trade-off between generation quality and efficiency: Although diffusion models can generate high-quality images, their computational cost is high, while sparse coding methods lack generation diversity.

[0024] This application proposes an image generation method, which is also an adaptive conditional image generation framework based on sparse coding and a diffusion model. By combining trainable sparse coding with a diffusion generation model, it achieves efficient conditional compression and high-fidelity image synthesis. The core components include a differentiable low-rank sparse coding module, which dynamically compresses high-dimensional conditional features into a compact, task-specific representation while preserving semantic fidelity; and a cross-scale attention mechanism, used to fuse the compressed conditional features with the diffusion process, enabling adaptive image generation under multi-resolution conditions. The differentiable low-rank sparse coding module achieves conditional compression through joint optimization of the low-rank matrix and sparse coding, while the cross-scale attention mechanism guides the noise prediction of the diffusion model through cross-resolution attention. This framework jointly optimizes the sparse reconstruction loss, diffusion loss, and perceptual loss, significantly improving computational efficiency while maintaining generation quality. Compared with existing conditional generation methods, this application combines the interpretability of sparse coding with the generative capability of diffusion models, exhibiting significant advantages in both conditional fidelity and computational efficiency. The adaptive conditional image generation framework proposed in this application aims to solve the problems of conditional compression and efficient image generation. It is suitable for image generation application scenarios that require high fidelity and low computational cost, such as image synthesis, image editing and image restoration.

[0025] This application provides an image generation method, an image generation apparatus, a computer device, and a computer-readable storage medium. The specific details are illustrated in the following embodiments. First, the image generation method in this application is described.

[0026] The image generation method provided in this application embodiment can be applied to, for example, Figure 1In this application environment, the client communicates with the server via a network. The server first obtains the target conditional text from the client, encodes it to obtain conditional features, decomposes these features to obtain a low-rank matrix and sparse coded features, and acquires a diffusion model consisting of K cascaded diffusion layers. The k-th diffusion layer is used as the current diffusion layer, and a preset feature map is used as its input. The output resolutions of any two diffusion layers are different. The server fuses the low-rank matrix, sparse coded features, and preset feature map through the current diffusion layer to obtain the next conditional feature, and uses this next conditional feature as the input to the (k+1)-th diffusion layer. The conditional feature output by the k-th diffusion layer in the diffusion model is used as the target feature map. The server performs image transformation on the target feature map to obtain the target generated image. In this application, high-dimensional conditional features are dynamically compressed into a compact low-rank matrix and sparse coded features while preserving semantic fidelity. Furthermore, for different resolutions, the compressed conditions are fused with the diffusion process to achieve adaptive image generation under multi-resolution conditions. As can be seen, this application can significantly improve computational efficiency while ensuring the quality of the generated output. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0027] It should be noted that in each specific embodiment of this application, when it is necessary to process data related to the user's identity or characteristics, such as text data or voice data, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of this data will comply with relevant laws, regulations and standards.

[0028] Reference Figure 2 , Figure 2 This is an optional flowchart of an image generation method provided in an embodiment of this application, which may include, but is not limited to, steps 201 to 205.

[0029] Step 201: Obtain the target condition text, encode the target condition text, and obtain the condition features.

[0030] Target conditional text refers to the input variables used to control the image generation process. For example, the target conditional text could be "a sunflower field under sunlight".

[0031] In the financial claims process, the target condition text includes the accident type, damaged parts, amount range, availability of evidence / documents, customer's claims history, risk level, and text description. The server generates matching claim form images and visualizations related to the claims process based on the image generation method described in this application. For example, the target condition text could be: "The vehicle hit a flower bed while reversing in a residential area, resulting in the right side of the rear bumper being broken, the right rear taillight being shattered, and the trunk lid being slightly deformed."

[0032] In the medical image management scenario, the target condition text is {image modality: chest CT, lesion: lung nodule, size: 6mm, location: upper right lung, morphology: spiculated}. The server outputs a chest image with spiculated nodules in the upper right lung based on the image generation method of this application.

[0033] In one example, a pre-defined conditional encoder can be used to encode the target conditional text into conditional features. Conditional encoders include text encoders, such as the BERT encoder.

[0034] Step 202: Decompose the conditional features to obtain the low-rank matrix and sparse coding features.

[0035] A low-rank matrix is ​​a matrix whose rank is much smaller than its number of rows and columns (i.e., the smaller of the number of rows and columns). The rank of a matrix is ​​the maximum number of linearly independent row vectors (or column vectors) in the matrix. Sparse coding features typically refer to feature representations learned through sparse coding methods, characterized by using only a small number of non-zero elements (i.e., most coefficients are zero or close to zero) to represent a sample.

[0036] In one example, given conditional features Differentiable low-rank sparse coding modules will incorporate conditional features. Decompose into low-rank matrices Sparse coding : , in, Representing feature dimension, This represents the compressed feature dimension, and r is a preset rank constraint.

[0037] In one embodiment, reference is made to Figure 3 Step 202 may include: Step 301: Use a preset dictionary matrix to perform dictionary learning on the conditional features to obtain dictionary-learned features; Step 302: Perform low-rank projection on the dictionary learning features to obtain dictionary projection features; Step 303: Perform differentiable sparse coding on the dictionary projection features to obtain sparse coded features; Step 304: Truncate the features learned from the dictionary to obtain a low-rank matrix.

[0038] In step 301, within the context of sparse coding, the dictionary matrix is ​​a set of basis vectors (also called dictionary feature vectors) used to represent the signal. The dictionary matrix can be randomly initialized according to actual needs. The dictionary learning process involves calculating the cosine similarity between the input conditionally encoded feature vectors and all vectors in the dictionary, and taking the dictionary feature corresponding to the maximum value as the feature used subsequently.

[0039] In one embodiment, step 301 may include: The first learned feature is obtained by multiplying the transpose of the dictionary matrix with the conditional features; The second learning feature is obtained by multiplying the first learning feature by the transpose of the conditional feature. The second learned feature is multiplied by the dictionary matrix to obtain the dictionary learned feature.

[0040] Specifically, dictionary learning features can be represented as: ,in, It is a learnable dictionary matrix. , Let k be the feature dimension and k be the number of feature vectors in the dictionary. These are conditional features.

[0041] In this way, the ability of dictionary learning features to express conditional text is improved through deep interaction between the dictionary matrix and conditional features.

[0042] In step 302, the purpose of low-rank projection is to project dictionary-learned features onto a low-dimensional space. This refers to projecting a high-dimensional data matrix or vector onto a low-dimensional subspace, resulting in a low-rank structure. This concept has wide applications in tasks such as dimensionality reduction, data compression, denoising, and matrix completion. For example, given a matrix X∈R... m×n The goal of low-rank projection is to find a low-rank approximate matrix X^ such that the rank of X^ satisfies rank(X^)=r min(m,n) and make it as close as possible to the original matrix.

[0043] In step 303, differentiable sparse coding can be implemented using Gumbel-Softmax reparameterization. For example, ,in, This represents the i-th element of the sparse coding feature. It's Gumbel noise. Let represent the activation probability of the i-th primitive. It is a temperature parameter.

[0044] In step 304, the purpose of feature truncation is to preserve the low-rank decomposition of the principal components. For example, ,in, The first r principal components are retained.

[0045] The advantage of the above embodiments lies in achieving end-to-end trainable conditional compression by combining dictionary learning, low-rank constraints, and Gumbel-Softmax reparameterization. Compared with traditional sparse coding methods, this application can automatically learn the optimal compression strategy while maintaining the semantic integrity of the conditions.

[0046] Step 203: Obtain the diffusion model, which consists of K cascaded diffusion layers. The k-th diffusion layer is used as the current diffusion layer, and a preset feature map is used as the input to the current diffusion layer. The diffusion model can be a conditional diffusion generator, based on an improved UNet architecture and integrating cross-scale attention mechanisms at multiple scales (resolutions). This design allows the conditional diffusion generator to adaptively utilize conditional information at different resolution stages, thereby achieving more refined image synthesis. The entire model is trained end-to-end, jointly optimizing conditional compression and image generation objectives.

[0047] In one example, if the current diffusion layer is the first diffusion layer in the diffusion model, the preset feature map is the latent variable distribution sampled from the Gaussian noise distribution; if the current diffusion layer is not the first coupling layer in the diffusion model, the preset feature map is the output of the previous diffusion layer.

[0048] It should be noted that K and k are both positive integers, and k∈[0,K]. The output resolutions of any two diffusion layers are not the same, and the output resolution of the (k+1)th diffusion layer is greater than the output resolution of the kth diffusion layer.

[0049] Step 204: The low-rank matrix, sparse coding features and preset feature map are fused through the current diffusion layer to obtain the next conditional feature, and the next conditional feature is used as the input of the (k+1)th diffusion layer; wherein, the conditional feature output by the Kth diffusion layer in the diffusion model is used as the target feature map.

[0050] In one embodiment, reference is made to Figure 4 Step 204 may include: Step 401: Project the preset feature map using the first projection matrix to obtain the diffusion query vector; Step 402: Perform feature projection on the low-rank matrix using the second projection matrix to obtain the conditional key vector; Step 403: Project the sparse coding features using the third projection matrix to obtain the conditional value vector; Step 404: Perform vector fusion based on the diffusion query vector, condition key vector, and condition value vector to obtain the next conditional feature.

[0051] In step 401, the first projection matrix is ​​used to map the preset feature map (or the conditional features output by the previous diffusion layer) into a diffusion query vector.

[0052] In step 402, the second projection matrix is ​​used to map the low-rank matrix into a conditional key vector.

[0053] In step 402, the third projection matrix is ​​used to map the sparse coded features into a conditional value vector.

[0054] In step 404, the diffusion query vector, condition key vector, and condition value vector can be fused using a preset attention calculation formula to obtain the next conditional feature.

[0055] The advantage of the above embodiments is that conditional features are incorporated into the diffusion process at different output resolutions of each diffusion layer, realizing multi-resolution conditional fusion of the diffusion process. It can automatically adjust the fusion method of conditional information according to the resolution of the current diffusion step, thereby maintaining conditional consistency at multiple scales.

[0056] It should be noted that by adaptively injecting low-rank matrices and sparse coding features into different resolution stages of the diffusion model, fine-grained image generation under multi-scale conditions is achieved. This combination not only improves generation efficiency but also enhances the model's ability to understand complex conditions.

[0057] In one embodiment, reference is made to Figure 5 Step 404 may include: Step 501: Multiply the transpose of the diffusion query vector and the condition key vector to obtain the initial attention weights; Step 502: Apply an activation function to the initial attention weights to obtain the target attention weights; Step 503: Modulate the conditional value vector based on the target attention weight to obtain the next conditionalized feature.

[0058] Specifically, for the input features of diffusion step t The calculation process for the next conditional feature is as follows: , in, , , These are the first, second, and third projective matrices, which are learnable, respectively. c represents the feature vector dimension, and T is the transpose operation. It is a sparse coding feature. The activation function is used. The output of the current diffusion layer becomes the next conditionalized feature, which is then used as the input feature for the next diffusion layer. .

[0059] Step 205: Perform image transformation on the target feature map to obtain the generated target image. For example, the target feature map can be transformed using an MLP layer to obtain the generated target image.

[0060] In one embodiment, obtaining the diffusion model in step 203 includes the following steps: (1) Obtain the sample condition text; (2) Encode the sample condition text to obtain sample condition features; (3) Decompose the sample conditional features to obtain the sample low-rank matrix and sample sparse coding features; (4) The sample low-rank matrix, sample sparse coding features and preset conditional features are fused through the diffusion layer of the initial diffusion model to obtain the sample conditional features of the diffusion layer. (5) Based on the sample conditional features of the last diffusion layer, perform image transformation to obtain the sample generated image; (6) Based on the sample conditional features, the sample low-rank matrix and the sample sparse coding features, the loss is constructed to obtain the sparse reconstruction loss term; (7) Based on the reference image and the generated image, the loss is constructed to obtain the image perception loss term; (8) The target loss function is obtained by weighted fusion of the sparse reconstruction loss term and the image perception loss term; (9) Adjust the parameters of the initial diffusion model based on the target loss function to obtain the diffusion model.

[0061] In step (1), the sample condition text refers to the input variables used to control the image generation process during model training. For example, the sample condition text is: "Patient age: 62 years old; Clinical diagnosis: Alzheimer's disease; Partial scan: T1-weighted image missing about 30% of the area".

[0062] In step (2), the sample condition text is encoded using a condition encoder to obtain sample condition features.

[0063] In step (3), the process of decomposing the sample conditional features can be referred to in step 202, which will not be repeated here.

[0064] In step (4), the process of generating sample conditional features can be referred to in step 204 to generate the next conditional feature, and will not be repeated here.

[0065] In step (5), the MLP can be used to perform image transformation on the sample conditional features of the diffusion layer to obtain the sample generated image.

[0066] In step (6), the sparse reconstruction loss term can be: ,in, For hyperparameters, Represents the square of the 2-norm of a vector or matrix. The 1-norm of a vector or matrix is ​​also known as the Manhattan norm or the sparse promoted norm.

[0067] In step (7), the image perception loss term can be: ,in, Indicates a reference image. Represents the perceptual features of the reference image. This indicates that the sample generates an image. This represents the perceptual features of the generated image from the sample.

[0068] In step (8), the first weight of the sparse creation loss term can be set to be less than the second weight of the image perception loss term, and the sparse reconstruction loss term and the image perception loss term can be weighted and fused based on the first weight and the second weight to obtain the target loss function.

[0069] In step (9), the parameters in the initial diffusion model are updated by the gradient of the target loss function to obtain the diffusion model.

[0070] In one embodiment, before step (9), obtaining the diffusion model in step 203 may further include: analyzing the current diffusion noise based on the sample conditional features and the preset conditional features; constructing a loss term based on the reference diffusion noise and the current diffusion noise; and updating the target loss function based on the diffusion loss term.

[0071] Specifically, the diffusion loss term can be: t is the time step. For reference diffused noise, For the current diffused noise, This represents the learnable parameters of the current diffusion layer.

[0072] Steps 201 to 205, as shown in the embodiments of this application, dynamically compress high-dimensional conditional features into compact low-rank matrices and sparse coding features while preserving semantic fidelity; and for different resolutions, the compressed conditions are fused with the diffusion process to achieve adaptive image generation under multi-resolution conditions.

[0073] In one example, consider a text-to-image generation task where the target conditional text is "a sunflower field under the sun." The generation process includes: 1. A text encoder converts the target conditional text into 768-dimensional conditional features. 2. Conditional features Compressed into a low-rank matrix of 128×32 and 32-dimensional sparse coding The diffusion model uses an attention mechanism to generate conditional features at different resolution stages using compression conditions. Finally, a high-fidelity image of a sunflower field is generated based on the last conditional feature, accurately reflecting the text description.

[0074] In summary, the present application achieves at least the following beneficial effects: 1. Computational efficiency: Conditional compression significantly reduces the computational burden of the diffusion model. 2. Generation quality: The multi-scale conditional fusion mechanism ensures high-fidelity image generation. 3. Adaptability: Dynamic compression and fusion strategies enable the model to adapt to various complex conditional inputs.

[0075] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0076] Please see Figure 6 An embodiment of this application also provides an image generation apparatus that can implement the above-described image generation method. Figure 6 This is a block diagram of the module structure of an image generation apparatus provided in an embodiment of this application. The apparatus includes: a conditional encoding module 601, a conditional decomposition module 602, a model acquisition module 603, a feature fusion module 604, and an image conversion module 605. Detailed descriptions of each functional module are as follows: Conditional encoding module 601 is used to acquire target conditional text, encode the target conditional text, and obtain conditional features; The condition decomposition module 602 is used to decompose the condition features to obtain a low-rank matrix and sparse coding features; The model acquisition module 603 is used to acquire a diffusion model, which includes K cascaded diffusion layers. The kth diffusion layer is used as the model acquisition module for the current diffusion layer. A preset feature map is used as the input of the current diffusion layer. The output resolutions of any two diffusion layers are different. The feature fusion module 604 is used to perform feature fusion on the low-rank matrix, the sparse coding features and the preset feature map through the current diffusion layer to obtain the next conditional feature, and use the next conditional feature as the input of the (k+1)th diffusion layer; wherein, the conditional feature output by the Kth diffusion layer in the diffusion model is used as the target feature map; The image conversion module 605 is used to perform image conversion on the target feature map to obtain the target generated image.

[0077] In one embodiment, the condition decomposition module 602 is specifically used for: The conditional features are learned by performing dictionary learning on the preset dictionary matrix to obtain dictionary-learned features; The dictionary learning features are subjected to low-rank projection to obtain dictionary projection features; Differentiable sparse coding is performed on the dictionary projection features to obtain the sparse coded features; The dictionary-learned features are truncated to obtain the low-rank matrix.

[0078] In one embodiment, the condition decomposition module 602 is specifically used for: The transpose of the dictionary matrix is ​​multiplied by the conditional feature to obtain the first learned feature; The first learned feature is multiplied by the transpose of the conditional feature to obtain the second learned feature; The second learned feature is multiplied by the dictionary matrix to obtain the dictionary learned feature.

[0079] In one embodiment, the feature fusion module 604 is specifically used for: The preset feature map is projected using the first projection matrix to obtain the diffusion query vector; The conditional key vector is obtained by performing feature projection on the low-rank matrix using the second projection matrix. The sparse coded features are projected using a third projection matrix to obtain a conditional value vector; The next conditional feature is obtained by performing vector fusion based on the diffusion query vector, the condition key vector, and the condition value vector.

[0080] In one embodiment, the feature fusion module 604 is specifically used for: The initial attention weights are obtained by multiplying the diffusion query vector and the transpose of the condition key vector. Applying an activation function to the initial attention weights yields the target attention weights; The conditional value vector is modulated based on the target attention weights to obtain the next conditionalized feature.

[0081] In one embodiment, the model acquisition module 603 is specifically used for: Obtain the sample condition text; The sample condition text is encoded to obtain sample condition features; The sample conditional features are decomposed to obtain the sample low-rank matrix and sample sparse coding features; The sample low-rank matrix, the sample sparse coding features, and the preset conditional features are fused through the diffusion layer of the initial diffusion model to obtain the sample conditional features of the diffusion layer. Image transformation is performed based on the sample conditional features of the last diffusion layer to obtain the sample generated image; Based on the sample conditional features, the sample low-rank matrix, and the sample sparse coding features, a loss is constructed to obtain a sparse reconstruction loss term. Based on the reference image and the sample generated image, a loss is constructed to obtain the image perception loss term; The target loss function is obtained by weighted fusion of the sparse reconstruction loss term and the image perception loss term. The initial diffusion model is adjusted based on the target loss function to obtain the diffusion model.

[0082] In one embodiment, before adjusting the parameters of the initial diffusion model based on the target loss function to obtain the diffusion model, the model acquisition module 603 is further configured to: The current diffuse noise is obtained based on the sample conditional features and the preset conditional features. The diffusion loss term is constructed based on the reference diffusion noise and the current diffusion noise. The target loss function is updated based on the diffusion loss term.

[0083] It should be noted that the specific implementation of this image generation device is basically the same as the specific embodiment of the image generation method described above, and will not be repeated here.

[0084] This invention provides an image generation apparatus that dynamically compresses high-dimensional conditional features into a compact low-rank matrix and sparse coded features while preserving semantic fidelity; and for different resolutions, it fuses the compressed conditions with the diffusion process to achieve adaptive image generation under multi-resolution conditions.

[0085] Specific limitations regarding the image generation apparatus can be found in the limitations of the image generation method described above, and will not be repeated here. Each module in the aforementioned image generation apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0086] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of an image generation method on the server side.

[0087] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of an image generation method on the client side.

[0088] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the target conditional text, encode the target conditional text, and obtain conditional features; The conditional features are decomposed to obtain a low-rank matrix and sparse coding features; Obtain a diffusion model, which includes K cascaded diffusion layers. The kth diffusion layer is used as the current diffusion layer, and a preset feature map is used as the input of the current diffusion layer. The output resolutions of any two diffusion layers are different. The low-rank matrix, the sparse coding features, and the preset feature map are fused through the current diffusion layer to obtain the next conditional feature, and the next conditional feature is used as the input of the (k+1)th diffusion layer; wherein, the conditional feature output by the Kth diffusion layer in the diffusion model is used as the target feature map; The target feature map is transformed to obtain the target generated image.

[0089] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the target conditional text, encode the target conditional text, and obtain conditional features; The conditional features are decomposed to obtain a low-rank matrix and sparse coding features; Obtain a diffusion model, which includes K cascaded diffusion layers. The kth diffusion layer is used as the current diffusion layer, and a preset feature map is used as the input of the current diffusion layer. The output resolutions of any two diffusion layers are different. The low-rank matrix, the sparse coding features, and the preset feature map are fused through the current diffusion layer to obtain the next conditional feature, and the next conditional feature is used as the input of the (k+1)th diffusion layer; wherein, the conditional feature output by the Kth diffusion layer in the diffusion model is used as the target feature map; The target feature map is transformed to obtain the target generated image.

[0090] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0091] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0093] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0094] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0095] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0098] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0099] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An image generation method, characterized in that, The method includes: Obtain the target conditional text, encode the target conditional text, and obtain conditional features; The conditional features are decomposed to obtain a low-rank matrix and sparse coding features; Obtain a diffusion model, which includes K cascaded diffusion layers. The kth diffusion layer is used as the current diffusion layer, and a preset feature map is used as the input of the current diffusion layer. The output resolutions of any two diffusion layers are different. The low-rank matrix, the sparse coding features, and the preset feature map are fused through the current diffusion layer to obtain the next conditional feature, and the next conditional feature is used as the input of the (k+1)th diffusion layer; wherein, the conditional feature output by the Kth diffusion layer in the diffusion model is used as the target feature map; The target feature map is transformed to obtain the target generated image.

2. The method according to claim 1, characterized in that, The decomposition of the conditional features to obtain a low-rank matrix and sparse coded features includes: The conditional features are learned by performing dictionary learning on the preset dictionary matrix to obtain dictionary-learned features; The dictionary learning features are subjected to low-rank projection to obtain dictionary projection features; Differentiable sparse coding is performed on the dictionary projection features to obtain the sparse coded features; The dictionary-learned features are truncated to obtain the low-rank matrix.

3. The method according to claim 2, characterized in that, The step of using a preset dictionary matrix to perform dictionary learning on the conditional features to obtain dictionary-learned features includes: The transpose of the dictionary matrix is ​​multiplied by the conditional feature to obtain the first learned feature; The first learned feature is multiplied by the transpose of the conditional feature to obtain the second learned feature; The second learned feature is multiplied by the dictionary matrix to obtain the dictionary learned feature.

4. The method according to any one of claims 1 to 3, characterized in that, The step of fusing features from the low-rank matrix, the sparse coded features, and the preset feature map through the current diffusion layer to obtain the next conditionalized features includes: The preset feature map is projected using the first projection matrix to obtain the diffusion query vector; The conditional key vector is obtained by performing feature projection on the low-rank matrix using the second projection matrix. The sparse coded features are projected using a third projection matrix to obtain a conditional value vector; The next conditional feature is obtained by performing vector fusion based on the diffusion query vector, the condition key vector, and the condition value vector.

5. The method according to claim 4, characterized in that, The step of fusing the diffusion query vector, the conditional key vector, and the conditional value vector to obtain the next conditionalized feature includes: The initial attention weights are obtained by multiplying the diffusion query vector and the transpose of the condition key vector. Applying an activation function to the initial attention weights yields the target attention weights; The conditional value vector is modulated based on the target attention weights to obtain the next conditionalized feature.

6. The method according to any one of claims 1 to 3, characterized in that, The acquisition diffusion model includes: Obtain the sample condition text; The sample condition text is encoded to obtain sample condition features; The sample conditional features are decomposed to obtain the sample low-rank matrix and sample sparse coding features; The sample low-rank matrix, the sample sparse coding features, and the preset conditional features are fused through the diffusion layer of the initial diffusion model to obtain the sample conditional features of the diffusion layer. Image transformation is performed based on the sample conditional features of the last diffusion layer to obtain the sample generated image; Based on the sample conditional features, the sample low-rank matrix, and the sample sparse coding features, a loss is constructed to obtain a sparse reconstruction loss term. Based on the reference image and the sample generated image, a loss is constructed to obtain the image perception loss term; The target loss function is obtained by weighted fusion of the sparse reconstruction loss term and the image perception loss term. The initial diffusion model is adjusted based on the target loss function to obtain the diffusion model.

7. The method according to claim 6, characterized in that, Before adjusting the parameters of the initial diffusion model based on the target loss function to obtain the diffusion model, the method further includes: The current diffuse noise is obtained based on the sample conditional features and the preset conditional features. The diffusion loss term is constructed based on the reference diffusion noise and the current diffusion noise. The target loss function is updated based on the diffusion loss term.

8. An image generation apparatus, characterized in that, The device includes: The conditional encoding module is used to acquire target conditional text, encode the target conditional text, and obtain conditional features; The condition decomposition module is used to decompose the conditional features to obtain a low-rank matrix and sparse coding features; The model acquisition module is used to acquire a diffusion model, which includes K cascaded diffusion layers. The kth diffusion layer is used as the model acquisition module for the current diffusion layer. A preset feature map is used as the input of the current diffusion layer. The output resolutions of any two diffusion layers are different. The feature fusion module is used to fuse the low-rank matrix, the sparse coding features, and the preset feature map through the current diffusion layer to obtain the next conditional feature, and use the next conditional feature as the input of the (k+1)th diffusion layer; wherein, the conditional feature output by the Kth diffusion layer in the diffusion model is used as the target feature map; The image conversion module is used to perform image conversion on the target feature map to obtain the target generated image.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the image generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the image generation method according to any one of claims 1 to 7.

Images