Visual generation method and device, electronic device, and storage medium
By generating masking tokens and noise tokens, and utilizing a combination of a backbone network, a diffusion head, and a denoising head, the problem of insufficient token prediction accuracy in visual generation tasks by autoregressive models is solved, achieving high-quality image generation and improved efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-11
- Publication Date
- 2026-03-17
AI Technical Summary
Autoregressive models suffer from insufficient token prediction accuracy and low image quality in visual generation tasks because fixed-order token prediction is unsuitable for the image content generation process.
By acquiring images and context tokens, masking tokens and noise tokens are generated. Conditional token generation is performed by deeply fusing global context information using the backbone network. Combined with a diffusion head and a denoising head, high-quality images are generated.
It improves the prediction accuracy of tokens, achieves high-quality image generation, solves the inconsistency problem between training and inference processes, and maintains high generation efficiency.
Smart Images
Figure CN120852214B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of visual generation technology, and in particular to a visual generation method, apparatus, electronic device, and storage medium. Background Technology
[0002] Autoregressive models are among the mainstream generative models today. Their core idea lies in generating content tokens sequentially based on a "next token prediction" mechanism. They have achieved great success in large language models (LLMs) within the field of Natural Language Processing (NLP). Inspired by this, some research has begun to apply autoregressive models to visual generation tasks, such as text-to-image generation.
[0003] However, applying autoregressive models to the field of vision presents inherent challenges. The generation process of image content differs from that of language text, which is inherently linear. Fixed-order token prediction (such as raster scanning order) is not optimal, resulting in insufficient prediction accuracy and low-quality generated images. Summary of the Invention
[0004] To address one of the aforementioned technical deficiencies, this application provides a solution.
[0005] A first aspect of this application provides a visual generation method, the method comprising:
[0006] Get the image and context token;
[0007] Based on the image, a visual generation token is generated; wherein, the visual generation token includes an occlusion token and a noise token;
[0008] Based on visually generated tokens and context tokens, conditional tokens that deeply integrate global context information are generated through the backbone network;
[0009] High-quality images are generated using a diffusion head and a denoising head, based on conditional tokens.
[0010] Optionally, a visual generation token is generated based on the image, including:
[0011] An encoder using a variational autoencoder converts an image into a sequence of visual tokens.
[0012] Randomly select a preset proportion of tokens from the visual token sequence;
[0013] Update the selected token to a masking token, inject Gaussian noise into the unselected tokens, and obtain a noise token;
[0014] Both the occlusion token and the noise token are defined as visual generation tokens.
[0015] Optionally, the backbone network consists of multiple identical layers;
[0016] Each layer is processed through the following steps:
[0017]
[0018] Where i is the layer identifier, z i The input is the i-th layer, SelfAttn() is the self-attention mechanism, norm() is the regularization process, and FFN() is the feedforward network. For the intermediate result of the i-th layer, z i+1 This is the output of the i-th layer;
[0019] The first layer takes visual generation tokens and context tokens as input, and the last layer outputs condition tokens that are deeply fused with global context information.
[0020] Optionally, based on conditional tokens, a high-quality image is generated via a diffusion head and a denoising head, including:
[0021] Based on conditional tokens, the expression of masking tokens is predicted using the diffusion head;
[0022] Based on conditional tokens, the denoised representation of noise tokens is predicted by a denoising head;
[0023] The representations of the masked tokens and the denoised representations of the noise tokens are input into the variational autoencoder according to the order of the tokens in the image to generate a high-quality image.
[0024] Optionally, the diffuser head is a converter;
[0025] Based on conditional tokens, the representation of masking tokens is predicted using the diffusion head, including:
[0026] By interacting with the information of the masking token and the conditional token through the cross-attention layer of the converter, the expression of the masking token is predicted.
[0027] Optionally, the denoising head includes parallel token refinement branches and token evaluation branches;
[0028] Based on conditional tokens, the denoised representation of noise tokens is predicted using a denoising head, including:
[0029] Minimize the difference between the refined token and the corresponding unselected token by refining the token of the noisy token through the token refinement branch;
[0030] The quality scores of the refined token and the corresponding unselected token of the noisy token are evaluated by the token evaluation branch. The refined token and the corresponding unselected token are then weighted and fused according to the quality scores to obtain the denoised representation of the noisy token.
[0031] Optionally, the token refinement branch uses a multilayer perceptron network to predict the refined token after denoising the noisy token based on the conditional token.
[0032] A second aspect of this application provides a vision generation apparatus, the apparatus comprising:
[0033] The acquisition module is used to obtain the image and context token;
[0034] The processing module is used to generate visual generation tokens based on the image; wherein, the visual generation tokens include occlusion tokens and noise tokens;
[0035] The backbone network is used to generate conditional tokens that deeply integrate global context information, based on visually generated tokens and context tokens.
[0036] A diffusion head and a denoising head are used to generate high-quality images based on conditional tokens.
[0037] A third aspect of this application provides an electronic device, comprising:
[0038] Memory;
[0039] Processor; and
[0040] Computer programs;
[0041] The computer program is stored in the memory and configured to be executed by the processor to implement the method described in the first aspect above.
[0042] In a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method described in the first aspect above.
[0043] This application provides a visual generation method, apparatus, electronic device, and storage medium. The method includes: acquiring an image and a context token; generating a visual generation token based on the image; wherein the visual generation token includes an occlusion token and a noise token; generating a conditional token that deeply fuses global contextual information through a backbone network based on the visual generation token and the context token; and generating a high-quality image based on the conditional token through a diffusion head and a denoising head. This method, after generating a visual generation token from the image, generates a conditional token that deeply fuses global contextual information through a backbone network based on the visual generation token and the context token. Based on the conditional token, a high-quality image is generated through a diffusion head and a denoising head. This fully utilizes global contextual information, improves the prediction accuracy of the token, and achieves high-quality image generation. Attached Figure Description
[0044] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0045] Figure 1 A flowchart illustrating a visual generation method provided in an embodiment of this application;
[0046] Figure 2 A schematic diagram illustrating the implementation principle of a visual generation method provided in an embodiment of this application;
[0047] Figure 3 A schematic diagram illustrating the implementation principle of a backbone network provided in an embodiment of this application;
[0048] Figure 4 A schematic diagram illustrating the implementation principle of a diffusion head provided in an embodiment of this application;
[0049] Figure 5 A schematic diagram illustrating the implementation principle of a noise reduction head provided in an embodiment of this application;
[0050] Figure 6 This is a schematic diagram of the structure of a vision generation device provided in an embodiment of this application;
[0051] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0052] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.
[0053] In developing this application, the inventors discovered that applying autoregressive models to the field of vision presents inherent challenges. The generation process of image content differs from that of language text, which is inherently linear. Fixed-order token prediction (such as raster scan order) is not optimal, resulting in insufficient token prediction accuracy and low-quality generated images.
[0054] To address the aforementioned problems, this application provides a visual generation method, apparatus, electronic device, and storage medium. The method involves: acquiring an image and a context token; generating a visual generation token based on the image; wherein the visual generation token includes an occlusion token and a noise token; generating a conditional token that deeply fuses global contextual information through a backbone network based on the visual generation token and the context token; and generating a high-quality image based on the conditional token through a diffusion head and a denoising head. This method, after generating a visual generation token from the image, generates a conditional token that deeply fuses global contextual information through a backbone network based on the visual generation token and the context token. Based on the conditional token, a high-quality image is generated through a diffusion head and a denoising head. This fully utilizes global contextual information, improves the prediction accuracy of the token, and achieves high-quality image generation.
[0055] See Figure 1 This embodiment provides a visual generation method, the implementation process of which is as follows:
[0056] 101, Get the image and context token.
[0057] Context tokens, such as category or text tokens, serve as contextual conditions for image generation, ensuring that the generated image conforms to the contextual conditions and guaranteeing the quality of the generated image.
[0058] For example, a context token is a multidimensional feature vector.
[0059] In such Figure 2 The implementation principle of the visual generation method shown is achieved through step 101. Figure 2 InputImage (input image) in the context of this.
[0060] 102. Generate a visual token based on the image.
[0061] The visual generation tokens include occlusion tokens and noise tokens.
[0062] Step 102 can be achieved through the following steps:
[0063] 102-1, an image is converted into a sequence of visual tokens by an encoder using a variational autoencoder (VAE).
[0064] In such Figure 2 The implementation principle of the visual generation method shown is achieved through step 102-1. Figure 2 The VAEEncoder (VAE encoding) in the code forms a visual token sequence. Figure 2 Visual Tokens (a sequence of visual tokens).
[0065] 102-2, Randomly select a preset proportion of tokens from the visual token sequence.
[0066] The token position is randomly selected during the selection process. A preset ratio can be set according to actual conditions; for example, the preset ratio can range from 70% to 100%.
[0067] 102-3, update the selected token to the masking token, inject Gaussian noise into the unselected token to obtain the noise token.
[0068] The initial values of the parameters in the masking token can be sampled from a Gaussian distribution.
[0069] In such Figure 2 The implementation principle of the visual generation method shown is achieved through step 102-3. Figure 2 Mask and noise in the context of [the topic].
[0070] 102-4, both the occlusion token and the noise token are defined as visual generation tokens.
[0071] In step 102, the image acquired in step 101 is first converted into a visual token sequence using a variational autoencoder (VAE). A portion of the tokens in this sequence are randomly replaced with occluded tokens (i.e., tokens at occluded locations are occluded tokens), while another portion of the tokens (unselected tokens, i.e., tokens at unoccluded locations) are injected with Gaussian noise to form noise tokens.
[0072] In practical implementation, after obtaining the visual generation token, it can be rearranged, such as... Figure 2 The reshape operation yields the occlusion token as the occlusion token, i.e. Figure 2 In the context of Masked TokensX″′, noise tokens are unmasked tokens, i.e. Figure 2 The Unmasked TokensX in the middle.
[0073] 103. Based on visually generated tokens and context tokens, condition tokens with deep fusion of global context information are generated through the backbone network.
[0074] The backbone network can be MAR Transformers (Masked Autoregressive ModelTransformers), which can mine bidirectional dependencies between all tokens.
[0075] MAR Transformers can be stacked from multiple standard transformer modules, meaning that the backbone network consists of multiple identical layers, each layer being a transformer.
[0076] Each layer is processed through the following steps:
[0077]
[0078] Where i is the layer identifier, z i The input is the i-th layer, SelfAttn() is the self-attention mechanism, norm() is the regularization process, and FFN() is the feedforward network. For the intermediate result of the i-th layer, z i+1 This is the output of the i-th layer.
[0079] The first layer takes visual generation tokens and context tokens as input, and the last layer outputs condition tokens that are deeply fused with global context information.
[0080] In such Figure 2 In the implementation principle of the visual generation method shown, during step 103, the backbone network (MARTransformers) can receive visual generation tokens (including occlusion tokens and noise tokens) and context tokens. It captures the long-range, bidirectional dependencies between all tokens in the sequence through a self-attention mechanism. For each token position in the visual generation tokens (whether occluded or unoccluded), a conditional token (i.e., one that deeply fuses global context information) is generated. Figure 2 The Conditional Tonkens Z provides the necessary conditional input for subsequent denoising and diffusion heads.
[0081] In practical implementation, the backbone network (MAR Transformers) can be implemented as follows: Figure 3 As shown, the backbone network (MAR Transformers) sequentially passes through regularization (i.e., Figure 3 LayerNorm in (i.e., self-attention mechanism) Figure 3 Self Attention and feedforward networks (i.e., ...) Figure 3 Feed Forward in the context of visual generation tokens and context tokens (i.e. Figure 3 In This process generates a conditional token that deeply integrates global context information.
[0082] In other words, the backbone network (MAR Transformers) performs self-attention operations on all regularized tokens, allowing visually generated tokens and context tokens to pass through the feedforward network. The output is a conditional token that incorporates global context information.
[0083] In such Figure 2In the implementation principle of the visual generation method shown, MAR Transformers is the backbone network of step 103. Step 103 is implemented through MAR Transformers to obtain Conditional Tokens Z (i.e., conditional tokens that deeply fuse global context information).
[0084] 104, Based on conditional tokens, generates high-quality images through a diffusion head and a denoising head.
[0085] The implementation process of step 104 is as follows:
[0086] 1. Based on conditional tokens, predict the representation of masking tokens using diffusion heads.
[0087] The diffusion head processes the masking token, utilizing the rich contextual information provided by the conditional token to facilitate the prediction of the masked token, and thus predict the expression of the masking token.
[0088] The diffusion head can solve the problem of insufficient use of context.
[0089] The diffuser is a converter that interacts with the masking token and conditional token information through the converter's cross-attention layer to predict the representation of the masking token. For example, the diffuser models the probability distribution of the masking token and predicts its content through a standard denoising diffuser process.
[0090] Unlike existing conditional features that only use a single token location, the diffusion head in this embodiment takes all conditional tokens (including those at unmasked and masked locations) output by the MARTransformer as a whole context sequence as input. Through a cross-attention layer, the diffusion head enables each masked token to be predicted to interact with the conditional tokens, allowing the prediction of each masked token to fully utilize the global information of the entire image, rather than being limited to local features. This significantly improves prediction accuracy and the overall consistency of the generated image.
[0091] In the actual implementation, the diffusion head starts with random Gaussian noise and, guided by the global context of the condition tokens, gradually generates all the masking tokens through a multi-step back diffusion process.
[0092] In such Figure 2 In the implementation principle of the visual generation method shown, the Diffusion Head is the diffusion head. The implementation principle of the diffusion head can be as follows: Figure 4As shown, the masking token undergoes noise (i.e., adding noise), LayerNorm (regularization), CrossAttention (cross-attention mechanism), scaling and other processing in sequence to predict the representation of the masking token.
[0093] In the specific implementation, the condition token obtained in step 103 and the occlusion token obtained in step 102 are used as inputs to the diffusion head. In the diffusion head, the condition token and the occlusion token (which have the same content and position) serve as condition information, and the occlusion token is given condition information through a scaling operation.
[0094] All conditional tokens are used in the cross-attention mechanism, allowing the occlusion token to utilize information from itself and surrounding partial image blocks.
[0095] A diffusion head is a network in which different input variables interact without any order.
[0096] 2. Based on conditional tokens, the denoised representation of noise tokens is predicted by a denoising head.
[0097] The noise reduction head processes the noise token and predicts the original noise-free version of the noise token.
[0098] Noise reduction heads can solve the problem of inconsistency between training and inference in existing technologies.
[0099] The denoising head includes parallel token refinement and token evaluation branches.
[0100] 1) Token Refinement Branch
[0101] Minimize the difference between the refined token and the corresponding unselected token by refining the token branch.
[0102] For example, the token refinement branch uses a multilayer perceptron (MLP) network to predict the refined token after denoising the noisy token based on the conditional token.
[0103] In practical implementation, noise tokens and their corresponding condition tokens can be concatenated, and the concatenated tokens can be input into the token refinement branch to obtain the output as the refined tokens.
[0104] The implementation principle of noise reduction head is as follows: Figure 5 As shown, the token refinement branch is... Figure 5 The Token Refinement Branch receives a noisy, unmasked token (i.e., a noisy token) and its corresponding conditional token (i.e., a conditional token), and then uses a multilayer perceptron (MLP) network to predict the "clean" version of the token after it has been denoised.
[0105] By refining the token branch, the difference between the refined token and the original real token can be minimized.
[0106] 2) Token Evaluation Branch
[0107] The quality scores of the refined token and the corresponding unselected token (the unselected token is the token before noise was added in step 102-3) of the noisy token are evaluated by the token evaluation branch. The refined token and the corresponding unselected token are weighted and fused according to the quality scores to obtain the denoised expression of the noisy token.
[0108] In the specific implementation, the concatenated noise token, its corresponding conditional token, and the refined token output from the token refinement branch are all input into the token evaluation branch. The token evaluation branch outputs a quality score for the noise token and its refined version. Based on the quality score, a weighted fusion is calculated for the unmasked token and its refined version to obtain the optimized noise token representation.
[0109] The implementation principle of noise reduction head is as follows: Figure 5 As shown, the token evaluation branch is Figure 5 The Token Assessment Branch receives a noisy token, a refined token (i.e., a refined token of the noisy token predicted by the token refinement branch), and a conditional token, and evaluates the quality scores of the two tokens before and after refinement.
[0110] The token evaluation branch assigns a quality score to each noisy token and its refined version. The final output token is a weighted fusion of these two scores, which dynamically adjusts the refinement intensity and preserves the better token expression.
[0111] The token evaluation branch allows for the incremental optimization and refinement of the tokens predicted during the generation process, making them closer to the actual data distribution and effectively mitigating the problem of declining generation quality caused by error accumulation.
[0112] In such Figure 2 In the implementation principle of the visual generation method shown, the Denoising Head is the noise reduction head. During training, the noise reduction head can purposefully add noise to the original tokens and learn to remove it. During inference, it can dynamically refine and evaluate the generated tokens. It can align the data distribution of the training and inference stages, thereby mitigating the performance degradation caused by error accumulation during inference and ensuring the stability and quality of image generation.
[0113] 3. Input the representation of the masked token and the denoised representation of the noise token into the variational autoencoder (VAE) according to the order of the tokens in the image to generate a high-quality image.
[0114] After obtaining the representation of the masked token and the denoised representation of the noise token, all the predicted and refined tokens are reconstructed into a complete sequence and converted back into the final pixel image by the VAE decoder.
[0115] In such Figure 2 The implementation principle of the visual generation method shown is to reshape the tokens to make them conform to a normal order of the image, input the reordered tokens into VAE, and obtain the pixel image through VAEDecoder.
[0116] The visual generation method provided in this embodiment acquires an image and a context token, accurately predicts the visual token of the occluded part of the image, and simultaneously denoises and thins the visual token of the unoccluded part, finally generating a high-quality image consistent with the context token.
[0117] The visual generation method provided in this embodiment fully utilizes global contextual information by introducing a transformer-based diffusion head to improve the prediction accuracy of occluded tokens. At the same time, it progressively refines noisy tokens through a denoising head, thereby effectively bridging the gap between training and inference, and ultimately achieving higher quality image generation.
[0118] Compared to existing technologies, the visual generation method provided in this embodiment achieves significant technical advantages through a denoised token prediction mechanism, improving generation quality, model robustness, and operational efficiency. Firstly, the converter-based diffusion head provided in this embodiment plays a crucial role in improving the quality and accuracy of generated images. Through a cross-attention mechanism, it enables the prediction of occluded tokens to utilize global contextual information from all unoccluded tokens. This more comprehensive use of context makes the predicted tokens more accurate, and the generated images are more consistent with the target in detail and structure, resulting in higher fidelity. Secondly, the visual generation method provided in this embodiment effectively solves the inconsistency problem between the training and inference processes, mainly thanks to the denoising head. The denoising head successfully aligns the data distribution between the training and inference stages by purposefully adding noise to the original tokens during training and learning to denoise them, and by dynamically refining and evaluating the generated tokens during inference. This alleviates the performance degradation caused by error accumulation during inference, ensuring stability and image generation quality. Finally, the visual generation method provided in this embodiment maintains high generation efficiency while achieving the aforementioned performance improvements. Since the backbone network (MAR Transformers) operates within the MAR framework, it has the inherent advantage of predicting multiple tokens in parallel. Therefore, compared to the standard diffusion model that requires hundreds or even thousands of iterations, the visual generation method provided in this embodiment is faster in inference speed, achieving a balance between performance and efficiency.
[0119] This embodiment provides a visual generation method that acquires an image and a context token; generates a visual generation token based on the image; wherein the visual generation token includes an occlusion token and a noise token; generates a conditional token that deeply fuses global context information through a backbone network based on the visual generation token and the context token; and generates a high-quality image based on the conditional token through a diffusion head and a denoising head. The method provided in this embodiment, after generating a visual generation token from the image, generates a conditional token that deeply fuses global context information through a backbone network based on the visual generation token and the context token, and generates a high-quality image based on the conditional token through a diffusion head and a denoising head. This method can fully utilize global context information, improve the prediction accuracy of the token, and achieve high-quality image generation.
[0120] Based on the same inventive concept of visual generation methods, this embodiment provides a visual generation device, see [link to relevant documentation]. Figure 6 The device includes:
[0121] Module 601 is used to obtain the image and context token.
[0122] Processing module 602 is used to generate a visual generation token based on the image.
[0123] Backbone network 603 is used to generate conditional tokens that deeply integrate global context information based on visual generation tokens and context tokens.
[0124] The diffusion head and denoising head 604 are used to generate high-quality images based on conditional tokens through the diffusion head and denoising head.
[0125] The processing module 602 is used to convert the image into a visual token sequence using a variational autoencoder. A preset proportion of tokens are randomly selected from the visual token sequence. The selected tokens are updated to occlusion tokens, and Gaussian noise is injected into the unselected tokens to obtain noise tokens. Both the occlusion tokens and the noise tokens are then determined as visually generated tokens.
[0126] The backbone network consists of multiple identical layers.
[0127] Each layer is processed through the following steps:
[0128]
[0129] Where i is the layer identifier, z i The input is the i-th layer, SelfAttn() is the self-attention mechanism, norm() is the regularization process, and FFN() is the feedforward network. For the intermediate result of the i-th layer, z i+1 This is the output of the i-th layer.
[0130] The first layer takes visual generation tokens and context tokens as input, and the last layer outputs condition tokens that are deeply fused with global context information.
[0131] The diffusion head and denoising head 604 are used to predict the representation of the occlusion token based on the conditional tokens. The denoising head then predicts the denoised representation of the noise token based on the conditional tokens. The representations of the occlusion tokens and the denoised representations of the noise tokens are input into the variational autoencoder according to the order of the tokens in the image to generate a high-quality image.
[0132] The diffuser head is a converter.
[0133] Based on conditional tokens, the representation of masking tokens is predicted using the diffusion head, including:
[0134] By interacting with the information of the masking token and the conditional token through the cross-attention layer of the converter, the expression of the masking token is predicted.
[0135] The denoising head includes parallel token refinement and token evaluation branches.
[0136] Based on conditional tokens, the denoised representation of noise tokens is predicted using a denoising head, including:
[0137] Minimize the difference between the refined token and the corresponding unselected token by refining the token branch.
[0138] The quality scores of the refined token and the corresponding unselected token of the noisy token are evaluated by the token evaluation branch. The refined token and the corresponding unselected token are then weighted and fused according to the quality scores to obtain the denoised representation of the noisy token.
[0139] The token refinement branch uses a multilayer perceptron network to predict the refined token after denoising the noisy token based on the conditional token.
[0140] The device provided in this embodiment generates a visual generation token based on an image, and then generates a conditional token that deeply fuses global contextual information through a backbone network based on the visual generation token and a context token. Based on the conditional token, a high-quality image is generated through a diffusion head and a denoising head. This fully utilizes global contextual information, improves the prediction accuracy of the token, and achieves high-quality image generation.
[0141] Based on the same inventive concept of visual generation methods, this embodiment provides an electronic device, which is as follows: Figure 7 As shown, it includes: a memory 701, a processor 702, and a computer program.
[0142] The computer program is stored in memory 701 and configured to be executed by processor 702 to implement the above-described visual generation method.
[0143] Specifically,
[0144] Get the image and context token.
[0145] Based on the image, a visual generation token is generated. This visual generation token includes an occlusion token and a noise token.
[0146] Based on visually generated tokens and context tokens, condition tokens that deeply integrate global context information are generated through the backbone network.
[0147] High-quality images are generated using a diffusion head and a denoising head, based on conditional tokens.
[0148] The generation of visual generation tokens based on images includes:
[0149] An encoder using a variational autoencoder converts an image into a sequence of visual tokens.
[0150] Randomly select a preset proportion of tokens from the visual token sequence.
[0151] Update the selected token to a masking token, inject Gaussian noise into the unselected tokens, and obtain a noise token.
[0152] Both the occlusion token and the noise token are defined as visual generation tokens.
[0153] The backbone network consists of multiple identical layers.
[0154] Each layer is processed through the following steps:
[0155]
[0156] Where i is the layer identifier, z i The input is the i-th layer, SelfAttn() is the self-attention mechanism, norm() is the regularization process, and FFN() is the feedforward network. For the intermediate result of the i-th layer, z i+1 This is the output of the i-th layer.
[0157] The first layer takes visual generation tokens and context tokens as input, and the last layer outputs condition tokens that are deeply fused with global context information.
[0158] Among them, high-quality images are generated based on conditional tokens through a diffusion head and a denoising head, including:
[0159] Based on conditional tokens, the expression of masking tokens is predicted using a diffusion head.
[0160] Based on conditional tokens, the denoised representation of noise tokens is predicted using a denoising head.
[0161] The representations of the masked tokens and the denoised representations of the noise tokens are input into the variational autoencoder according to the order of the tokens in the image to generate a high-quality image.
[0162] The diffuser head is a converter.
[0163] Based on conditional tokens, the representation of masking tokens is predicted using the diffusion head, including:
[0164] By interacting with the information of the masking token and the conditional token through the cross-attention layer of the converter, the expression of the masking token is predicted.
[0165] The denoising head includes parallel token refinement and token evaluation branches.
[0166] Based on conditional tokens, the denoised representation of noise tokens is predicted using a denoising head, including:
[0167] Minimize the difference between the refined token and the corresponding unselected token by refining the token branch.
[0168] The quality scores of the refined token and the corresponding unselected token of the noisy token are evaluated by the token evaluation branch. The refined token and the corresponding unselected token are then weighted and fused according to the quality scores to obtain the denoised representation of the noisy token.
[0169] The token refinement branch uses a multilayer perceptron network to predict the refined token after denoising the noisy token based on the conditional token.
[0170] The electronic device provided in this embodiment has a computer program executed by a processor to generate a visual generation token based on an image. Then, based on the visual generation token and the context token, a condition token with deep fusion of global context information is generated through a backbone network. Based on the condition token, a high-quality image is generated through a diffusion head and a denoising head. This can make full use of global context information, improve the prediction accuracy of the token, and achieve high-quality image generation.
[0171] Based on the same inventive concept as the visual generation method, this embodiment provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the above-described visual generation method.
[0172] Specifically,
[0173] Get the image and context token.
[0174] Based on the image, a visual generation token is generated. This visual generation token includes an occlusion token and a noise token.
[0175] Based on visually generated tokens and context tokens, condition tokens that deeply integrate global context information are generated through the backbone network.
[0176] High-quality images are generated using a diffusion head and a denoising head, based on conditional tokens.
[0177] The generation of visual generation tokens based on images includes:
[0178] An encoder using a variational autoencoder converts an image into a sequence of visual tokens.
[0179] Randomly select a preset proportion of tokens from the visual token sequence.
[0180] Update the selected token to a masking token, inject Gaussian noise into the unselected tokens, and obtain a noise token.
[0181] Both the occlusion token and the noise token are defined as visual generation tokens.
[0182] The backbone network consists of multiple identical layers.
[0183] Each layer is processed through the following steps:
[0184]
[0185] Where i is the layer identifier, z i The input is the i-th layer, SelfAttn() is the self-attention mechanism, norm() is the regularization process, and FFN() is the feedforward network. For the intermediate result of the i-th layer, z i+1 This is the output of the i-th layer.
[0186] The first layer takes visual generation tokens and context tokens as input, and the last layer outputs condition tokens that are deeply fused with global context information.
[0187] Among them, high-quality images are generated based on conditional tokens through a diffusion head and a denoising head, including:
[0188] Based on conditional tokens, the expression of masking tokens is predicted using a diffusion head.
[0189] Based on conditional tokens, the denoised representation of noise tokens is predicted using a denoising head.
[0190] The representations of the masked tokens and the denoised representations of the noise tokens are input into the variational autoencoder according to the order of the tokens in the image to generate a high-quality image.
[0191] The diffuser head is a converter.
[0192] Based on conditional tokens, the representation of masking tokens is predicted using the diffusion head, including:
[0193] By interacting with the information of the masking token and the conditional token through the cross-attention layer of the converter, the expression of the masking token is predicted.
[0194] The denoising head includes parallel token refinement and token evaluation branches.
[0195] Based on conditional tokens, the denoised representation of noise tokens is predicted using a denoising head, including:
[0196] Minimize the difference between the refined token and the corresponding unselected token by refining the token branch.
[0197] The quality scores of the refined token and the corresponding unselected token of the noisy token are evaluated by the token evaluation branch. The refined token and the corresponding unselected token are then weighted and fused according to the quality scores to obtain the denoised representation of the noisy token.
[0198] The token refinement branch uses a multilayer perceptron network to predict the refined token after denoising the noisy token based on the conditional token.
[0199] The computer-readable storage medium provided in this embodiment has a computer program thereon that is executed by a processor to generate a visual generation token based on an image. Then, based on the visual generation token and the context token, a conditional token with deep fusion of global context information is generated through a backbone network. Based on the conditional token, a high-quality image is generated through a diffusion head and a denoising head. This can make full use of global context information, improve the prediction accuracy of the token, and achieve high-quality image generation.
[0200] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.
[0201] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0204] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0205] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method of visual generation, characterized by, The method comprises: acquiring an image and a context token; generating a visual generation token according to the image; wherein the visual generation token comprises a masking token and a noise token; generating a condition token deeply fused with global context information by a backbone network based on the visual generation token and the context token; generating a high-quality image by a diffusion head and a denoising head based on the condition token; wherein the generating of the high-quality image by the diffusion head and the denoising head based on the condition token comprises: predicting an expression of the masking token by the diffusion head based on the condition token; predicting a denoised expression of the noise token by the denoising head based on the condition token; inputting the expression of the masking token and the denoised expression of the noise token into a variational autoencoder in the order of the tokens in the image to generate the high-quality image.
2. The method of claim 1, wherein, The generating of the visual generation token according to the image comprises: converting the image into a visual token sequence by an encoder of the variational autoencoder; randomly selecting tokens of a preset proportion from the visual token sequence; updating the selected tokens into the masking token and injecting Gaussian noise into the unselected tokens to obtain the noise token; determining the masking token and the noise token as the visual generation token.
3. The method of claim 1, wherein, The backbone network is composed of multiple identical layers; each layer is processed by the following steps: ； ； wherein, is a layer identifier, is a first layer input, is a self-attention mechanism, is a regularization procedure, is a forward network, is an intermediate result of a first layer, is an output of a first layer; the input of the first layer is the visual generation token and the context token, and the output of the last layer is the condition token deeply fused with global context information.
4. The method of claim 1, wherein, The diffusion head is a converter. The predicting of the expression of the masking token by the diffusion head based on the condition token comprises: interacting information of the masking token and the condition token by a cross-attention layer of the converter to predict the expression of the masking token.
5. The method of claim 1, wherein, The denoising head comprises a token refinement branch and a token evaluation branch in parallel. The predicting of the denoised expression of the noise token by the denoising head based on the condition token comprises: minimizing the difference between the refined token of the noise token and the corresponding unselected token by the token refinement branch; evaluating the quality score of the refined token of the noise token and the corresponding unselected token by the token evaluation branch, and performing weighted fusion on the refined token and the corresponding unselected token according to the quality score to obtain the denoised expression of the noise token.
6. The method of claim 5, wherein, The token refinement branch predicts the refined token of the noise token after denoising based on the condition token through a multi-layer perceptron network.
7. A visual generating apparatus characterized by comprising: The device comprises: an acquisition module for acquiring an image and a context token; a processing module for generating a visual generation token according to the image; wherein the visual generation token comprises a masking token and a noise token; a backbone network for generating a condition token deeply fused with global context information based on the visual generation token and the context token; a diffusion head and a denoising head for predicting an expression of the masking token by the diffusion head based on the condition token, predicting a denoised expression of the noise token by the denoising head based on the condition token, and inputting the expression of the masking token and the denoised expression of the noise token into a variational autoencoder in the order of the tokens in the image to generate a high-quality image.
8. An electronic device, comprising: comprising: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-6. a computer program stored thereon; the computer program being executed by a processor to implement the method of any one of claims 1-6.
9. A computer-readable storage medium, characterized in that,