Method and device for generating image based on model

Abstract

The embodiment of the invention provides a method and device for generating an image based on a model. The method comprises the steps that a first image is generated by an autoregression model based on prompt information, the prompt information is used for indicating generation of visual content, and the first image comprises a plurality of first image blocks which are sequentially generated; determining first position information of a first low-quality image block in the plurality of first image blocks, wherein the first low-quality image block is a first image block of which the quality score does not meet a preset requirement; and inputting the first position information into an autoregression model, and regenerating a second image block corresponding to the first position information by the autoregression model at least based on the plurality of target image blocks or the key vectors and the value vectors corresponding to the plurality of target image blocks. According to the method, local image block regeneration can be realized through accurate positioning of the position information of the image blocks.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of image processing technology, and specifically, to a method and apparatus for generating images based on a model. Background Technology

[0002] Video content generation technology, as an important research direction in the field of artificial intelligence, aims to automatically expand and generate corresponding visual content based on user-provided text, images, or video instructions. It has broad application prospects and significant commercial value.

[0003] In recent years, the successful application of autoregressive models (AR models) in large language models has provided new ideas for multimodal generation tasks, such as the LlamaGen model and the VAR (Visual Autoregressive) model. These models decompose the long sequence generation problem into continuous single-step reasoning by progressively predicting the next image token in the sequence, significantly reducing modeling complexity.

[0004] In the process of generating a series of image patches using an AR model, each generated patch is determined by sampling the predicted distribution of the next image token. The randomness inherent in the predicted distribution can lead to variations in the quality of the generated image patches. However, due to the limitation of unidirectional inference in AR models, when poor-quality image patches exist in the generated image, current methods require regenerating the entire image to correct these localized defects, resulting in high computational costs. Summary of the Invention

[0005] This specification provides a model-based image generation scheme that can achieve local image block regeneration through precise positioning of image block location information.

[0006] In a first aspect, embodiments of this specification provide a method for generating images based on a model, comprising: generating a first image from an autoregressive model based on prompt information, the prompt information being used to indicate the generation of visual content, the first image comprising a plurality of sequentially generated first image blocks; determining first position information of a first low-quality image block among the plurality of first image blocks, the first low-quality image block being a first image block whose quality score does not meet a preset requirement; inputting the first position information into the autoregressive model, and having the autoregressive model regenerate a second image block corresponding to the first position information based at least on the plurality of target image blocks or the key vectors and value vectors corresponding to the plurality of target image blocks, the plurality of target image blocks comprising: a plurality of first image blocks generated before the generation of the first low-quality image block, and a plurality of first image blocks generated after the generation of the first low-quality image block.

[0007] In some embodiments, the step of regenerating the second image patch corresponding to the first location information by the autoregressive model based at least on the plurality of target image patches or the key vectors and value vectors corresponding to the plurality of target image patches includes: regenerating the second image patch corresponding to the first location information by the autoregressive model based on the plurality of target image patches and the prompt information, or the key vectors and value vectors corresponding to the plurality of target image patches and the prompt information.

[0008] In some embodiments, the first image includes multiple image patch groups, and the autoregressive model generates the image patch groups in parallel during the generation of the first image. The multiple first image patches constitute the first image patch group, and the multiple target image patches also include several image patches from other image patch groups.

[0009] In some embodiments, generating the first image from an autoregressive model based on prompt information includes: generating the first image and quality scores for each of the first image patches in the first image from the autoregressive model based on prompt information.

[0010] In some embodiments, all image blocks of the first image patch are divided into multiple sets, each set including at least one of the first image patches; generating the quality score of the first image and each of the first image patches in the first image using an autoregressive model based on the prompt information includes: generating the first image patch in the first set using an autoregressive model based on the prompt information and the position information of the first image patch in the first set; generating the quality score of the first image patch in the first set and the first image patch in the second set using an autoregressive model based on the position information of the first image patch in the first set and the first image patch in the second set; iteratively generating the first image patch and the quality score of the first image patch in other sets.

[0011] In some embodiments, the autoregressive model is trained by: acquiring a sample image, the sample image comprising multiple sample image patches, each sample image patch having a quality label and an image label; generating a predicted image and a quality score for each sample image patch by the autoregressive model based on sample cue information, the multiple sample image patches in the sample image, and the position information of the multiple sample image patches; and adjusting the network parameters of the autoregressive model based on the difference between the predicted image patch and the image label of the corresponding sample image patch, and the difference between the quality score of the sample image patch and the quality label of the corresponding sample image patch.

[0012] In some embodiments, the quality label is used to mark the sample image as a positive sample image patch or a negative sample image patch; the step of generating a predicted image and the quality score of each sample image patch by the autoregressive model based on sample cue information, multiple sample image patches in the sample image, and the position information of the multiple sample image patches includes: constructing a first sequence based on sample cue information, multiple sample image patches in the sample image, and the position information of the multiple sample image patches, the first sequence including the sample cue information and multiple sample sets, the sample sets including at least one sample image patch and / or the position information of at least one sample image patch; constructing a second sequence based on the negative sample image patch and the position information of the negative sample image patch; and generating a first predicted image and the quality score of each sample image patch corresponding to the first sequence, and a second predicted image patch and the quality score of each sample image patch corresponding to the second sequence, by the autoregressive model based on the first sequence and the second sequence.

[0013] In some embodiments, the negative sample image patch is obtained by at least one of the following methods: generating a negative sample image patch through random noise; swapping the positions of image patches in the original image corresponding to the sample image to obtain a negative sample image patch; or obtaining image patches that the autoregressive model predicts incorrectly as negative sample image patches through teacher-forced training.

[0014] In some embodiments, the autoregressive model includes multiple attention layers connected in series, each attention layer including an attention module, and key vectors and value vectors corresponding to the multiple target image patches: key matrices and value matrices corresponding to each attention module, the key matrix including multiple key vectors corresponding to the multiple target image patches respectively, and the value matrix including multiple value vectors corresponding to the multiple target image patches respectively. The step of regenerating the second image patch corresponding to the first location information by the autoregressive model based at least on the key vectors and value vectors corresponding to the multiple target image patches includes the first attention module in the first attention layer of the multiple attention layers performing the following steps: calculating a query vector corresponding to the first location information based on an input feature vector corresponding to the first location information; calculating a first attention vector corresponding to the first location information based on the query vector, the key matrix, and the value matrix corresponding to the first attention module; and generating an output feature vector corresponding to the first location information based on the first attention vector, which serves as the input feature vector corresponding to the first location information for the attention module in the next attention layer.

[0015] In some embodiments, each attention layer includes multiple attention modules connected in parallel. Generating an output feature vector corresponding to the first location information based on the first attention vector includes: summarizing the first attention vectors of each attention module in the first attention layer to obtain a second attention vector; and generating the output feature vector based on the second attention vector. In some embodiments, after regenerating the second image patch corresponding to the first location information, the method further includes: performing a quality assessment on the second image patch to obtain a quality score for the second image patch; comparing the quality scores of the second image patch corresponding to the first location information with those of the first low-quality image patch; and, in response to the second image patch having a higher quality score than the first low-quality image patch, replacing the first low-quality image patch in the first image with the second image patch to obtain a second image.

[0016] In some embodiments, the step of evaluating the quality of the second image patch to obtain a quality score for the second image patch includes: inputting at least one of the second image patches into the autoregressive model to obtain a quality score for the second image patch.

[0017] In some embodiments, the method further includes: in response to the number of times the image patch is regenerated not reaching a preset number of correction iterations, or the image quality of the second image not meeting a preset quality requirement, determining second location information of the second low-quality image patch in the second image; and regenerating a third image patch corresponding to the second location information by the autoregressive model based on the second location information.

[0018] In some embodiments, determining the first location information of the first low-quality image block among the plurality of first image blocks includes: determining a preset number of first image blocks with the lowest quality score as the first low-quality image block based on the sorting of the quality scores of the first image blocks; and determining the first location information according to the position of the first low-quality image block in the first image.

[0019] In some embodiments, the position information of the first image block is obtained through two-dimensional rotational position encoding.

[0020] Secondly, embodiments of this specification provide an image generation apparatus based on a model, comprising: an initial generation module, configured to generate a first image from an autoregressive model based on prompt information, the prompt information being used to indicate the generation of visual content, the first image comprising a plurality of sequentially generated first image blocks; a quality assessment module, configured to determine first position information of a first low-quality image block among the plurality of first image blocks, the first low-quality image block being a first image block whose quality score does not meet a preset requirement; and a regeneration module, configured to input the first position information into the autoregressive model, and regenerate a second image block corresponding to the first position information by the autoregressive model based at least on the plurality of target image blocks or the key vectors and value vectors corresponding to the plurality of target image blocks, the plurality of target image blocks including: a plurality of first image blocks generated before the generation of the first low-quality image block, and a plurality of first image blocks generated after the generation of the first low-quality image block.

[0021] Thirdly, embodiments of this specification provide a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the implementations in the first aspect.

[0022] In the solution provided by the above embodiments of this specification, after the autoregressive model generates the initial first image, the first location information of the first low-quality image patch with poor quality in the first image is determined. Then, based on the first location information, the autoregressive model regenerates the image patch at that location, thereby enabling targeted calibration of the locally poor-quality image patch. In this way, local image patch regeneration is achieved through precise positioning of the image patch location information, avoiding full image recalculation, and reducing computational costs while ensuring the generation of high-quality visual content. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of a scene where images are generated based on a model in the embodiments of this specification;

[0025] Figure 2 This is a flowchart of a model-based image generation method in the embodiments of this specification;

[0026] Figure 3 This is a schematic diagram of the structure of an autoregressive model in one of the embodiments of this specification;

[0027] Figure 4 This is a schematic diagram of the calculation process of the attention mechanism in the embodiments of this specification;

[0028] Figure 5 This is a schematic diagram of the input and output sequences of the model in the embodiments of this specification;

[0029] Figure 6 This is a flowchart illustrating the iterative process of generating images based on a model in the embodiments of this specification;

[0030] Figure 7 This is a schematic diagram of the input sequence and label sequence in the model training process of the embodiments of this specification;

[0031] Figure 8 This is a schematic diagram of the structure of the device for generating images based on a model in the embodiments of this specification. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0033] As mentioned earlier, visual content generation technology refers to the automatic generation of high-fidelity visual content such as images or videos that conform to the description, based on given unimodal or multimodal data such as text, images, and videos. In the field of visual generation, autoregressive models provide a powerful paradigm for visual content generation that offers high quality, high controllability, and consistency with language models due to their excellent performance in multimodal instruction following and expansion capabilities. However, since current autoregressive models generate image patches one by one in raster order, the generation of each image patch depends on the previously output sequence. This limitation of unidirectional reasoning means that current methods need to regenerate the entire image to correct poorly performing image patches, resulting in high computational costs.

[0034] To better explain the solutions of the embodiments in this specification, the existing visual content generation solutions based on autoregressive models will be described first.

[0035] The cues are used to instruct the model to generate visual content. For example, instructing the model to generate an image of a flower, or to replace a cat in an image with a dog. When using an autoregressive model to generate visual content, given cues such as text, images, or videos, these cues need to be converted into an input C that the autoregressive model can understand. C contains several tokens. For text-based cues, a token generally refers to the smallest unit or basic element used to represent processed text in natural language processing. A text segmenter can be used to convert text into tokens. For image or video-based cues, a token can be understood as follows: after segmenting the input image (or video frame) into fixed-size image blocks, each image block is mapped to a vector representation. These vector representations are tokens, and each token represents a local image block in the image. A visual tokener can be used to convert images or videos into image tokens.

[0036] The autoregressive model generates the probability distribution of the next image token based on the already generated sequence of image patches, and determines the next image patch output by the model based on sampling of the probability distribution. The autoregressive model has a one-way dependency, that is, each time it generates the probability distribution of the next token, it only considers the tokens that have already been generated. For example, when generating the nth token, it needs to use the previous n-1 tokens as conditions.

[0037] Taking image generation as a simple example, given the prompt information, only decoder-only AR models will sequentially predict discrete tokens x = {x}. n} n＝1:N , where each tokenx n All are generated based on previously generated tokens. The training objective of the autoregressive model is to maximize the probability of the model generating the ground truth sequence, which can be expressed by the following formula:

[0038]

[0039] Where, p θ This represents a probability distribution predictor defined by parameter θ. As can be seen from the formula, each generated token x... n The generation of the probability distribution depends on x n All previous TokenX <nCurrently, only the decoder autoregressive model unidirectionally generates a token sequence and then samples based on a probability distribution to determine image patches, continuing until each image patch in the generated sequence is obtained, resulting in a complete generated image. This limitation of unidirectional reasoning means that current methods need to regenerate the entire image to correct poorly performing local image patches, resulting in high computational costs.

[0040] Human artists typically begin their creative process by sketching a rough draft on a blank canvas with bold brushstrokes. They then refine and correct these rough sketches as they progress. Once the sketch is finalized, the artist can more clearly identify and address any issues, ultimately creating a beautiful artwork.

[0041] Inspired by this, the embodiments of this specification propose a method for generating images based on a model. Figure 1 This diagram illustrates a process for generating images based on a model according to an embodiment of this specification. Figure 1 As shown, firstly, an initial first image is generated by an autoregressive model based on cue information. This can be viewed as a sketch created by a human artist. The cue information is used to indicate the generation of visual content. The first image includes multiple sequentially generated first image blocks; for example, the first image may include nine image blocks. Then, the first position information of the first low-quality image block among the multiple first image blocks is determined. The first low-quality image block is the first image block whose quality score does not meet a preset requirement; for example,... Figure 1 The first low-quality image patches are those labeled 2 and 8. Next, the first location information is input into an autoregressive model, which regenerates the second image patch corresponding to the first location information based on at least a plurality of target image patches or their corresponding key and value vectors. The plurality of target image patches includes several first image patches generated before the generation of the first low-quality image patch, and several first image patches generated after the generation of the first low-quality image patch. In some embodiments, the quality of the new image can then be re-evaluated; if the expected quality is not achieved, the regeneration step is returned, similar to a human artist refining and correcting flaws in a rough sketch. This method is an improvement on current autoregressive model-based generation schemes, enabling the regeneration of locally poor-quality image patches through precise local location information, avoiding the need to regenerate the entire image and reducing computational costs.

[0042] The method for generating images based on models, as proposed in this specification, will be described in detail below.

[0043] Figure 2 This is a flowchart of a model-based image generation method according to an embodiment of this specification. The method can be executed by any device, platform, or cluster of devices with computing and processing capabilities, including steps 201-203 as shown below.

[0044] like Figure 2 As shown, in step 201, the first image is generated by the autoregressive model based on the prompt information.

[0045] The prompt information is used to instruct the generation of visual content. For example, it can be one or more of the text, image, or video information provided by the user. Depending on the visual content generation task, the generated image in this embodiment can be a single image or an image frame from a video. This embodiment does not limit the specific form and content of the user instruction information. For example, when the visual content generation task is text-to-image, the given text information could be: "A red apple is covered by water droplets on a black background"; when the visual content generation task is text-to-video, the given text information could be: "The cathedral is illuminated when fireworks explode in the night sky"; when the visual content generation task is image-to-video, the given image information could be a picture containing two huskies.

[0046] The first image comprises multiple first image blocks generated sequentially. These first image blocks can be all the image blocks in the first image, or they can be multiple image blocks within a small local region of the first image. For example, the first image can be divided into multiple image blocks according to a raster order from top to bottom and from left to right. These multiple first image blocks can be all the image blocks obtained from the above division, or they can be partial image blocks corresponding to a certain local region in the first image after the above division. For ease of description, image blocks will be represented by image tokens below.

[0047] In this step, the prompt information is first input into the autoregressive model, which generates an image token sequence to obtain the first image as the initial image. The first image can be represented as x. (0) The superscript number 0 indicates the initial iteration number. The first image block consists of N first image blocks, each of which can be represented as...

[0048] It should be noted that the autoregressive model used in this embodiment can be an autoregressive model with any network structure. Specifically, it can be an autoregressive model based on the Transformer decoder, such as an autoregressive model with only a decoder or an autoregressive model with an encoder-decoder architecture.

[0049] For example, such as Figure 3As shown, an autoregressive model may include an embedding layer, M sequentially connected Transformer decoders (hereinafter referred to as Transform modules, also known as attention layers), a linear layer, and activation layers (such as Softmax or Sigmoid). Each Transformer module may sequentially include a multi-head attention module, a residual connection and normalization module (Add & Norm), a feedforward neural network, and another residual connection and normalization module. It can be understood that the Transformer module is not limited to including... Figure 1 The structure shown is not limited to the one described above; it can include many other existing structures.

[0050] like Figure 3 As shown, before inputting the prompt information into the autoregressive model, a token segmenter can be used to convert the prompt information into a token sequence containing multiple tokens. For example, for the text included in the prompt information, the text can be segmented based on a pre-set dictionary in the token segmenter, which includes the correspondence between word segments and tokens. Then, each word segment can be converted into its corresponding token based on this correspondence in the dictionary.

[0051] Next, the above token sequence can be input into the embedding layer, which maps each token to a high-dimensional embedding vector, thus obtaining an embedding vector sequence containing multiple embedding vectors. This embedding vector sequence can then be input... Figure 3 The system consists of multiple stacked Transformer modules. Within each Transformer module, the data is processed through multi-head attention modules, feedforward neural networks, residual connections, and layer normalization. Finally, after passing through linear layers and activation layers, the generated first image is output.

[0052] It should be noted that, considering the attention mechanism itself is not sensitive to position (i.e., it cannot directly capture the sequential information of the input sequence), after mapping the cue information (such as text or image) into an embedding vector sequence through the embedding layer, positional encoding is usually added to each embedding vector. Then, the embedding vector sequence including the positional encoding is input into multiple stacked Transformer modules. The purpose of this positional encoding is to mark the order of words in the text when the cue information is text, and to mark the spatial order of image blocks when the cue information is an image. This embodiment does not limit the encoding method used in this process; for example, sine / cosine positional encoding, learnable positional encoding, or two-dimensional positional encoding can be used.

[0053] In step 202, the first position information of the first low-quality image block among the plurality of first image blocks is determined.

[0054] The first low-quality image block is an image block whose quality score does not meet the preset requirements, that is, an image block in the first image with poor visual content quality. There may be one or more first low-quality image blocks. It can be used This represents the quality score of the image patch. The higher the quality score, the higher the visual content quality of the image patch. For example, the quality score can be a number in the range of 0 to 1.

[0055] This embodiment does not impose restrictions on preset requirements. For example, the preset requirement may be that the quality score reaches a quality score threshold. If the quality score of a certain first image block is lower than the quality score threshold, then the first image block is determined to be a first low-quality image block whose quality score does not meet the preset requirement. Another example is that the preset requirement may be that the quality score of the image block ranks among the top in the first image blocks of the first image. Based on the ranking of the quality scores of the first image blocks, a preset number of first image blocks with the lowest quality scores can be determined as the first low-quality image blocks. For example, the c first image blocks with the lowest scores can be determined as the first low-quality image blocks whose quality scores do not meet the preset requirement.

[0056] It should be noted that this embodiment does not limit the method of obtaining the quality score of the image patch. In some embodiments, the quality of the image patch can be evaluated by an external discriminator other than the autoregressive model or by other models. In other embodiments, in order to reduce the computational complexity of the model and reduce computational costs, the quality of the image patch can also be evaluated by the autoregressive model itself to obtain the quality score of each image patch. This process will be described in detail later and will not be discussed here.

[0057] In this step, after determining the first low-quality image block based on the quality score of each first image block in the first image, the first position information can be determined based on the position of the first low-quality image block in the first image. Each first image block has position information, which indicates the position of the first image block in the first image. This position information can be used as a position indicator token to input into the autoregressive model to indicate the generation of the image block at the corresponding position.

[0058] This embodiment does not limit the representation of positional information. For example, the positional code can be obtained by encoding the number of rows h and columns w of the image patch in the image, and this positional code can be used as the positional information. This embodiment does not limit the encoding method used for positional encoding. A unique positional code can be generated for each position in the sequence using sine and cosine functions of different frequencies, or 2D-RoPE (2D Rotary Positional Encoding) can be used for encoding; for example, learnable positional parameters can be set for each image patch, and these positional parameters can be used as positional information. In one implementation, the positional encoding method used for the first positional information can be the same as the positional encoding method added to the embedding vector obtained by encoding the prompt information in step 201 before entering the multi-head attention module.

[0059] For example, taking encoding using 2D-RoPE as an example, for the nth image patch x in any image n The location information corresponding to this image patch can be represented as a location indicator token p. n As shown in the formula below:

[0060] p n =RoPE(e,h,w) (2)

[0061] Here, e is a shared learnable embedding that can be rotated according to the 2D-RoPE method based on the two-dimensional coordinates (h,w).

[0062] In step 203, the first location information is input into the autoregressive model, and the autoregressive model regenerates the second image block corresponding to the first location information based on at least multiple target image blocks or the key vectors and value vectors corresponding to multiple target image blocks.

[0063] The multiple target image blocks include: several first image blocks generated before the generation of the first low-quality image block, and several first image blocks generated after the generation of the first low-quality image block. This embodiment does not limit the specific image blocks included in the multiple target image blocks. For example, it can refer to all image blocks in the already generated first image, or it can refer to the information of the remaining image blocks in the first image other than the first low-quality image block, or it can refer to a predetermined number of first image blocks in the first image that are spatially adjacent to the first low-quality image block. The multiple target image blocks provide rich contextual information, which can make the transition between the newly generated second image block and its adjacent initially generated first image blocks in the first image smoother, thereby enhancing the overall consistency and coordination of the image.

[0064] This step can be implemented in two ways, which will be explained below:

[0065] Implementation Method 1: Input the first location information into the autoregressive model, and the autoregressive model regenerates the second image block corresponding to the first location information based on at least multiple target image blocks.

[0066] In this implementation, multiple target image patches and initial location information can be simultaneously input into the autoregressive model. It is understood that when multiple target image patches are input into the model, each target image patch can carry its location information within the first image, thus providing the model with richer contextual information to assist in the regeneration of the second image patch. The calculation method for this location information can be found in the preceding description and will not be repeated here. Figure 3 As shown, before inputting multiple target image blocks and first location information into the autoregressive model, the multiple target image blocks and first location information can be converted into token sequences containing multiple tokens by a token segmenter.

[0067] Next, the above token sequence can be input into the embedding layer, which maps each token to a high-dimensional embedding vector, thus obtaining an embedding vector sequence containing multiple embedding vectors. This embedding vector sequence can then be input... Figure 3 The multiple stacked Transformer modules in the model are processed through multi-head attention modules, feedforward neural networks, residual connections and layer normalization layers within each Transformer module. Finally, after passing through linear layers and activation layers, the generated second image block corresponding to the first position information is output.

[0068] In other embodiments, to further improve the quality of the regenerated second image patch, the second image patch corresponding to the first position information can be regenerated by an autoregressive model based on multiple target image patches and prompt information. In this embodiment, the prompt information, multiple target image patches, and the first position information can be simultaneously input into the autoregressive model. Similar to the above embodiments, before inputting the prompt information, multiple target image patches, and the first position information into the autoregressive model, a token segmenter can be used to convert the prompt information, multiple target image patches, and the first position information into token sequences including multiple tokens, and then input them. Figure 3 The autoregressive model in the image is used to process the data and generate the second image block corresponding to the first location information.

[0069] Method 2: Input the first location information into the autoregressive model, and the autoregressive model regenerates the second image block corresponding to the first location information based on the key vectors and value vectors corresponding to at least multiple target image blocks.

[0070] Due to the caching mechanisms of autoregressive models, such as KV-cache (Key-Value cache), the model stores prompt information and key and value vectors corresponding to each image patch in the first image during the generation process. The autoregressive model can utilize this information as contextual information to regenerate the second image patch corresponding to the first positional information. This allows the generation process of the second image patch to fully utilize the contextual information provided by the first image, resulting in higher image quality for the regenerated second image patch. Furthermore, this regeneration process actually employs a global attention mechanism, which, in the above manner, makes the connection between the newly generated second image patch and its adjacent initially generated first image patches in the first image smoother, thereby enhancing the overall consistency and harmony of the image.

[0071] Taking the KV-Cache mechanism as an example, in step 201 above, when the model generates the image patch sequence, due to the existence of the self-attention mechanism, in order to generate the nth image patch, the model needs to calculate its attention to the cue information and all the previous n-1 image patches. This attention calculation depends on the cue information and the key and value vectors corresponding to the previous n-1 patches. The KV-Cache mechanism stores the calculated key and value vectors in the cache each time, so that the model does not need to recalculate the key and value vectors corresponding to the previous cue information and image patches every time it generates the next image patch. After generating the first image, the KV-Cache stores the intermediate key-value vectors required for calculating the attention corresponding to the generated first image patch sequence in the cache. These key-value vectors represent the context information of the generated first image.

[0072] In some embodiments, the autoregressive model may include multiple attention layers in series, such as Figure 3The system consists of M cascaded Transform modules. Each attention layer includes an attention module, also known as an attention head. For example, a multi-head attention module within a Transform module may contain multiple attention heads. The key vectors and value vectors corresponding to multiple target image patches can specifically refer to the key matrix and value matrix corresponding to each attention module. The key matrix includes multiple key vectors corresponding to each target image patch, and the value matrix includes multiple value vectors corresponding to each target image patch. When the autoregressive model regenerates the second image patch corresponding to the first location information based on at least the key vectors and value vectors corresponding to multiple target image patches, the first attention module in the first attention layer of the multiple attention layers can perform the following steps: First, based on the input feature vector corresponding to the first location information, calculate the query vector corresponding to the first location information. Then, based on the query vector, the key matrix and value matrix corresponding to the first attention module, calculate the first attention vector corresponding to the first location information. Finally, based on the first attention vector, generate the output feature vector corresponding to the first location information, which serves as the input feature vector corresponding to the first location information for the attention module in the next attention layer.

[0073] The following example will be used to explain in detail the process of generating the second image patch in this implementation.

[0074] Assuming the first image contains 10,000 image blocks, and following the generation order from top to bottom and left to right, the first position information p 5000 The corresponding first low-quality image patch is the 5000th image patch. After generating the first image, the autoregressive model's cache can store the cue information from each attention module and the key vector K of the multiple generated target image patches (e.g., all 10000 image patches). s Sum vector V s Information such as key vectors and value vectors of multiple target image patches will be used to regenerate an image. It should be understood that in other examples, the attention calculation can also be performed using both cue information and the key vectors and value vectors of the target image patches. The process can also refer to the following procedure.

[0075] Taking the multi-head attention mechanism used in the autoregressive model as an example, multiple attention heads (i.e., multiple attention modules) are used in parallel to perform attention calculations within the multi-head attention module. The first position information p... 5000 After inputting the autoregressive model, the first position information p 5000 Transformed into an input feature vector e at the embedding layer 5000 Then enter Figure 3The first Transform module in the algorithm processes the data through a multi-head attention module. This multi-head attention module includes multiple first attention modules, each containing a Q-parameter matrix W. Q K-parameter matrix W K and V parameter matrix W V Where Q represents query, K represents key, and V represents value. Attention vectors can be calculated within each first attention module. This embodiment does not limit the number of first attention modules in a multi-head attention module, such as... Figure 4 The diagram illustrates a multi-head attention module comprising three first attention modules. It should be noted that each first attention module stores a key matrix, a value matrix, and a Q-parameter matrix W. Q K-parameter matrix W K and V parameter matrix W V They are all different. For example, in the first attention module 1, the input feature vector e can be... 5000 A query vector Q corresponding to the first location information is generated through a linear mapping layer. 5000 :

[0076] Q 5000 =W Q e 5000 (3)

[0077] In one embodiment, when multiple target image patches refer to information about all image patches of the generated first image (i.e., the 10,000 generated image patches), the query vector Q is continued to be used in the first attention module. 5000 The attention weight matrix is ​​calculated by performing a dot product and normalization on the hint information in the cache and the key vector corresponding to each target image patch (the first to 10000th first image patches). Then, the attention weight matrix corresponding to each first image patch is used to weight all the corresponding value vectors in the cache (i.e., the value vectors corresponding to the first to 10000th first image patches) to obtain the first attention vector Z output by the first attention module. 5000 :

[0078]

[0079] in, is a scaling factor, d is the dimension of the key vector, and its purpose is to prevent the dot product value from becoming too large, which could lead to gradient vanishing or exploding. The Softmax function is used for normalization.

[0080] In another implementation, when multiple target image blocks refer to image blocks other than the first low-quality image block (i.e., the already generated first image blocks 1-4999 and 5001-10000) and the cache also stores the key vector and value vector of instruction information, there are two processing methods:

[0081] One approach is to use the query vector Q in the first attention module. 5000 The attention weight matrix is ​​calculated by performing a dot product and normalization on the key vectors corresponding to the hints in the cache and the target image patch, respectively. Then, the attention weight matrix corresponding to each target image patch is used to weight the corresponding value vector V in all caches. s (That is, the value vectors corresponding to the prompt information and the generated first image blocks 1-4999 and 5001-10000 respectively), to obtain the first attention vector Z output by the first attention module. 5000 In this way, the information of the first low-quality image block can be avoided during the regeneration of the second image block, thus preventing the low-quality image block from affecting the regenerated image block.

[0082] Another approach is to generate the query vector Q. 5000 At the same time, the input feature vector e 5000 Each layer of linear mapping generates a newly generated key vector K′ corresponding to the first position information. 5000 Sum vector V′ 5000 :

[0083] K′ 5000 =W K e 5000 (5)

[0084] V′ 5000 =W V e 5000 (6)

[0085] Newly generated key vector K′ 5000 Sum vector V′ 5000 The key and value vectors corresponding to the original 5000th first image patch stored in the cache can be replaced to obtain a new KV cache. Then, the first attention vector Z mentioned above is performed based on the new KV cache. 5000 The calculation is as follows. In this way, not only can the information of the first low-quality image block not be used in the process of regenerating the second image block, but also the information of the regenerated second image block (i.e. the regenerated 5000th image block) can be used in subsequent rounds of regenerating other image blocks (the case of regenerating image blocks in multiple rounds of iteration will be explained below).

[0086] In another implementation, when multiple target image blocks refer to information about first image blocks in the first image that are spatially adjacent to the first low-quality image block, the key vector and value vector used in the calculation process of the above formula (4) can be the key vector and value vector of the first image block that are spatially adjacent to the first low-quality image block. This embodiment does not limit the definition of spatial proximity. For example, multiple image blocks within a preset range (e.g., a 4×4 or 3×3 range, or four image blocks before and after) around the first low-quality image block can be determined as spatially adjacent target image blocks. For example, taking a 3×3 range as an example, for... Figure 1 For the first low-quality image block 2 in the image, the image blocks that are spatially adjacent to it are image blocks numbered 1, 3, 4, 5 and 6.

[0087] Understandably, in some embodiments, when using a single-head attention mechanism, each attention layer includes an attention module, which can be based on the first attention vector Z output by the attention module of the first attention layer (e.g., the first Transform module). 5000 The calculations of other layers in the Transformer module are then performed, such as residual connection and normalization, to obtain the output feature vector corresponding to the first position information output by the Transformer module. Using the input feature vector corresponding to the first position information as the next layer attention module (i.e., the multi-head attention module in the second Transform module), perform a new round of calculations with the same process as described above.

[0088] In some other embodiments, when using a multi-head attention mechanism, each attention layer includes multiple attention modules connected in parallel, such as multiple attention heads connected in parallel within a multi-head attention module. After each attention head in the multi-head attention module of the first Transform module outputs its corresponding first attention vector, the first attention vectors of each attention module in the first attention layer can be aggregated to obtain a second attention vector, and then an output feature vector is generated based on the second attention vector.

[0089] Specifically, the first attention vectors output by multiple first attention modules can be concatenated and linearly transformed to obtain the second attention vector. For example... Figure 4 As shown, the outputs of the first attention module 1, the first attention module 2, and the first attention module 3 are respectively... and The concatenation is performed, and a linear transformation is applied using a mapping matrix to obtain the second attention vector output by the multi-head attention model. After obtaining the second attention vector, calculations can be continued in other layers of the Transformer module based on it, such as residual connections, normalization layers, and feedforward neural network layers, to obtain the output feature vector of the Transformer module.

[0090] Then, based on the same process, this output feature vector can be used as the input feature vector for the second transform module to perform a new round of identical calculations. Specifically, first, based on the Q-parameter matrix W in the multi-head attention module of the second transform module... Q′ K-parameter matrix W K′ and V parameter matrix W V′ Execute formulas (3) to (5) to obtain the query vector, key vector, and value vector corresponding to the first position information in the second Transformer module. Then, continue to execute formula (6) to obtain the attention vector. Finally, perform subsequent calculations to obtain the output feature vector of the second transform module. This process is repeated in the other transform modules until the output feature vector of the Mth Transformer module is obtained. This can be viewed as the feature vector corresponding to the regenerated second image patch (the image patch at position 5000). Finally, based on the output feature vector of this Mth Transformer module... Continue processing with linear and activation layers to obtain the second image patch corresponding to the regenerated first location information.

[0091] In this process, the global attention mechanism enables the model to simultaneously consider the information of global image patches (several first image patches generated before the first low-quality image patch is generated, and several first image patches generated after the first low-quality image patch is generated). The query vector corresponding to the 5000th image patch will not only calculate attention with the key vector and value vector of the preceding image patches, but also with the key vector and value vector of the following image patches.

[0092] In this way, the image patch at a specified location can be regenerated. The regenerated second image patch can replace the first low-quality image patch in the original first image to obtain a second image with better image quality, thus avoiding the need for full image recalculation.

[0093] In some embodiments, in order to accelerate the generation of the first image and reduce computational costs in the above process, the image blocks in the first image can be grouped to achieve parallel generation of multiple first image blocks.

[0094] The following explains the grouping of image patches generated in parallel.

[0095] In this embodiment, all image blocks in the first image are divided into multiple image block groups. During the process of generating the first image, the autoregressive model generates multiple image blocks in each image block group in parallel. The multiple first image blocks can constitute a first image block group. At this time, the multiple target image blocks also include several image blocks in other image block groups.

[0096] It is understood that each group of image patches (hereinafter referred to as the set) includes at least one image patch. For example, image x = {x...} n} n＝1:N The image patches can be divided into K sets: the first set contains {x1}, the second set contains {x2, x3}, the third set contains {x4, x5}, and the last set (k-th set) contains {x... N-2 ,x N-1 ,x N This embodiment does not limit the way the set is divided. Considering the locality of visual content, the correlation between adjacent regions, and the sharing of common visual cues, this embodiment can group spatially adjacent image patches as a set to utilize this spatial dependency and generate images better. In other embodiments, other grouping methods can also be used. For example, since image patches in highly similar regions (such as large areas of blue sky or regions with similar textures) rely on essentially the same context during generation, multiple image patches in highly similar regions can be generated in parallel as a set. Or, since image patches in semantically unrelated regions (such as regions located in the upper right and lower right corners of an image, respectively) do not depend on each other's information as context, multiple image patches in semantically unrelated regions can be generated as a set.

[0097] The first image patch group can refer to any one of multiple image patch groups. Assuming the first image patch group refers to the aforementioned second set, after the first image is generated, the information stored by the autoregressive model includes not only information about several other first image patches within the second set containing the first low-quality image patch (e.g., when the first low-quality image patch is x2, the information about several other first image patches within the set containing x2 refers to the information about x3 within the second set containing x2), but also information about several image patches in other image patch groups, such as the first set, the third set, and other sets. During the regeneration of the second image patch, the autoregressive model can utilize this information as contextual information.

[0098] In some embodiments, considering that the image quality of the second image block of the regenerated image block may not be higher than that of the first image block, directly replacing the first image block with the second image block may reduce the image quality of the initial first image. After step 203, the following steps may be performed: First, the quality of the second image block is evaluated to obtain a quality score for the second image block; then, the quality scores of the second image block corresponding to the first location information and the first low-quality image block are compared; then, in response to the quality score of the second image block being higher than that of the first low-quality image block, the first low-quality image block in the first image is replaced with the second image block to obtain the second image.

[0099] In practice, the second image block can be evaluated using the same quality assessment method as the first image block to obtain its quality score. For example, if the autoregressive model has a quality assessment function, at least one second image block can be input into the autoregressive model to obtain its quality score. When the quality score of the second image block is higher than that of the first image block, it indicates that the quality of the regenerated second image block is better than that of the original first image block. In this case, the first low-quality image block in the first image can be replaced with the second image block to correct the first image and obtain a second image of better quality. Otherwise, if the quality score of the second image block is not higher than that of the first image block, no replacement is performed.

[0100] In some embodiments, the first image can be iteratively optimized to continuously improve the image quality. Specifically, in response to the fact that the number of times the image patch is regenerated does not reach a preset number of correction iterations, or that the image quality of the second image does not meet the preset quality requirements, the second position information of the second low-quality image patch in the second image is determined, and the third image patch corresponding to the second position information is regenerated by the autoregressive model based on the second position information.

[0101] In one example, we can assume the preset number of correction iterations is three. Generating the second image patch is the first iteration. Since the preset number of correction iterations has not been reached, we continue to determine the second location information of the second low-quality image patches that do not meet the preset score requirements based on the quality scores of each image patch in the second image. For example, we identify the c image patches with the lowest quality scores in the second image as the second low-quality image patches. Then, we input the second location information of the second low-quality image patches into the autoregressive model to obtain the third image patch. We can compare the quality scores of the second low-quality image patches and the third image patch corresponding to the second location information. In response to the third image patch having a higher quality score than the second low-quality image patch, we replace the second low-quality image patch in the second image with the third image patch to obtain the third image. This is the second iteration, and the preset number of correction iterations has still not been reached. We continue to determine the third location information of the third low-quality image patches that do not meet the preset score requirements based on the quality scores of each image patch in the third image. For example, we identify the c image patches with the lowest quality scores in the third image as the third low-quality image patches. Then, we input the third location information of the third low-quality image patches into the autoregressive model to obtain the fourth image patch. The quality scores of the third low-quality image block and the fourth image block corresponding to the third location information can be compared. In response to the fact that the quality score of the fourth image block is higher than that of the third low-quality image block, the third low-quality image block in the third image is replaced with the fourth image block to obtain the fifth image. This is the third iteration. When the preset number of correction iterations is reached, the iteration stops and the fifth image is used as the target image for this visual generation task.

[0102] In another example, the quality of the resulting image can be evaluated after each iteration. If it meets the preset quality requirements, the iteration stops; otherwise, the next iteration continues. For example, if the second image meets the preset quality requirements, the iteration stops; otherwise, the next iteration is performed to obtain the third image.

[0103] In other examples, the iteration could also stop when the quality of the image no longer improves compared to previous images.

[0104] The autoregressive model proposed in this embodiment has the ability to iteratively modify generated image patches. First, an initial image is generated, and then the worst quality image patch in the initial image is iteratively corrected until a satisfactory image quality is achieved. This avoids redundant calculations and speeds up inference, thus realizing the self-correction of image patches.

[0105] Based on the above embodiments, this specification also proposes a scheme to integrate image generation and quality assessment into the same autoregressive model, so that the autoregressive model has the ability to self-assess and self-calibrate without introducing an additional model. Thus, in step 201 above, the autoregressive model can generate the first image and the quality scores of each first image block in the first image based on the prompt information.

[0106] The following section explains the autoregressive model that has quality assessment capabilities.

[0107] Understandingly, the task of generating the first image using an autoregressive model can be viewed as a classification task for each image token, with the autoregressive model outputting a probability distribution for each token. When using this model to assess the quality of image patches, it can be considered a binary classification task, distinguishing between high-quality and low-quality patches. The autoregressive model outputs the probability value of an image patch belonging to either a high-quality or low-quality image patch. In this specification, this probability value can be used as the quality score of the image patch. For example, the probability value of an image patch belonging to a high-quality image patch can be used as the quality score, which ranges from 0 to 1. The higher the probability value of an image patch belonging to a high-quality image patch, the higher its quality score, and vice versa. Through model training, the model can learn how to assess the quality of input image patches; the training method will be explained later.

[0108] To replace the previous autoregressive model's raster sequence generation method and enable the model to output individual image patches and their quality scores based on different input information, this specification proposes two types of input tokens: a location instruction token (i.e., location information) for generating image patches at corresponding locations, and an image token (i.e., an image patch) for quality evaluation. When the location instruction token is input into the autoregressive model, the model can regenerate the image patch at the specified location based on the location information, without needing to regenerate the image patch sequence. When the image token is input into the autoregressive model, the model can evaluate the quality of the image patch and obtain its quality score.

[0109] In this embodiment, the prompt information can be input into the autoregressive model first, and each first image patch can be iteratively output to obtain the first image. Then, each first image patch can be iteratively input into the autoregressive model to obtain the quality score corresponding to each first image patch. Alternatively, the prompt information can be input into the autoregressive model, which first generates the first first image patch. Then, the first first image patch is input into the model to obtain the quality score of the first image patch. This process is repeated iteratively to generate each first image patch and its corresponding quality score. This embodiment does not restrict the specific generation order.

[0110] In one implementation, to accelerate the image generation and quality assessment process described above, image patches and their corresponding quality scores can be generated in parallel. In this implementation, the quality scores of multiple image patches can be considered as a set. In each iteration, the model output is no longer a single image patch or its instruction score, but rather a set of image patches and / or their quality scores. When regenerating the image patches located at the positions of the c lowest quality-score image patches, these c image patches can also be considered as a set, and the model can simultaneously generate the c image patches from this set based on their positional information. This reduces the number of model iterations, accelerates the inference process, and allows the model to balance generation efficiency and image quality.

[0111] The parallel generation process is explained below.

[0112] This embodiment divides multiple image patches in an image into multiple sets, each set including at least one image patch. For example, image x = {x...} n} n＝1:N It can be divided into K sets {x1}, {x2, x3}, ..., {x N-2 ,x N-1 ,x N This embodiment does not restrict the way the set is divided. Considering the locality of visual content, the correlation between adjacent regions, and the sharing of common visual cues, this embodiment can group spatially adjacent image patches as a set to utilize this spatial dependency and generate images better. Furthermore, considering the varying generation difficulty at different stages, there are few available visual cues when the model first starts generating image patches, and generating too many image patches may lead to semantic conflicts between visual content. Setting a smaller number of image patches in the initial generation set helps reduce generation difficulty and improve the generation quality in the initial AR generation steps.

[0113] In some embodiments, the number of image patches in each set can be positively correlated with the generation order of the sets. That is, the number of image patches in each generated set increases iteratively. This embodiment does not limit the positive correlation; for example, it can be a function curve that conforms to a positive correlation, or it can be randomly set. In one example, the positive correlation can refer to a sine function relationship, where the change in the number of image patches in the set conforms to a sine function curve. Specifically, adjacent image patches are divided into a set, and then the number of image patches in each set increases as the sine curve changes. Gradually increasing the size of the set in a sine curve manner also prevents the generation of too many image patches in a single iteration in later steps.

[0114] The method for dividing the set of quality scores of the image patches to be generated can be the same as or different from the method for dividing the set of image patches; this will not be elaborated here.

[0115] For the generation process, the quality evaluation of the image patch generated in the previous iteration and the generation of the image patch in the next iteration can also be processed in parallel, thus forming... Figure 5 The diagram shows the input sequence used to input the model and the corresponding output sequence of the model during each iteration. For example... Figure 5 As shown, in the first iteration of the model, based on the prompt information C and the position information p1 of the first image patch in the first set, the autoregressive model generates the first image patch x1 in the first set. Then, in the second iteration, based on the position information p2 and p3 of the first image patch in the first set and the first image patch in the second set, the autoregressive model generates the quality score s1 of the first image patch in the first set, as well as the first image patches x2 and x3 in the second set. Next, in the third iteration, based on the position information p4 and p5 of the first image patch in the second set and the first image patch in the third set, the autoregressive model generates the quality scores s2 and s3 of the first image patch in the second set, as well as the first image patches x4 and x5 in the third set… and so on, iteratively generating the first image patches and their quality scores in other sets until the (K+1)th iteration, where the first image patch x1 in the Kth set is generated. N-2 ,x N-1 and x N Input the autoregressive model and obtain the quality score s of the first image patch in the Kth set of the autoregressive model output. N-2 ,s N-1 and S N Thus, the autoregressive model generates the first image and the quality scores of each first image patch within the first image. During this generation process, a bidirectional attention mechanism can be used within a set, and a causal attention mechanism can be used between different sets. Utilizing a key-value (KV) cache can accelerate the computation of causal attention between sets, thereby speeding up the inference process.

[0116] The following is combined with Figure 6 The method for generating images based on models in this embodiment will be described in detail.

[0117] like Figure 6 As shown, the process of outputting the image based on the model mainly consists of two parts: the generation of the initial first image and the correction iteration process. To facilitate a clear and intuitive understanding of the generation process, this scheme divides the image into 9 image blocks. In practical applications, the size of the image blocks is... Figure 6 The one shown is much smaller.

[0118] First, in the first iteration of the initial first image generation process (each iteration is indicated by a dashed box in the figure), the prompt information C (represented by C on a blue background in the figure) and the position information of the first image patch 1 in the first set (represented by a number on a yellow background in the figure) are input into the autoregressive model. The model outputs the first image patch 1 in the first set. The prompt information C is used to instruct the model to generate an image of a cat on the grass.

[0119] Then, in the second iteration of the initial first image generation process, the position information of the first image block 1 in the first set and the first image blocks 2 and 3 in the second set are input into the autoregressive model. The model outputs the quality score of the first image block 1 as 0.7 (the quality score is represented by the gray background in the figure) and the first image blocks 2 and 3 in the second set.

[0120] The first image patch and its location information of the next set are input sequentially until the last iteration. The first image patches 8 and 9 in the last set are input into the autoregressive model. The model outputs a quality score of 0.5 for the first image patch 8 and a quality score of 0.6 for the first image patch 9.

[0121] For example, the preset number of image patches to be corrected in each iteration can be 2. In this step, from the first image x... (0) The quality score s of each first image patch (0) The two first image blocks with the lowest quality scores (first image block 5 and first image block 6) are identified as the first low-quality image blocks.

[0122] Next, during the first correction iteration, the position information of the first image patch 5 and the first image patch 6 are input into the autoregressive model in the form of a set to obtain the regenerated second image patch 5 and the second image patch 6. These regenerated second image patches replace the first low-quality image patch in the first image, resulting in the second image x. (1) And the quality score s of each image patch in the second image (1) The two image blocks with the lowest quality scores (image block 2 and image block 8) were then identified as the second low-quality image blocks.

[0123] Next, in the second correction iteration, the position information of image patch 2 and image patch 8 is input into the autoregressive model in the form of a set to obtain the regenerated image patch 2 and image patch 8. These regenerated image patches replace the second low-quality image patch in the second image to obtain the third image x. (2) And the quality score s of each image patch in the third image (2) Continue to identify the two image blocks with the lowest quality scores (image block 3 and image block 9) as the third low-quality image blocks until the number of iterations reaches the preset correction iteration number, or the image quality of the image meets the preset quality requirements.

[0124] This self-calibration mechanism enables the model to repeatedly evaluate and optimize its output image patches, thereby improving the overall accuracy and coherence of the generated visual content without relying on external models.

[0125] The training process of the autoregressive model used in the above embodiments is described below.

[0126] In one embodiment, the autoregressive model is trained in the following manner:

[0127] First, sample images and sample cue information are obtained. The sample images consist of multiple sample image patches, each with a quality label and an image label. The quality label is used to mark the true quality score of each sample image patch; for example, a high-quality sample image patch is labeled as 1, and a low-quality sample image patch is labeled as 0. Sample images can be obtained by processing the original high-quality image, for example, by reducing the image quality of local image patches in the original image. The image labels are the individual image patches corresponding to the original high-quality image.

[0128] Then, the autoregressive model, based on the sample cue information, multiple sample image patches in the sample image, and the positional information of these patches, generates a predicted image and quality scores for each sample image patch. The predicted image is the initial image generated by the autoregressive model based on the sample cue information. Inputting the positional information of the sample image patches into the autoregressive model yields the predicted image patches at the corresponding positions. Inputting the sample image patches into the autoregressive model yields their quality scores. For a more detailed explanation of the generation process of the predicted image patches and the quality scores of each sample image patch, please refer to the previous section on the first image patch and the quality scores of each first image patch; it will not be repeated here.

[0129] Finally, the network parameters of the autoregressive model are adjusted based on the differences between the image labels of the predicted image patch and the corresponding sample image patch, as well as the differences between the quality scores of the sample image patch and the quality labels of the corresponding sample image patch.

[0130] Specifically, based on the difference between the image labels of the predicted image patch and the corresponding sample image patch, a loss function for the image generation task can be calculated. By minimizing this loss function, the model can learn to predict the possible category distribution of the image patch sequence. Similarly, based on the difference between the quality score of the sample image patch and the quality label of the corresponding sample image patch, a loss function for the quality assessment task can be calculated. By minimizing this loss function, the model can predict whether an image patch belongs to a high-quality or low-quality image patch. This embodiment does not limit the loss function used above; for example, the cross-entropy loss function can be used.

[0131] In this embodiment, the sum of the loss functions for image generation and quality assessment can be defined as the training loss. By adjusting the network parameters to minimize the training loss, the accuracy of both image generation and quality assessment can be improved simultaneously. This process can be achieved using supervised learning. Network training ends when the network iteration termination condition is met. This embodiment does not impose restrictions on the network iteration termination condition. For example, this condition could be reaching a certain number of iterations or the training loss falling below a certain threshold. Through the above iterative training, the model is trained to perform both generation and evaluation operations simultaneously within a single autoregressive model.

[0132] The training method for autoregressive models with parallel generation capabilities will be explained next.

[0133] To enable the model to output a set in each iteration, instead of a single token, image patches can be input into the autoregressive model as a sequence of sets during the training phase, such as... Figure 7 As shown, the autoregressive model is specifically trained in the following way:

[0134] First, obtain the sample images and sample cue information. The sample images consist of multiple sample image patches, each with a quality label and an image label. The quality label is used to mark the sample image as a positive sample image patch x. + or negative sample image patch x - For example, positive sample image patches are labeled as 1, and negative sample image patches are labeled as 0. Sample images can be obtained by processing high-quality original images; for example, reducing the image quality of local image patches in the original image to obtain negative sample image patches, and identifying other high-quality image patches in the original image as positive sample image patches. The number of negative sample image patches can be determined by setting the proportion r of negative sample image patches in the original image. Image labels are the individual image patches corresponding to the original image, such as... Figure 7 The image labels shown.

[0135] Then, based on the sample prompt information, multiple sample image blocks in the sample image, and the position information of the multiple sample image blocks, a first sequence is constructed. The first sequence includes the sample prompt information and multiple sample sets, and the sample sets include at least one sample image block and / or the position information of at least one sample image block.

[0136] The first sequence is used to input the autoregressive model to generate the initial first predicted image and the quality score of each first predicted image patch in the first predicted image. When constructing the first sequence, the position information of the sample image patches can be combined with the sample image patches to obtain a sample set. According to the order of the image patches in the image, multiple sample sets are spliced ​​together to obtain the first sequence. Regarding the partitioning method of the sample set, please refer to the partitioning method of the set in the above embodiment, which will not be repeated here.

[0137] like Figure 7 As shown, Figure 7 The first and second sequences of the input autoregressive model are separated by a dashed line. The first sequence includes, in sequence: prompt information C, the position information p1 of sample image patch 1 in the first sample set, and the sample image patch in the first sample set. And the location information p2 and p3 of sample image blocks 2 and 3 in the second sample set, sample image blocks in the second sample set. and and the sample image patch in the last sample set and

[0138] Next, a second sequence is constructed based on the negative sample image patches and their location information. During training, negative sample image patches requiring further optimization are added to the end of the first sequence to simulate the iterative optimization process. Figure 7 As shown, assuming the preset number of correction iterations is 2, and the preset number of negative sample images, or low-quality image blocks, in each iteration is 2, the second sequence includes: the position information p3 and p9 of negative sample image blocks 3 and 9 in the first sequence (to simulate the process of regenerating high-quality image blocks after iteration), and positive sample image blocks. and (To simulate the process of evaluating newly generated image patches), the position information p2 and p8 of sample image patches 2 and 8 (to simulate the process of regenerating high-quality image patches from low-quality image patches determined in the next iteration), negative sample image patches. and positive sample image patches (The regenerated image patch was also set to a negative sample image patch with probability r to simulate the situation where low-quality image patch output may still occur when regenerating the image patch).

[0139] Then, the input sequence consisting of the first sequence and the second sequence is sequentially input into the autoregressive model according to the order of the sample set. Based on the first sequence and the second sequence, the autoregressive model generates the first predicted image corresponding to the first sequence and the quality scores of each sample image block, as well as the second predicted image block corresponding to the second sequence and the quality scores of each sample image block.

[0140] Finally, the network parameters of the autoregressive model are adjusted based on the difference between the image labels of the first predicted image patch and the corresponding sample image patch, as well as the difference between the quality score of the sample image patch and the quality label of the corresponding sample image patch.

[0141] like Figure 7 The label sequence shown is divided by a dashed line from the label sequence of the output sequence corresponding to the first and second sequences. When the input is the location information of a sample image patch, the label in the label sequence is the image label corresponding to that sample image patch. When the input is a positive sample image patch, the label in the label sequence is the quality label 1 corresponding to the positive sample image patch. When the input is a negative sample image patch, the label in the label sequence is the quality label 0 corresponding to the negative sample image patch. The difference between the model's output sequence and the label sequence can be measured using a loss function. Then, backpropagation is used to adjust the network parameters to minimize the loss function, thus training the model. This training method allows the model to learn to generate and evaluate the quality of multiple image patches in the set in parallel, accelerating the model's inference process.

[0142] In the training process described above, the quality assessment task is set as a binary classification prediction task, namely, distinguishing between high-quality (positive samples) and low-quality (negative samples) image patches. Sample images can be obtained by processing the high-quality original image, for example, by reducing the image quality of local image patches in the original image to obtain negative sample image patches. This embodiment does not restrict the generation method of negative sample image patches. Positive sample image patches in the high-quality original image are defined as positive sample image patches with a quality label of 1, while negative sample image patches are assigned a quality label of 0. The quality score is the probability value of an image patch being predicted as a positive sample image patch.

[0143] In one implementation, negative sample image patches can be obtained through at least one of the following methods:

[0144] 1. Generate negative sample image patches using random noise.

[0145] For each image patch in the original image, any few patches can be replaced with negative sample image patches generated from random noise. For example, random visual content from a discrete codebook of a VAE (Variational Autoencoder) can be used for replacement. Although the replaced content is valid within the codebook, it is inconsistent with the image's context. This simulates errors caused by random sampling from a classification distribution.

[0146] In other examples, negative sample image patches can also be generated by adding random perturbations or offsets to image patches of the original image.

[0147] 2. Swap the positions of the image blocks in the original image corresponding to the sample image to obtain the negative sample image block.

[0148] By repositioning image patches in the original image, the patches are moved to different spatial locations to obtain negative sample image patches. These patches retain valid visual content, but their spatial positions are misaligned. This method of generating negative sample image patches can enhance the model's spatial sensitivity and improve its ability to maintain the continuity of adjacent visual content.

[0149] 3. Image patches that are incorrectly predicted by the autoregressive model are obtained through teacher-mandated training and used as negative sample image patches.

[0150] To train the model to recognize and correct its own errors, a third type of negative sample image is introduced, derived from erroneous predictions generated during teacher-forced training. To ensure the predictions align with real visual content, teacher-forced predictions based on a sequence of labeled data (Ground Truth) are used for training. The specific method is as follows: Before formal training, forward propagation, backpropagation, and parameter optimization are performed according to standard AR training steps. This process operates based on labeled data; that is, at each prediction step, the model's previously generated content is not used, but rather labeled data is forced as the input for the current step. During this process, erroneously predicted image patches are collected as negative sample image patches. It is important to note that errors in teacher-forced predictions differ from errors in actual inference. In inference, errors originate from both model predictions and random sampling, leading to error accumulation. However, prediction errors in teacher-forced training stem from the model's inherent inaccuracy; since each step is based on accurate labeled data, errors do not accumulate.

[0151] It should be noted that the above processing can be applied to image patches of a predetermined proportion in the original image to obtain negative sample image patches, while the unprocessed image patches are positive sample image patches. This results in a sample image that includes both positive and negative sample image patches. By randomly combining the above three types of negative sample image patches, the robustness of training can be ensured, and the model's ability to handle various real-world inference scenarios can be enhanced.

[0152] Figure 8 This is a schematic diagram of the structure in the embodiments of this specification. This device can be applied to any device, platform, or cluster of devices with computing and processing capabilities. The device includes:

[0153] The initial generation module 801 is used to generate a first image based on prompt information from an autoregressive model. The prompt information is used to indicate the generation of visual content. The first image includes multiple first image blocks generated sequentially.

[0154] The quality assessment module 802 is used to determine the first position information of the first low-quality image block among a plurality of first image blocks. The first low-quality image block is the first image block whose quality score does not meet the preset requirements.

[0155] The regeneration module 803 is used to input the first location information into the autoregressive model, and the autoregressive model regenerates the second image block corresponding to the first location information based at least on the plurality of target image blocks or the key vectors and value vectors corresponding to the plurality of target image blocks. The plurality of target image blocks include: a plurality of first image blocks generated before the generation of the first low-quality image block, and a plurality of first image blocks generated after the generation of the first low-quality image block.

[0156] In one implementation, the regeneration module 803 is specifically used to regenerate the second image block corresponding to the first position information by an autoregressive model based on multiple target image blocks and prompt information, or the key vectors and value vectors corresponding to multiple target image blocks and prompt information.

[0157] In one implementation, the first image includes multiple image block groups. The autoregressive model generates image block groups in parallel during the generation of the first image. The multiple first image blocks constitute the first image block group. The multiple target image blocks also include several image blocks from other image block groups.

[0158] In one implementation, the initial generation module 801 is specifically used to generate a first image and quality scores of each first image block in the first image based on prompt information using an autoregressive model.

[0159] In some embodiments, all image blocks of the first image block are divided into multiple sets, each set including at least one first image block; the initial generation module 801 is specifically used to generate the first image block in the first set by an autoregressive model based on the prompt information and the position information of the first image block in the first set; based on the position information of the first image block in the first set and the first image block in the second set, the quality score of the first image block in the first set and the first image block in the second set are generated by an autoregressive model; and the first image block in other sets and the quality score of the first image block are generated iteratively.

[0160] In some embodiments, the autoregressive model is trained as follows: a sample image is acquired, comprising multiple sample image patches, each with a quality label and an image label; the autoregressive model generates a predicted image and a quality score for each sample image patch based on sample cue information, the multiple sample image patches in the sample image, and the location information of the multiple sample image patches; the network parameters of the autoregressive model are adjusted based on the difference between the predicted image patch and the image label of the corresponding sample image patch, and the difference between the quality score of the sample image patch and the quality label of the corresponding sample image patch.

[0161] In some embodiments, quality labels are used to mark sample images as positive or negative sample image patches. An autoregressive model, based on sample cue information, multiple sample image patches in the sample image, and the location information of these multiple sample image patches, generates a predicted image and quality scores for each sample image patch. This includes: constructing a first sequence based on sample cue information, multiple sample image patches in the sample image, and the location information of these multiple sample image patches; the first sequence includes sample cue information and multiple sample sets, each sample set including at least one sample image patch and / or the location information of at least one sample image patch; constructing a second sequence based on negative sample image patches and their location information; and generating a first predicted image and quality scores for each sample image patch corresponding to the first sequence, and a second predicted image and quality scores for each sample image patch corresponding to the second sequence, based on the first and second sequences, using the autoregressive model.

[0162] In some embodiments, negative sample image blocks are obtained by at least one of the following methods: generating negative sample image blocks through random noise; swapping the positions of image blocks in the original image corresponding to the sample image to obtain negative sample image blocks; or obtaining image blocks that are incorrectly predicted by the autoregressive model through teacher-forced training as negative sample image blocks.

[0163] In some embodiments, the autoregressive model includes multiple attention layers connected in series, each attention layer including an attention module, key vectors and value vectors corresponding to multiple target image patches, and key matrices and value matrices corresponding to each attention module. The key matrix includes multiple key vectors corresponding to the multiple target image patches respectively, and the value matrix includes multiple value vectors corresponding to the multiple target image patches respectively. When the regeneration module 803 regenerates the second image patch corresponding to the first location information by the autoregressive model based at least on the key vectors and value vectors corresponding to the multiple target image patches, the regeneration module 803 includes the following steps performed by the first attention module in the first attention layer of the multiple attention layers: calculating a query vector corresponding to the first location information based on the input feature vector corresponding to the first location information; calculating a first attention vector corresponding to the first location information based on the query vector, the key matrix and value matrix corresponding to the first attention module; and generating an output feature vector corresponding to the first location information based on the first attention vector, which serves as the input feature vector corresponding to the first location information for the attention module in the next attention layer.

[0164] In some embodiments, each attention layer includes multiple attention modules connected in parallel. When the regeneration module 803 generates an output feature vector corresponding to the first position information based on the first attention vector, it specifically summarizes the first attention vectors of each attention module in the first attention layer to obtain a second attention vector; and generates an output feature vector based on the second attention vector.

[0165] In some embodiments, the apparatus further includes an image calibration module (not shown) for performing a quality assessment on the second image block after regenerating the second image block corresponding to the first location information to obtain a quality score of the second image block; comparing the quality scores of the second image block corresponding to the first location information with those of the first low-quality image block; and replacing the first low-quality image block in the first image with the second image block in response to the first low-quality image block having a higher quality score than the first low-quality image block, thereby obtaining the second image.

[0166] In some embodiments, when the image calibration module is used to perform quality assessment on the second image block and obtain a quality score for the second image block, it is specifically used to input at least one second image block into an autoregressive model to obtain a quality score for the second image block.

[0167] In some embodiments, the regeneration module 803 is further configured to, in response to the fact that the number of times the image patch is regenerated has not reached a preset number of correction iterations, or that the image quality of the second image does not meet a preset quality requirement, determine the second location information of the second low-quality image patch in the second image; and based on the second location information, regenerate the third image patch corresponding to the second location information by an autoregressive model.

[0168] In some embodiments, the quality assessment module 802 is specifically used to determine the first image block with the lowest quality score as a first low-quality image block based on the sorting of the quality scores of the first image blocks; and to determine first position information based on the position of the first low-quality image block in the first image.

[0169] In some embodiments, the position information of the first image block is obtained through two-dimensional rotational position encoding.

[0170] This specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method described in any of the above embodiments.

[0171] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the above embodiments.

[0172] This specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0173] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, enabling them to generalize. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0174] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0175] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0176] Those skilled in the art will recognize that the functions described in the various embodiments disclosed in this specification in one or more of the examples above can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0177] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0178] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above descriptions are merely specific implementations of the multiple embodiments disclosed in this specification and are not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the multiple embodiments disclosed in this specification should be included within the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A method for generating an image based on a model, the method comprising: generating, by an autoregressive model, a first image based on a prompt, the prompt being used to indicate generation of visual content, the first image comprising a plurality of first image blocks generated sequentially; determining first position information of a first low-quality image block in the plurality of first image blocks, the first low-quality image block being a first image block whose quality score does not meet a preset requirement; inputting the first position information into the autoregressive model, and regenerating, by the autoregressive model, a second image block corresponding to the first position information based on at least a plurality of target image blocks or key vectors and value vectors corresponding to the plurality of target image blocks, the plurality of target image blocks comprising a plurality of first image blocks generated before the first low-quality image block and a plurality of first image blocks generated after the first low-quality image block.

2. The method of claim 1, wherein, The autoregressive model regenerates the second image block corresponding to the first position information based on at least the plurality of target image blocks or the key vectors and value vectors corresponding to the plurality of target image blocks. The autoregressive model regenerates the second image block corresponding to the first position information based on the plurality of target image blocks and the prompt or the key vectors and value vectors corresponding to the plurality of target image blocks and the prompt.

3. The method of claim 1, wherein, The first image comprises a plurality of image block groups, and the autoregressive model generates the image block groups in parallel during generation of the first image, the plurality of first image blocks constituting a first image block group, and the plurality of target image blocks further comprising a plurality of image blocks in other image block groups.

4. The method of claim 1, wherein, The autoregressive model generates the first image based on the prompt, comprising: The autoregressive model generates the first image and a quality score of each first image block in the first image based on the prompt.

5. The method of claim 4, wherein, All of the first image blocks are divided into a plurality of sets, and each set comprises at least one first image block. The autoregressive model generates the first image and the quality score of each first image block in the first image based on the prompt, comprising: The autoregressive model generates the first image block in the first set based on the prompt and position information of the first image block in the first set; The autoregressive model generates the quality score of the first image block in the first set and a first image block in a second set based on position information of the first image block in the first set and the first image block in the second set; and The autoregressive model iteratively generates the first image block and the quality score of the first image block in other sets.

6. The method of claim 4, wherein, The autoregressive model is trained in the following manner: obtaining sample images and sample prompts, the sample images comprising a plurality of sample image blocks, each sample image block having a quality label and an image label; The autoregressive model generates a predicted image and a quality score of each sample image block based on the sample prompt, a plurality of sample image blocks in the sample image, and position information of the plurality of sample image blocks. adjust network parameters of the autoregressive model based on a difference between the predicted image block and an image label of the sample image block at the corresponding position, and a difference between a quality score of the sample image block and a quality label of the sample image block at the corresponding position.

7. The method of claim 6, wherein, the quality label is used to mark the sample image as a positive sample image block or a negative sample image block; the generating, by the autoregressive model, of the predicted image and the quality score of each sample image block based on the sample prompt information, the plurality of sample image blocks in the sample image, and the position information of the plurality of sample image blocks includes: constructing a first sequence based on the sample prompt information, the plurality of sample image blocks in the sample image, and the position information of the plurality of sample image blocks, the first sequence including the sample prompt information and a plurality of sample sets, each sample set including at least one sample image block and / or position information of at least one sample image block; constructing a second sequence based on the negative sample image block and the position information of the negative sample image block; generating, by the autoregressive model, a first predicted image corresponding to the first sequence and a quality score of each sample image block, and a second predicted image block corresponding to the second sequence and a quality score of each sample image block based on the first sequence and the second sequence.

8. The method of claim 7, wherein, the negative sample image block is obtained by at least one of the following ways: generating a negative sample image block by random noise; obtaining a negative sample image block by swapping the positions of image blocks in an original image corresponding to the sample image; obtaining a negative sample image block predicted to be wrong by the autoregressive model through teacher forced training.

9. The method of claim 1, wherein, The autoregressive model includes a plurality of attention layers connected in series, each of the attention layers includes an attention module, and the plurality of target image blocks correspond to key vectors and value vectors, a key matrix and a value matrix corresponding to each attention module, the key matrix includes a plurality of key vectors corresponding to the plurality of target image blocks respectively, the value matrix includes a plurality of value vectors corresponding to the plurality of target image blocks respectively, and the regenerating, by the autoregressive model, of the second image block corresponding to the first position information based on at least the key vectors and the value vectors corresponding to the plurality of target image blocks includes performing the following steps by a first attention module in a first layer attention layer in the plurality of attention layers: calculating a query vector corresponding to the first position information based on an input feature vector corresponding to the first position information; calculating a first attention vector corresponding to the first position information based on the query vector, the key matrix and the value matrix corresponding to the first attention module; generating an output feature vector corresponding to the first position information based on the first attention vector as an input feature vector corresponding to the first position information for an attention module in a next layer attention layer.

10. The method of claim 9, wherein, Each attention layer includes a plurality of attention modules connected in series, and the generating of the output feature vector corresponding to the first position information based on the first attention vector includes: The first attention vectors of each attention module in the first layer attention layer are aggregated to obtain a second attention vector; Based on the second attention vector, the output feature vector is generated.

11. The method of claim 1, wherein, After the second image block corresponding to the first position information is regenerated, the method further comprises: The quality of the second image block is evaluated to obtain a quality score of the second image block; The quality scores of the second image block and the first low-quality image block corresponding to the first position information are compared; In response to the quality score of the second image block being higher than the quality score of the first low-quality image block, the first low-quality image block in the first image is replaced by the second image block to obtain a second image.

12. The method of claim 11, wherein, The quality of the second image block is evaluated to obtain a quality score of the second image block, comprising: The second image block is input into the autoregressive model to obtain the quality score of the second image block.

13. The method of claim 1, wherein, The method further comprises: In response to the number of regenerated image blocks not reaching a preset correction iteration number, or the image quality of the second image not meeting a preset quality requirement, the second position information of a second low-quality image block in the second image is determined; Based on the second position information, a third image block corresponding to the second position information is regenerated by the autoregressive model.

14. The method of claim 1, wherein, The first position information of the first low-quality image block in the plurality of first image blocks is determined, comprising: Based on the ranking of the quality scores of the first image blocks, a preset number of first image blocks with the lowest quality scores are determined as the first low-quality image blocks; According to the position of the first low-quality image block in the first image, the first position information is determined.

15. The method of any one of claims 1-14, wherein, The position information of the first image block is obtained by two-dimensional rotation position encoding.

16. An apparatus for generating an image based on a model, the apparatus comprising: An initial generation module configured to generate, by an autoregressive model, a first image based on prompt information, the prompt information being used to indicate the generation of visual content, the first image comprising a plurality of first image blocks generated in sequence, the plurality of target image blocks comprising a plurality of first image blocks generated before the first low-quality image block and a plurality of first image blocks generated after the first low-quality image block; A quality evaluation module configured to determine first position information of a first low-quality image block in the plurality of first image blocks, the first low-quality image block being a first image block with a quality score not meeting a preset requirement; A regeneration module configured to input the first position information into the autoregressive model, and regenerate, by the autoregressive model, a second image block corresponding to the first position information based on at least the plurality of target image blocks or key vectors and value vectors corresponding to the plurality of target image blocks, the plurality of target image blocks comprising a plurality of first image blocks generated before the first low-quality image block and a plurality of first image blocks generated after the first low-quality image block.

17. A computing device comprising a memory having executable code stored therein and a processor that, when executing the executable code, implements the method of any one of claims 1-15.

Images