Auto-regression image generation method and device based on multiple solution terminals

By adding a parallel decoding head to the autoregressive model and using historical image sets for training, combined with a speculative decoding strategy, the problems of computational delay and high cost of the autoregressive model in image generation are solved, and efficient image generation is achieved.

CN120599069APending Publication Date: 2025-09-05BEIJING AUTONAVI YUNMAP TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510713979.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-29
Publication Date
2025-09-05

AI Technical Summary

Technical Problem

Autoregressive models have problems with high computational latency, high computational cost, and poor generation quality in image generation. Especially when generating images from text, there are few existing acceleration methods.

Method used

Parallel decoding heads are added to the initial autoregressive model and trained using historical image sets. Images are generated through a speculative decoding strategy, which reduces training resource consumption and improves generation quality.

Benefits of technology

The process of autoregressive text generation to image is accelerated, the computational cost is reduced, while maintaining or improving the quality and diversity of image generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120599069A_ABST
    Figure CN120599069A_ABST
Patent Text Reader

Abstract

The invention provides an autoregression image generation method and device based on multiple solution wharfs. The method comprises the following steps: adding a plurality of parallel solution wharfs on an initial autoregression model; inputting a historical image set serving as a training sample set into a plurality of parallel solution wharfs for training to obtain a plurality of trained parallel solution wharfs; inputting a to-be-recognized image into the plurality of trained parallel solution wharfs, and predicting the positions of the plurality of visual lexical elements in the model reasoning process by using the plurality of trained parallel solution wharfs to obtain a target prediction result; and based on the target prediction result, using a speculation decoding strategy to perform decoding operation on the plurality of visual lexical elements to obtain a target autoregression image corresponding to the plurality of visual lexical elements. Under the condition that the autoregression image generation quality and the diversity pre-training model are ensured, the autoregression text is accelerated to generate the image, the image generation quality is improved, the training resource consumption of the autoregression model is reduced, and the calculation cost is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image generation, and in particular to a method and device for generating autoregressive images based on multiple decoding heads. Background Art

[0002] Autoregressive models are widely used in various fields, including language generation, image generation, and video generation, through stepwise prediction techniques. In the field of text-to-image generation, autoregressive models demonstrate excellent generation quality, capable of generating high-quality, high-resolution images. They also have great potential for model scalability, laying the foundation for understanding and generating unified native multimodal models.

[0003] However, since the autoregressive model must predict visual tokens one by one, hundreds or even thousands of forward calculations are required to generate a complete image. Therefore, the autoregressive paradigm (a method of gradually generating sequences based on historical data) has a high latency during inference. Currently, there is still relatively little research on acceleration methods for autoregressive text-to-image models.

[0004] In related technologies, the autoregressive model that supports text generation to images usually contains billions of parameters, which makes training-based generative model acceleration methods such as self-consistency distillation have the disadvantages of high computational cost, slow image generation efficiency, and poor image generation quality. Summary of the Invention

[0005] The purpose of this application is to provide an autoregressive image generation method and device based on multiple decoding heads, aiming to accelerate the generation of images from autoregressive text while ensuring the quality of autoregressive image generation and a diverse pre-trained model, improve the quality of image generation, reduce the training resource consumption of the autoregressive model, and reduce the computational cost.

[0006] According to a first aspect of the present application, a method for generating an autoregressive image based on multiple decoding heads is provided, comprising:

[0007] Add several parallel decoding heads to the initial autoregressive model;

[0008] Using the historical image set as a training sample set to input into the plurality of parallel decoding heads for training, thereby obtaining the trained plurality of parallel decoding heads;

[0009] Inputting the image to be recognized into the trained multiple parallel decoding heads, and using the trained multiple parallel decoding heads to predict the positions of multiple visual word units during the model inference process to obtain a target prediction result;

[0010] Based on the target prediction results, a speculative decoding strategy is used to perform decoding operations on the plurality of visual words to obtain target autoregressive images corresponding to the plurality of visual words.

[0011] In an optional embodiment, several parallel decoding heads are added to the initial autoregressive model, including:

[0012] The final hidden state h at position t of the initial autoregressive model t Add k decoding heads;

[0013] The k decoding heads are arranged in parallel.

[0014] In an optional embodiment, the image to be recognized is input into the plurality of trained parallel decoding heads, and the positions of a plurality of visual word units are predicted by using the plurality of trained parallel decoding heads during the model inference process to obtain a target prediction result, including:

[0015] Predicting the first visual word at the t+1th position using the original prediction head of the initial autoregressive model to obtain a first vocabulary containing the predicted first visual word-units;

[0016] The k-th decoding head is used to predict the second visual word at the t+k+1-th position, to obtain a second vocabulary containing a plurality of predicted second visual words.

[0017] In an optional embodiment, after adding a plurality of parallel decoding heads to the initial autoregressive model, the method includes:

[0018] The decoding head parameters corresponding to the plurality of parallel decoding heads are set to be consistent with the prediction head parameters corresponding to the original prediction head of the initial autoregressive model.

[0019] In an optional embodiment, a historical image set is used as a training sample set to input into the plurality of parallel decoding heads for training, thereby obtaining the trained plurality of parallel decoding heads, including:

[0020] Freezing the initial autoregressive model backbone model;

[0021] Inputting the historical image set into the plurality of parallel decoding heads, and obtaining prediction results output by the plurality of parallel decoding heads including predicted positions of a plurality of predicted visual word units;

[0022] Calculating a loss value using a cross entropy loss function based on the prediction results including the predicted positions of the plurality of predicted visual word-grams and the historical results including the historical positions of the plurality of historical visual word-grams;

[0023] When the loss value is less than or equal to a preset loss threshold, the training is determined to be completed, and the trained plurality of parallel decoding heads are obtained;

[0024] When the loss value is greater than a set loss threshold, the training parameters are adjusted until the training is completed to obtain the trained plurality of parallel decoding heads.

[0025] In an optional embodiment, based on the target prediction result, a speculative decoding strategy is used to perform a decoding operation on the plurality of visual words to obtain target autoregressive images corresponding to the plurality of visual words, including:

[0026] Selecting a plurality of candidate word-grams from the plurality of visual word-grams;

[0027] Performing word segmentation processing on the multiple candidate word units that have been screened to obtain candidate word units after word segmentation processing;

[0028] Verifying the candidate word units after the word segmentation processing to obtain target candidate word units that need to be retained, and accepting the target candidate word units;

[0029] The target autoregressive image is generated based on the target candidate word.

[0030] In an optional embodiment, verifying the candidate word after the word segmentation processing includes:

[0031] The candidate word units after the word segmentation processing are verified using a rejection sampling algorithm.

[0032] According to a second aspect of the present application, there is provided an autoregressive image generation device based on multiple decoding heads, comprising:

[0033] an adding unit, configured to add a number of parallel decoding heads to the initial autoregressive model;

[0034] A training unit is configured to use a historical image set as a training sample set to input the plurality of parallel decoding heads for training, thereby obtaining a plurality of trained parallel decoding heads;

[0035] a prediction unit configured to input the image to be recognized into the plurality of trained parallel decoding heads, and use the plurality of trained parallel decoding heads to predict the positions of a plurality of visual word units during a model inference process to obtain a target prediction result;

[0036] The decoding unit is configured to perform a decoding operation on the plurality of visual words using a speculative decoding strategy based on the target prediction result to obtain target autoregressive images corresponding to the plurality of visual words.

[0037] According to a third aspect of the present application, the present application further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned autoregressive image generation methods based on multiple decoding heads.

[0038] According to the fourth aspect of the present application, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-mentioned autoregressive image generation methods based on multiple decoding heads.

[0039] Compared with related technologies, the technical solution of this application has the following advantages:

[0040] This application does not require the separate training of a new small model, but only requires the training of a decoding head with a small number of parameters. By fine-tuning the initial autoregressive model with a small number of parameters, the consumption of training resources is reduced and the quality of image generation is improved. The solution of this application ensures the quality of autoregressive image generation and the diversity of pre-trained models, accelerates the generation of images from autoregressive text, reduces the training resource consumption of the autoregressive model, and reduces the computational cost.

[0041] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purpose and other advantages of the present application can be achieved and obtained through the structures and processes indicated in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. It is obvious that the drawings described below are certain embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 A schematic diagram of a flow chart of an autoregressive image generation method based on multiple decoding heads provided in an embodiment of the present disclosure;

[0044] Figure 2 A schematic diagram of the structure of a parallel decoding head provided in an embodiment of the present disclosure;

[0045] Figure 3 A schematic diagram of a process for performing decoding operations using a speculative decoding strategy according to an embodiment of the present disclosure;

[0046] Figure 4 A schematic structural diagram of an autoregressive image generation device based on multiple decoding heads provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] like Figure 1 As shown, Figure 1 A flowchart of a method for generating an autoregressive image based on multiple decoding heads provided in an embodiment of the present disclosure is provided. The method for generating an autoregressive image based on multiple decoding heads provided in an embodiment of the present disclosure includes the following steps:

[0049] S101. Add several parallel decoding heads to the initial autoregressive model.

[0050] S102: Use the historical image set as a training sample set to input into the plurality of parallel decoding heads for training, and obtain the trained plurality of parallel decoding heads.

[0051] S103: Input the image to be recognized into the trained multiple parallel decoding heads, and use the trained multiple parallel decoding heads to predict the positions of multiple visual word units during the model inference process to obtain a target prediction result.

[0052] S104 : Based on the target prediction result, a speculative decoding strategy is used to perform a decoding operation on the plurality of visual words to obtain target autoregressive images corresponding to the plurality of visual words.

[0053] This application does not require the separate training of a new small model, but only requires the training of a decoding head with a small number of parameters. By fine-tuning the initial autoregressive model with a small number of parameters, the consumption of training resources is reduced and the quality of image generation is improved. The solution of this application ensures the quality of autoregressive image generation and the diversity of pre-trained models, accelerates the generation of images from autoregressive text, reduces the training resource consumption of the autoregressive model, and reduces the computational cost.

[0054] In an optional embodiment, several parallel decoding heads are added to the initial autoregressive model, including:

[0055] The final hidden state h at position t of the initial autoregressive model t k decoding heads are added to the above; where position t refers to the position of the visual word to be predicted;

[0056] The k decoding heads are arranged in parallel.

[0057] In an optional embodiment, the image to be recognized is input into the plurality of trained parallel decoding heads, and the positions of a plurality of visual word units are predicted by using the plurality of trained parallel decoding heads during the model inference process to obtain a target prediction result, including:

[0058] The original prediction head of the initial autoregressive model is used to predict the first visual word at the t+1th position, and a first vocabulary containing several predicted first visual words is obtained. The first visual word can be expressed as

[0059] The k-th decoding head is used to predict the second visual word at the t+k+1-th position, and a second vocabulary containing several predicted second visual words is obtained. The second visual word can be expressed as

[0060] like Figure 2 As shown, Figure 2 A schematic diagram of the structure of a parallel decoding head provided in an embodiment of the present disclosure, comprising Figure 2 As shown, the autoregressive image generation model is the initial autoregressive model, which includes the original prediction head, Transformer layer, and input embedding layer. Decoding head 1, decoding head 2, ..., decoding head k are the parallel decoding heads added to the initial autoregressive model in the disclosed embodiment.

[0061] It should be noted that the user can set the number of additional parallel decoding heads according to actual business needs, and the embodiments of the present disclosure do not limit this.

[0062] In an optional embodiment, after adding a plurality of parallel decoding heads to the initial autoregressive model, the method includes:

[0063] The decoding head parameters corresponding to the plurality of parallel decoding heads are set to be consistent with the prediction head parameters corresponding to the original prediction head of the initial autoregressive model.

[0064] During model initialization, the disclosed embodiment sets the decoding head parameters corresponding to all parallel decoding heads to be consistent with the prediction head parameters corresponding to the prediction head of the initial autoregressive model, thereby ensuring that the prediction results of the parallel decoding heads are aligned with the prediction results of the initial autoregressive model. This initialization method ensures that the parallel decoding heads are consistent with the prediction results of the initial autoregressive model during the model inference process, thereby significantly accelerating the generation speed of the autoregressive image while ensuring the quality of the autoregressive image generation.

[0065] In an optional embodiment, a historical image set is used as a training sample set to input into the plurality of parallel decoding heads for training, thereby obtaining the trained plurality of parallel decoding heads, including:

[0066] Freezing the initial autoregressive model backbone model;

[0067] Inputting the historical image set into the plurality of parallel decoding heads, and obtaining prediction results output by the plurality of parallel decoding heads including predicted positions of a plurality of predicted visual word units;

[0068] Calculating a loss value using a cross entropy loss function based on the prediction results including the predicted positions of the plurality of predicted visual word-grams and the historical results including the historical positions of the plurality of historical visual word-grams;

[0069] When the loss value is less than or equal to a preset loss threshold, the training is determined to be completed, and the trained plurality of parallel decoding heads are obtained;

[0070] When the loss value is greater than a set loss threshold, the training parameters are adjusted until the training is completed to obtain the trained plurality of parallel decoding heads.

[0071] It should be noted that users can set the loss threshold according to actual business needs, and the embodiments of the present disclosure do not limit this.

[0072] Optionally, the historical visual word at the (t+k+1)th historical position is represented as y t+k+1 , the cross entropy loss function L of the k-th decoding head k Expressed as: Where, It indicates that the k-th decoding head predicts the second visual word at the (t+k+1)-th position, and t represents the position of the initial autoregressive model.

[0073] In an optional embodiment, based on the target prediction result, a speculative decoding strategy is used to perform a decoding operation on the plurality of visual words to obtain target autoregressive images corresponding to the plurality of visual words, including:

[0074] Selecting a plurality of candidate word-grams from the plurality of visual word-grams;

[0075] Performing word segmentation processing on the multiple candidate word units that have been screened to obtain candidate word units after word segmentation processing;

[0076] Verifying the candidate word units after the word segmentation processing to obtain target candidate word units that need to be retained, and accepting the target candidate word units;

[0077] The target autoregressive image is generated based on the target candidate word.

[0078] In an optional embodiment, verifying the candidate word after the word segmentation processing includes:

[0079] The candidate word units after the word segmentation processing are verified using a rejection sampling algorithm.

[0080] like Figure 3 As shown, Figure 3 A flowchart of a decoding operation using a speculative decoding strategy is provided for an embodiment of the present disclosure. The embodiment of the present disclosure trains parallel decoding heads so that the autoregressive model can simultaneously predict the positions of multiple visual word units through the parallel decoding heads during each model inference. In order to solve the problem of large errors in the newly trained decoding heads and poor quality of autoregressive image generation, the embodiment of the present disclosure adopts a speculative decoding strategy to perform multi-word verification and modification on the initial autoregressive model, thereby improving the overall model inference efficiency. In addition, the parallel decoding head of the embodiment of the present disclosure is based on the initial autoregressive model, and does not require re-training a small model separately. Instead, it only requires training a parallel decoding head with a small number of parameters, which reduces the consumption of training resources and speeds up the training efficiency and subsequent autoregressive image generation efficiency.

[0081] like Figure 4 As shown, Figure 4 This is a schematic structural diagram of an autoregressive image generation device based on multiple decoding heads provided in an embodiment of the present disclosure. The present disclosure provides an autoregressive image generation device based on multiple decoding heads, comprising:

[0082] An adding unit 41 is configured to add a plurality of parallel decoding heads to the initial autoregressive model;

[0083] The training unit 42 is configured to use the historical image set as a training sample set to input the plurality of parallel decoding heads for training, thereby obtaining the trained plurality of parallel decoding heads;

[0084] The prediction unit 43 is configured to input the image to be recognized into the plurality of trained parallel decoding heads, and use the plurality of trained parallel decoding heads to predict the positions of a plurality of visual word units during the model inference process to obtain a target prediction result;

[0085] The decoding unit 44 is configured to perform a decoding operation on the plurality of visual word-grams using a speculative decoding strategy based on the target prediction result to obtain target autoregressive images corresponding to the plurality of visual word-grams.

[0086] The above-mentioned device can be implemented by the autoregressive image generation method based on multiple decoding heads provided in the embodiment of the first aspect above. The specific implementation method can be found in the description of the embodiment of the first aspect and will not be repeated here.

[0087] According to an embodiment of the present disclosure, a computer-readable storage medium stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the aforementioned multi-decoding head-based autoregressive image generation method of each embodiment of the present disclosure are performed.

[0088] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0089] The computer device according to an embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache). The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc.

[0090] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is configured to execute the computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the aforementioned multi-decoding head-based autoregressive image generation method of each embodiment of the present disclosure.

[0091] It is understood that the model structures, names, and parameters described in the above embodiments are merely examples. Those skilled in the art may also readily conceive of combinations and adjustments to the structural features of the above embodiments as needed, and should not limit the concepts of this application to the specific details of the above examples.

[0092] Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating autoregressive images based on multiple decoding heads, characterized in that: include: Add several parallel decoding heads to the initial autoregressive model; Using the historical image set as a training sample set to input into the plurality of parallel decoding heads for training, thereby obtaining the trained plurality of parallel decoding heads; Inputting the image to be recognized into the trained multiple parallel decoding heads, and using the trained multiple parallel decoding heads to predict the positions of multiple visual word units during the model inference process to obtain a target prediction result; Based on the target prediction results, a speculative decoding strategy is used to perform decoding operations on the plurality of visual words to obtain target autoregressive images corresponding to the plurality of visual words.

2. The method for generating an autoregressive image based on multiple decoding heads according to claim 1, wherein: Add several parallel decoding heads to the initial autoregressive model, including: The final hidden state h at position t of the initial autoregressive model t Add k decoding heads; The k decoding heads are arranged in parallel.

3. The method for generating an autoregressive image based on multiple decoding heads according to claim 2, wherein: Inputting the image to be recognized into the trained multiple parallel decoding heads, using the trained multiple parallel decoding heads to predict the positions of multiple visual word units during the model inference process, and obtaining target prediction results, including: Predicting the first visual word at the t+1th position using the original prediction head of the initial autoregressive model to obtain a first vocabulary containing the predicted first visual word-units; The k-th decoding head is used to predict the second visual word at the t+k+1-th position, and a second vocabulary including a plurality of predicted second visual words is obtained.

4. The method for generating an autoregressive image based on multiple decoding heads according to claim 1, wherein: After adding several parallel decoding heads to the initial autoregressive model, the method includes: The decoding head parameters corresponding to the plurality of parallel decoding heads are set to be consistent with the prediction head parameters corresponding to the original prediction head of the initial autoregressive model.

5. The method for generating an autoregressive image based on multiple decoding heads according to claim 1, wherein: The historical image set is used as a training sample set to be input into the plurality of parallel decoding heads for training, thereby obtaining the trained plurality of parallel decoding heads, including: Freezing the initial autoregressive model backbone model; Inputting the historical image set into the plurality of parallel decoding heads, and obtaining prediction results output by the plurality of parallel decoding heads including predicted positions of a plurality of predicted visual word units; Calculating a loss value using a cross entropy loss function based on the prediction results including the predicted positions of the plurality of predicted visual word-grams and the historical results including the historical positions of the plurality of historical visual word-grams; When the loss value is less than or equal to a preset loss threshold, the training is determined to be completed, and the trained plurality of parallel decoding heads are obtained; When the loss value is greater than a preset loss threshold, the trained plurality of parallel decoding heads are obtained by adjusting the training parameters until the training is completed.

6. The method for generating an autoregressive image based on multiple decoding heads according to claim 1, wherein: Based on the target prediction result, a speculative decoding strategy is used to perform a decoding operation on the plurality of visual words to obtain target autoregressive images corresponding to the plurality of visual words, including: Selecting a plurality of candidate word-grams from the plurality of visual word-grams; Performing word segmentation processing on the multiple candidate word units that have been screened to obtain candidate word units after word segmentation processing; Verifying the candidate word units after the word segmentation processing to obtain target candidate word units that need to be retained, and accepting the target candidate word units; The target autoregressive image is generated based on the target candidate word.

7. The method for generating an autoregressive image based on multiple decoding heads according to claim 6, wherein: Verifying the candidate word after the word segmentation process includes: The candidate word units after the word segmentation processing are verified using a rejection sampling algorithm.

8. An autoregressive image generation device based on multiple decoding heads, characterized in that: include: an adding unit, configured to add a number of parallel decoding heads to the initial autoregressive model; A training unit is configured to use a historical image set as a training sample set to input the plurality of parallel decoding heads for training, thereby obtaining a plurality of trained parallel decoding heads; a prediction unit configured to input the image to be recognized into the plurality of trained parallel decoding heads, and use the plurality of trained parallel decoding heads to predict the positions of a plurality of visual word units during a model inference process to obtain a target prediction result; The decoding unit is configured to perform a decoding operation on the plurality of visual words using a speculative decoding strategy based on the target prediction result to obtain target autoregressive images corresponding to the plurality of visual words.

9. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the autoregressive image generation method based on multiple decoding heads as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the autoregressive image generation method based on multiple decoding heads according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Auto-regression model parallel decoding method based on lexical tree dynamic generation

    CN118153523A

  • Pre-training for scene text detection

    US20240119743A1