Adaptive image generation method and system based on multi-modal routing

Abstract

The invention discloses a self-adaptive image generation method and system based on multi-modal routing, and the method comprises the steps: extracting the multi-scale features of an input image, and generating three different tokens with progressive increase of continuous information keeping capability: respectively extracting visual and text modal information from the image and corresponding text description, and generating a multi-modal information abstract after fusion; inputting the multi-modal information abstract into a learnable soft router, and dynamically selecting a token modeling path based on an image complexity label; a three-stage training strategy optimization model is adopted; in the inference stage, a trained soft router dynamically selects a token path to complete image generation according to input text description and an image information abstract predicted by an autoregressive Transform. According to the method, a dynamic router and three quantification and modeling strategies with different complexities are fused, and dynamic modeling path selection is realized in a reasoning stage through a soft router module. According to the method, on the premise of ensuring the generation quality, the reasoning efficiency is effectively improved, and good efficiency is shown.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation, and more particularly to a self-adaptive image generation method and system based on multi-modal routing. BACKGROUND

[0002] In recent years, visual autoregressive generation models, as an important branch of generative artificial intelligence, have shown broad prospects in image synthesis, image-text generation and other tasks (Yu L, Lezama J, Gundavarapu N B, et al. Language Model Beats Diffusion--Tokenizer is Key to Visual Generation [J]. arXiv preprint arXiv: 2310.05737, 2023. Such models usually discretize continuous image data into token sequences and use transformer structures to predict tokens one by one, thereby realizing step-by-step image generation. The autoregressive generation framework not only enables cross-modal transfer with the technological progress of large-scale language models, but also becomes one of the mainstream paradigms of visual generation due to its good controllability and scalability.

[0003] However, compared with continuous modeling methods such as diffusion models (Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models [J]. Advances in neural information processing systems, 2020, 33: 6840-6851. Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis [J]. Advances in neural information processing systems, 2021, 34: 8780-8794.), autoregressive models still have a significant gap in image detail preservation. The root cause lies in the severe loss of continuous information in the vector quantization process. This detail loss caused by discretization is particularly evident in high-resolution, complex semantic or structural fine image generation.

[0004] Meanwhile, in current text-driven image generation tasks, the complexity of target images shows high heterogeneity, involving color, texture, spatial structure, and other aspects of change. Therefore, the traditional unified modeling path is difficult to balance generation quality and reasoning efficiency at the same time. An ideal generation system should be able to dynamically select different modeling methods according to the complexity of the text prompt, thereby effectively controlling the consumption of computing resources while maintaining image quality.

[0005] To solve the above problems, some existing researches have tried to introduce mixed token design (Kou S, Jin J, Liu Z, et al. Orthus: Autoregressive interleaved image-text generation with modality-specific heads [J]. arXiv preprint arXiv:2412.00127, 2024.), continuous residual modeling (Tang H, Wu Y, Yang S, et al. Hart: Efficient visual generation with hybrid autoregressive transformer [J]. arXiv preprint arXiv:2410.10812, 2024.) and other strategies to improve generation details, but none of them can systematically solve the contradiction between continuous information preservation and modeling efficiency. At the same time, there is still a lack of a mechanism that can adaptively select token modeling path according to the input text to dynamically respond to the complexity changes of the generation task.

[0006] Therefore, it is urgent to propose an autoregressive visual generation method with dynamic selection capability, which can realize optimal scheduling of modeling path according to the input prompt on the basis of fully preserving the continuous features of images, so as to narrow the quality gap between autoregressive and diffusion models and improve the overall reasoning efficiency. SUMMARY

[0007] Therefore, the present application provides a multi-modal routing-based adaptive image generation method and system, which can overcome the defects of low detail fidelity, poor reasoning efficiency and lack of dynamic modeling capability of visual autoregressive generation methods, and aims to alleviate the problem of using a modeling method with uniform complexity for all samples in the text-to-image task, thereby affecting efficiency or quality.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] In a first aspect, the present application provides a multi-modal routing-based adaptive image generation method, comprising the following steps:

[0010] S1. Extracting multi-scale features of the input image to generate three types of tokens with increasing ability to preserve continuous information, including: obtaining discrete tokens through vector quantization method, hybrid tokens based on Top-K weighted aggregation, and continuous tokens modeled by a lightweight diffusion MLP;

[0011] S2. Extracting visual and text modal information from the image and corresponding text description respectively, and generating a multi-modal information summary after fusion;

[0012] S3. Inputting the multi-modal information summary into a learnable soft router to dynamically select the token modeling path based on the image complexity label;

[0013] S4. Using a three-stage training strategy to optimize the model:

[0014] Stage one: training the decoder to reconstruct the feature maps of the three types of tokens;

[0015] Stage two: jointly training the decoder and the soft router according to the image complexity label;

[0016] Stage three: training the autoregressive Transformer to generate a multi-token fusion output;

[0017] S5. In the inference stage, according to the input text description and the image information summary predicted by the autoregressive Transformer, the trained soft router dynamically selects the token path to complete the image generation.

[0018] Further, the step S1 includes:

[0019] Feature encoding is performed on the multi-scale residual image extracted from the input image to construct a pre-trained codebook with size K and channel number D for feature vector quantization; wherein the feature map F i of the i-th feature scale is represented as:

[0020]

[0021] where h i w i is the number of i-th scale feature vectors; for the j-th feature vector f ij in the i-th feature map, the quantization method of the three types of tokens includes:

[0022] (1) Discrete token: vector quantization is performed through the pre-trained codebook C, and the vector index z ij corresponding to the highest similarity vector is obtained to obtain the quantized discrete token f ij :

[0023]

[0024] (2) Mixed token: based on Top-K weighted fusion, select the Top-K nearest similar feature vector indexes from the pre-trained codebook C, denoted as:

[0025]

[0026] wherein Ω(f ij ) is the Top-K function selected Top-K nearest similar feature vector index set; w k is the weight calculated for the K similar codebook vectors; c k is the kth codebook vector;

[0027] (3) Continuous token: residual prediction is performed on the feature map constructed by the discrete token through a lightweight diffusion multi-layer perception model, thereby realizing continuous modeling, denoted as wherein is the residual feature map predicted by the lightweight diffusion MLP; F dis is the discrete representation generated after the image feature is vector quantized.

[0028] Further, the step S2 comprises:

[0029] The input image and the corresponding text are respectively subjected to feature extraction, and in the training process, the image adopts an encoder based on VQ-GAN to extract discrete visual features, while in the inference process, the visual features after scale alignment are directly used after the output of the Transformer; the text adopts a pre-trained language model to extract text features;

[0030] The extracted image features and text features are subjected to element-wise Hadamard multiplication, realizing semantic alignment and fusion;

[0031] The fused result is mapped to a unified dimension of multi-modal information summary through a linear layer.

[0032] Further, the soft router in the step S3 comprises:

[0033] (1) Input layer: receiving the multi-modal summary vector as the input feature of the router;

[0034] (2) Multi-layer perception structure: containing multiple fully connected layers and an activation function ReLU, for nonlinear transformation of the summary vector;

[0035] (3) Output layer: outputting a three-dimensional weight vector, representing the selection probability of each token path;

[0036] (4) Discrimination criterion: the cross-entropy loss function is used, and the image complexity label provided in the training stage is used as a supervision signal to guide the soft router to learn to select the optimal token modeling path based on the text description for images of different complexity.

[0037] Further, the three-stage training strategy in step S4 includes:

[0038] (1) Stage one: token generation stage, freeze the visual encoder and pre-trained codebook, train the visual decoder to learn the reconstruction ability of discrete tokens, mixed tokens and continuous tokens respectively, and use reconstruction loss;

[0039] (2) Stage two: router training stage, freeze the visual encoder and pre-trained codebook, unfreeze the visual decoder, train the router to generate token selection probability distribution through supervised learning method using cross-entropy loss with image complexity label as supervision;

[0040] (3) Stage three: unified autoregressive generation stage, train the Transformer autoregressive decoder based on the frozen token path, and the loss function is the autoregressive generation loss of the multi-token fusion output, including negative log likelihood loss and text-image consistency constraint loss.

[0041] Further, the step S5 includes:

[0042] In the inference stage, the text description of the generated image is input, and the text feature is obtained through the large language model;

[0043] The text feature is input into the autoregressive model for first-step discrete prediction, and the discrete visual residual feature map is output;

[0044] The visual residual feature map and the text feature are fused in multiple modalities to enter the soft routing dynamic token path selection;

[0045] Based on all the predicted tokens, the complete feature map is constructed, and the generated image is decoded through the visual decoder.

[0046] In a second aspect, the embodiments of the present application also provide an adaptive image generation system based on multi-modal routing, comprising:

[0047] A multi-scale feature extraction module is used to extract multi-scale features of an input image to generate three types of tokens with increasing continuous information retention capabilities, including: discrete tokens obtained through vector quantization method, mixed tokens based on Top-K weighted aggregation, and continuous tokens modeled through lightweight diffusion MLP;

[0048] A multi-modal fusion module is configured to extract visual and text modal information from the image and the corresponding text description respectively, and generate a multi-modal information summary after fusion.

[0049] A soft router module is configured to input the multi-modal information summary into a learnable soft router, and dynamically select a token modeling path based on an image complexity label.

[0050] A training module is configured to optimize the model by adopting a three-stage training strategy: in stage one, the decoder is trained to reconstruct feature maps of three types of tokens; in stage two, the decoder and the soft router are jointly trained according to the image complexity label; and in stage three, the autoregressive Transformer is trained to generate a multi-token fusion output.

[0051] A generation module is configured to, in an inference stage, generate an image according to an input text description and an image information summary predicted by the autoregressive Transformer, and dynamically select a token path by the trained soft router to complete the image generation.

[0052] According to the technical solution described above, compared with the prior art, the present application has the following technical advantages:

[0053] The present application is based on the connection between image complexity and text description, and designs three different quantization modes corresponding to low, medium and high complexity image representations, and realizes dynamic modeling path selection in the inference stage through the soft router module. This method effectively improves the inference efficiency while ensuring the generation quality, and shows good efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0055] Figure 1 A multi-modal routing based adaptive image generation method flowchart is provided for the present application.

[0056] Figure 2 A multi-modal routing based adaptive image generation framework principle diagram is provided for the present application. DETAILED DESCRIPTION

[0057] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0058] Embodiment 1

[0059] The embodiments of the present application disclose a self-adaptive image generation method based on multi-modal routing, as shown in Figure 1 , which includes S1-S5:

[0060] S1. Extracting multi-scale features of an input image to generate three types of tokens with increasing ability to retain continuous information: including: obtaining discrete tokens by vector quantization method, mixed tokens based on Top-K weighted aggregation, and continuous tokens modeled by lightweight diffusion MLP;

[0061] This step designs three types of tokens with different abilities to retain continuous information, including discrete, mixed and continuous tokens, and constructs a unified decoder to uniformly decode the feature maps constructed by the above token types.

[0062] Specifically, this step S1 includes:

[0063] Feature encoding is performed on the multi-scale residual image. We will prepare a pre-trained codebook with a size of K and a channel number of D in advance for feature vector quantization. The feature map F i of the i-th feature scale can be represented as:

[0064]

[0065] where h i w i is the number of i-th scale feature vectors, and D is the channel of the feature map F i . For the j-th feature vector f ij in the i-th feature map, the way to quantize it into three types of tokens includes:

[0066] (1) Discrete token: directly vector quantization through the pre-trained codebook C, retaining key content but compressing semantic details. By calculating the vector with the highest similarity in the codebook, the corresponding vector index z ij is obtained, and the discrete token after quantization of f ij can be represented as:

[0067]

[0068] (2)Mixed token: based on Top-K weighted fusion, the Top-K similar feature vector indexes are selected from the pre-trained codebook, and a single representation is aggregated through a learnable linear transformation, which has both discrete and continuous characteristics, which can be represented as:

[0069]

[0070] where Ω(f ij ) is the Top-K function selected Top-K similar feature vector index set; w k is the weight calculated for K similar codebook vectors; c k is the kth codebook vector.

[0071] (3)Continuous token: residual prediction is performed on the discrete token constructed feature map through a lightweight diffusion multi-layer perception model, thereby realizing continuous modeling, which can be represented as where is the residual feature map predicted by the lightweight diffusion MLP; F dis is the discrete representation generated after the image feature is quantized.

[0072] S2. Extract visual and text modal information from the image and corresponding text description respectively, and generate a multi-modal information summary after fusion;

[0073] Specifically, it includes:

[0074] S21: Extract features from the input image and text respectively. In the training process, the image adopts the encoder based on VQ-GAN to extract discrete visual features, while in the inference process, the visual features are directly used after the output of the Transformer and scale alignment. The text adopts a pre-trained language model to extract text features. The VQ-GAN encoder includes two parts: CNN to extract continuous features and VQ to quantize discrete features.

[0075] S22: Element-wise Hadamard multiplication of image features and text features to realize semantic alignment and fusion;

[0076] S23: The result of fusion is mapped to a unified dimension summary vector through a linear layer, which is used to guide the selection of the subsequent token path.

[0077] S3. Input the multi-modal information summary into a learnable soft router to dynamically select the token modeling path based on image complexity; wherein the design of the soft router includes:

[0078] (1) Input layer: receiving multi-modal summary vectors as input features of the router;

[0079] (2) Multi-layer perceptron structure: contains multiple fully connected layers, activation function ReLU, for non-linear transformation of the summary vector;

[0080] (3) Output layer: output three-dimensional weight vector, representing the selection probability of each token path;

[0081] (4) Discriminant criterion: cross-entropy loss function is used, and the image complexity label provided in the training stage is used as a supervision signal to guide the soft router to learn to select the optimal token modeling path based on the text description for images of different complexity.

[0082] S4. Adopting a three-stage training strategy to optimize the model:

[0083] Stage one: training the decoder to reconstruct the feature map of three types of tokens; in the token generation stage, the visual encoder and the pre-trained codebook are frozen, and the visual decoder is trained to learn the reconstruction ability of discrete tokens, mixed tokens and continuous tokens respectively, and the reconstruction loss is used.

[0084] Stage two: jointly training the decoder and the soft router according to the image complexity label; in the router training stage, the visual encoder and the pre-trained codebook are frozen, and the visual decoder is unfrozen; through supervised learning, the cross-entropy loss is used with the image complexity label as supervision to train the router to generate token selection probability distribution; the decoder should be unfrozen, and the purpose is to balance the efficiency and quality of the joint training of the decoder and the router.

[0085] Stage three: training the autoregressive Transformer to generate multi-token fusion output; in the unified autoregressive generation stage, the Transformer autoregressive decoder is trained on the basis of frozen token paths, and the loss function is the autoregressive generation loss of multi-token fusion output, including NLL (negative log likelihood) loss and image-text consistency constraint loss.

[0086] S5. In the inference stage, the input text description and the image information summary predicted by the autoregressive Transformer are used to dynamically select the token path by the trained soft router to complete the image generation.

[0087] In step S5, the text description for generating an image is input in the inference stage, and the text feature is obtained by a large language model; the text feature is input into the autoregressive model to perform first-step discrete prediction and output discrete visual residual feature map; the visual residual feature map is fused with the text feature to dynamically select the token path by the soft router; based on all the predicted tokens, a complete feature map is constructed, which is decoded into a generated image by a visual decoder.

[0088] The present application realizes the establishment of the relationship between the text description and the image complexity by the collaborative design of three types of token quantization methods (discrete token, hybrid token, and continuous token) with increasing computational complexity and modeling performance, and then realizes the on-demand modeling based on the text description, while ensuring the high quality of the generation and greatly reducing the computational overhead.

[0089] In specific implementation, the following 11 steps can be referred to:

[0090] Step 1: Multi-scale visual feature extraction. The input image N is processed by the multi-scale residual of the image encoder to obtain S scale feature maps F = {F 1 ,F 2 ,F 3 ,...,F S}, wherein the i-th feature map can be represented as:

[0091]

[0092] wherein h i w i is the number of i-th scale feature vectors, and D is the channel of the feature map F i .

[0093] Step 2: Discrete token construction. For the j-th feature vector f ij in the i-th scale residual feature map, a simple vector quantization method is used to discretize the multi-scale feature map, and the pre-trained multi-scale codebook is used to select the nearest neighbor codebook vector:

[0094]

[0095] wherein z ij is the index of the nearest neighbor codebook vector, and c k represents the k-th vector in the pre-trained codebook; therefore the corresponding discrete vector is obtained.

[0096]

[0097] Step 3: Hybrid token construction. For the j-th feature vector f ij in the i-th scale residual feature map, the similarity of all codebook vectors is calculated, and the Top-K most similar codebook vectors are selected:

[0098]

[0099] Then sigmoid function is used to calculate weight normalization w on the obtained K similar codebook vectors k And get the final mixed token based on this weight And the mixed token residual feature map of the i-th scale:

[0100]

[0101] Step 4: Continuous token construction. For the feature map encoded by the visual encoder, first decouple the continuous feature map representation into discrete items and residual items:

[0102]

[0103] Where F represents the original continuous multi-scale feature map, e Sj The j-th residual vector of the residual feature map is represented as a fitting target modeled using a lightweight Diffusion MLP. Thus, it is possible to construct a continuous feature map by adding it back to the discrete token during training:

[0104]

[0105] Where is a lightweight Diffusion MLP based on the visual information summary (discrete residual feature map of the first scale) Predicted residual vector of the full feature map.

[0106] Step 5: Image-text feature fusion. Obtain the visual information summary of the image The text input has a large language model to extract the text information summary Both are implemented through Hadamard multiplication to achieve multi-modal fusion:

[0107]

[0108] Generate a multi-modal information summary x τ So as to guide the soft routing to make decisions in subsequent step 7.

[0109] Step 6: Complexity label division in the training phase. Define the image complexity score in the data set According to the maximum likelihood estimation, the mean μ and standard deviation σ are obtained, and based on this, the image complexity N k of the image Divide the image into three categories:

[0110] Low complexity:

[0111] Medium complexity:

[0112] High complexity:

[0113] Step 7: Build a soft router and supervise training. The fused multi-modal information summary x τ is input into a learnable soft router, which is trained to predict the complexity category through supervised learning, and the three token modeling paths are selected based on the soft router output results:

[0114]

[0115] where Discrete() represents discrete token; Hybrid() represents hybrid token; Continuous() represents continuous token;

[0116] Step 8: Unified training of Decoder. Freeze the parameters of the soft router, and in this phase, skip the step of selecting the quantized path through the soft router. Train the Decoder for the three types of tokens in parallel, so as to enhance the ability of the decoder to reconstruct the feature maps quantized by the three types of tokens, and the goal is to minimize the reconstructed image:

[0117] L Rec = MSE(N deocded ,N true ),

[0118] where N deocded represents the reconstructed image decoded by the decoder, and N true represents the input original image.

[0119] Step 9: Joint training of soft router and Decoder. Guide the soft router output according to the image complexity label, and the goal is the cross-entropy loss function L IC , which is optimized together with the image reconstruction loss during training:

[0120] L Router = λ1L Rec + λ2L IC ,

[0121] where λ1 and λ2 are the weights balancing the two loss functions; L Rec is the image reconstruction loss; L IC is the complexity classification loss.

[0122] Step 10: Autoregressive Transformer training. Freeze the decoder and soft router, and train the generation module - autoregressive Transformer. Different loss functions are used for the three types of tokens: cross-entropy loss function L dis, continuous token uses diffusion loss function L diff , mixed token uses soft cross-entropy loss function L hyb , so the total objective function is:

[0123] L AR = L dis + L diff + L hyb .

[0124] Step 11: Dynamic router in inference phase. After training, in the inference phase, only input text is used to obtain text feature summary and the next token is predicted by the Transformer using the quantization method of discrete token It is regarded as a visual feature summary, which is fused with the original text feature summary to obtain a multi-modal information summary:

[0125]

[0126] It will be input into the soft router, and the appropriate token modeling path will be selected according to the output to complete the subsequent prediction and thus complete the image generation.

[0127] Example 1: Image reconstruction task based on ImageNet and MJHQ-30K

[0128] This embodiment aims to verify the expression ability and fidelity effect of the three token modeling paths (discrete, mixed, and continuous) proposed in the present application in the image reconstruction scene. In this example, based on the ImageNet and MJHQ-30K data sets, the image reconstruction task is carried out to verify the quantization and reconstruction performance of different types of tokens.

[0129] Corresponding to the invention content, this example includes the following steps:

[0130] Step 1: Referring to Figure 2 , based on the ImageNet and MJHQ-30K validation set, randomly sample image order as input;

[0131] Step 2: Use the image encoder to encode the image into a multi-scale feature map

[0132] Step 3: Sample discrete, mixed, and continuous token quantization methods in turn to reconstruct the image.

[0133] This example is based on the PyTorch framework and simulated using distributed data parallel on 1 80G NVIDIA A100 GPU.

[0134] Verification Example 1: This verification example evaluates the three token reconstruction methods in Example 1 and compares them with VAR and other methods. The results are as follows:

[0135]

[0136] As can be seen from the table, the three token modeling methods of the present application are superior to the basic VAR model and Diffusion model, among which the continuous token path is the best in retaining details. Compared with the method in HART which also uses continuous tokens to construct feature maps, competitive results are also achieved, verifying the upper limit of DynamicVAR (the present application) in image generation tasks.

[0137] Example 2: Text-to-image generation task based on MJHQ-30K dataset

[0138] This example takes text-to-image generation as the background to verify the advantages of the present application in image-text semantic modeling and dynamic modeling path control, generation quality, and inference speed. The experimental data is based on the MJHQ-30K dataset, and the generated image resolution is 1024x1024; the input is natural language prompt text; the text is generated by Flan-T5 to generate a text summary and image summary fusion

[0139] The summary vector is input into the trained Router for complexity prediction, and the token path is selected according to the following rules, and finally the image generation is completed by the autoregressive Transformer.

[0140] This example is based on the MJHQ-30K dataset to carry out the text-to-image generation task, and verifies

[0141] The modeling effect and efficiency of the DynamicVAR method of the present application in high-resolution images. This example is based on the PyTorch framework and uses distributed data parallelism on 1 80G NVIDIA A100 GPUs for simulation.

[0142] Verification Example 2: Comparison with mainstream diffusion / AR models

[0143] This verification example compares DynamicVAR with various diffusion models (such as SDXL, PixArt-Σ) and mainstream autoregressive models (such as LlamaGen, HART) on the MJHQ-30K benchmark to evaluate the quality of generated images (FID, CLIP-Score) and inference efficiency (latency, throughput).

[0144] The results of the generation quality experiment are as follows:

[0145]

[0146]

[0147] The inference efficiency experimental results are as follows:

[0148]

[0149] It can be seen from the results that the Dynamic VAR of the application can realize 3-7 times of inference throughput rate improvement while ensuring that the CLIP-Score and FID are equivalent to those of the existing optimal model, and even 7.4x generation speed can be reached in the scene with low text complexity, greatly improving the generation efficiency of the text-to-image task.

[0150] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0151] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for adaptive image generation based on multi-modal routing, characterized in that, The method comprises the following steps: S1. Extracting multi-scale features of an input image to generate three types of tokens with increasing ability to retain continuous information, including: obtaining discrete tokens through vector quantization, hybrid tokens based on Top-K weighted aggregation, and continuous tokens modeled by a lightweight diffusion MLP; S2. Extracting visual and text modality information from the image and the corresponding text description respectively, and generating a multi-modal information summary after fusion; S3. Inputting the multi-modal information summary into a learnable soft router to dynamically select a token modeling path based on image complexity; S4. Optimizing the model using a three-stage training strategy: Stage one: training the decoder to reconstruct the feature maps of the three types of tokens; Stage two: jointly training the decoder and the soft router according to the image complexity label; Stage three: training the autoregressive Transformer to generate a multi-token fusion output; S5. In the inference stage, the trained soft router dynamically selects the token path to complete the image generation based on the input text description and the image information summary predicted by the autoregressive Transformer. 2.The method of claim 1, wherein, The step S1 comprises: The multi-scale residual image extracted from the input image is feature coded to construct a pre-training codebook with a size of K and a channel number of D for feature vector quantization; wherein the feature map F of the i-th feature scale i is represented as: wherein h i w i is the number of i-th scale feature vectors; for the j-th feature vector f ij The way of quantizing into three types of tokens includes: (1) Discrete token: vector quantization by pre-training codebook C, by calculating the highest similarity vector in the codebook to obtain its corresponding vector index z ij , f ij The quantized discrete token is represented as: (2) Hybrid token: based on Top-K weighted fusion, select the positions corresponding to the top K vectors with the highest similarity to the vector to be quantized from the pre-trained codebook C, denoted as: where Ω(f ij ) is a Top-K function selected Top-K nearest similar feature vector index set; w k is the weight calculated for K similar codebook vectors; c k is the kth codebook vector; (3) Continuous token: residual prediction is performed on the feature map constructed by the discrete token through a lightweight diffusion multi-layer perception model, so as to realize continuous modeling, represented as wherein is the residual feature map predicted by the lightweight diffusion MLP; F dis is the discrete representation generated after the image feature is vector quantized. 3.The method of claim 1, wherein, The step S2 comprises: Extract features from the input image and the corresponding text respectively. In the training process, the image uses a VQ-GAN-based encoder to extract discrete visual features, while in the inference process, the visual features are directly obtained after the output of the Transformer and are scaled. The text uses a pre-trained language model to extract text features; Perform element-wise Hadamard multiplication on the extracted image features and text features to achieve semantic alignment and fusion; The fused result is mapped to a unified dimension multi-modal information summary by a linear layer.

4. The adaptive image generation method based on multi-modal routing according to claim 1, characterized in that, The soft router in step S3 comprises: (1) Input layer: receives the multi-modal summary vector as the input feature of the router; (2) Multi-layer perceptron structure: contains multiple fully connected layers and ReLU activation function, used for nonlinear transformation of the summary vector; (3) Output layer: outputs a three-dimensional weight vector representing the selection probability of each token path; (4) Discrimination criterion: uses the cross-entropy loss function, and uses the image complexity label provided in the training stage as a supervision signal to guide the soft router to learn the optimal token modeling path based on the text description for images of different complexity.

5. The adaptive image generation method based on multi-modal routing according to claim 1, characterized in that, The three-stage training strategy in step S4 comprises: (1) Stage one: token generation stage, freeze the visual encoder and the pre-trained codebook, train the visual decoder to learn the reconstruction ability of discrete tokens, hybrid tokens and continuous tokens respectively, and use the reconstruction loss; (2) Stage two: router training stage, freeze the visual encoder and the pre-trained codebook, unfreeze the visual decoder, and use the cross-entropy loss with the image complexity label as supervision to train the router to generate token selection probability distribution through supervised learning; (3) Phase three: unified autoregressive generation phase, training the Transformer autoregressive decoder on the basis of freezing the token path, and the loss function is the autoregressive generation loss of the multi-token fusion output, including the negative log-likelihood loss and the text-image consistency constraint loss. 6.The method of claim 1, wherein, The step S5 comprises: In the inference stage, input the text description of the generated image, and obtain the text feature through the large language model; Input the text feature into the autoregressive model to perform first-step discrete prediction and output discrete visual residual feature maps; The visual residual feature maps and the text feature are fused in multiple modalities to enter the soft router dynamic selection token path; Based on all the predicted tokens, a complete feature map is constructed, which is decoded into a generated image through a visual decoder.

7. A multi-modal routing based adaptive image generation system, characterized in that, Comprise: A multi-scale feature extraction module is used to extract multi-scale features of an input image, and generate three types of tokens with increasing ability to retain continuous information, including: discrete tokens obtained by vector quantization method, mixed tokens based on Top-K weighted aggregation, and continuous tokens modeled by a lightweight diffusion MLP; A multi-modal fusion module is used to extract visual and text modal information from the image and corresponding text description respectively, and generate a multi-modal information summary after fusion; A soft router module is used to input the multi-modal information summary into a learnable soft router, and dynamically select a token modeling path based on image complexity; A training module adopts a three-stage training strategy to optimize the model: phase one: training the decoder to reconstruct the feature maps of the three types of tokens; phase two: jointly training the decoder and the soft router according to the image complexity label; phase three: training the autoregressive Transformer to generate multi-token fusion output; A generation module is used to dynamically select a token path by the trained soft router to complete image generation according to the input text description and the image information summary predicted by the autoregressive Transformer in the inference stage.

Images