An information processing method based on hybrid diffusion expert and adaptive length mechanism

Abstract

This disclosure provides an information processing method based on hybrid diffusion experts and an adaptive length mechanism. For multi-task scenarios involving both text and images, a hybrid diffusion system with modal decoupling but shared backbone is employed to decouple dual-expert diffusion and achieve a shared attention backbone for the input text and image tasks. A structured attention mask is implemented through a bidirectional attention mechanism within the modality, enabling block-based inference with adaptive length augmentation at the data layer. During the inference phase, the final output consists of the text and images required for the task. By constructing a hybrid diffusion framework, this approach effectively solves the problems of objective function mismatch and gradient interference during heterogeneous modal joint training, significantly improving the stability of model training and multi-task collaborative performance. Simultaneously, it avoids the repeated calculation of prefix information during iterative denoising, greatly reducing inference latency and computational overhead while ensuring generation quality.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically, to an information processing method based on a hybrid diffusion expert and an adaptive length mechanism. Background Technology

[0002] With the rapid development of deep learning technology, multimodal understanding and generation has become one of the core research directions in the field of artificial intelligence. Multimodal Large Language Models (MLLMs) aim to process information from multiple modalities, such as text and images, simultaneously, achieving cross-modal understanding and reasoning capabilities. In recent years, multimodal methods based on diffusion models have attracted widespread attention, among which Diffusion Large Language Models (dLLMs) have demonstrated unique advantages as a powerful alternative to traditional autoregressive language models. DLLMs are based on Masked Diffusion Models (MDMs), achieving text generation through iterative parallel denoising of masked token sequences. Compared to autoregressive models, diffusion language models possess bidirectional context modeling capabilities and inference parallelism, sparking significant interest from academia and industry. Large-scale diffusion language models, such as LLaDA and Dream, have demonstrated performance comparable to traditional autoregressive models.

[0003] However, existing technologies still face significant challenges when extending diffusion models to unified multimodal scenarios. Due to the inherent heterogeneity between the discrete features of text and the continuous features of images in terms of state space and noise destruction processes, direct joint training often leads to objective function mismatch and gradient interference, making it difficult for the model to simultaneously achieve high-precision text understanding and high-fidelity image generation. Furthermore, traditional masked diffusion models are limited by a pre-defined fixed sequence length and lack a dynamic termination mechanism like autoregressive models, making them inflexible for open-ended variable-length generation tasks. Moreover, their inference process generally relies on global attention mechanisms, requiring repeated calculations of the fixed input image and prompts at each denoising step, resulting in significant computational redundancy and inference delays, severely limiting the model's efficiency in practical applications.

[0004] Existing technologies mainly construct a unified multimodal architecture through autoregressive models or diffusion models. Among them, autoregressive models discretize images and concatenate them with text sequences for prediction; while diffusion model schemes are mainly divided into two categories: one is the pure mask diffusion model, which attempts to convert images into discrete tokens and use a mask prediction mechanism in conjunction with text; the other is the hybrid training model, which attempts to directly train the mask diffusion of discrete text and the latent space diffusion of continuous images in a single large model.

[0005] However, all of the above solutions have significant drawbacks. For pure masked diffusion models, forcibly discretizing continuous image signals (such as through VQ-VAE) leads to lossy compression of visual information, making it difficult for the generated image quality to match mainstream continuous diffusion models (such as Stable Diffusion). For direct hybrid training models, the inherent heterogeneity between discrete text and continuous images in terms of state space and noise destruction processes causes severe objective function mismatch and gradient interference, resulting in unstable training and suboptimal performance. In addition, existing diffusion architectures generally have the limitation of a fixed output length, lacking the flexibility to handle variable-length sequences, and the computational redundancy brought by the global attention mechanism during inference severely restricts the running efficiency. Summary of the Invention

[0006] The purpose of this disclosure is to provide an information processing method based on hybrid diffusion experts and adaptive length mechanism, which aims to solve the problems of training conflict and fixed-length generation limiting inference efficiency caused by heterogeneous fusion of discrete text and continuous images in the prior art.

[0007] In general, this paper presents an information processing method based on hybrid diffusion experts and an adaptive length mechanism, targeting multi-task scenarios involving both text and images. For input text and image tasks, a hybrid diffusion system with modal decoupling but shared backbone is employed for dual-expert diffusion decoupling and to achieve a shared attention backbone. A structured attention mask is implemented through the design of an intramodal bidirectional attention mechanism, enabling block-based inference with adaptive length augmentation at the data layer. During the inference phase, a forward propagation is first performed on the input image and prompt words to calculate and cache their key-value features. Then, a fixed-length full-mask block is initialized as the initial generation unit. After entering the iterative denoising loop, the current... The mask block undergoes multi-step prediction, outputting the token probability distribution at each position in each step. Only when the confidence of the predicted token at a certain position exceeds a preset threshold is it updated to a deterministic token; otherwise, the mask state is retained to proceed to the next step. During the denoising process, it continuously detects whether a high-confidence terminator [EOS] has been generated. If [EOS] is detected, the generation process is terminated, and subsequent content is truncated based on that position to output the final result. If [EOS] still does not appear after the current block is denoised, the model adds the valid text generated in the current block to the cache and automatically appends a new full mask block to continue the next round of generation. Finally, the text and image required by the task are output. During the training phase, training is performed using a data center-style adaptive length augmentation training strategy.

[0008] The specific implementation of the hybrid diffusion system with decoupled modes but shared backbone is to decompose the multimodal task into two parallel expert modules: understanding experts and generating experts. The input to the understanding expert includes a text input branch and an image input branch. The text input branch first processes the input text through a tokenizer, then through embedding and positional encoding to form an input text vector. The image input branch first extracts high-dimensional features through a visual understanding encoder, then maps them to the embedding space of the language model through an MLP projection layer to form visual semantic tokens. These visual tokens are concatenated with text tokens to form a... The masked token sequence is fed as a joint input sequence into a discrete diffusion language model for text understanding and image understanding. Then, it passes through an intramodal bidirectional attention mechanism shared with the generative expert to output a text response with the vocabulary distribution at each position. The understanding expert is trained using a masking diffusion mechanism, which randomly masks some tokens according to time steps during the forward pass and learns to recover these masked tokens during the backward pass, thereby minimizing the cross-entropy loss; the cross-entropy loss function is: The input to the generative expert is the image generation branch. After the image passes through the visual generative encoder, it is mapped to the embedding space of the language model through the MLP projection layer to form a visual semantic token. For the visual semantic token, the image is generated by a variational autoencoder and a generative network based on the DiT architecture, and the generated image is output by the intramodal bidirectional attention mechanism shared with the understanding expert. The generative expert is trained using a flow matching or continuous diffusion mechanism. The intramodal bidirectional attention mechanism is a Transformer-based self-attention backbone network.

[0009] The specific implementation of the intramodal bidirectional attention mechanism is as follows: a special attention mask matrix is ​​designed to divide the input sequence logic into different modal regions such as image blocks, cue word blocks, and generation blocks. A full attention mechanism is implemented within each modal block to ensure that tokens can perceive each other in both directions to capture rich local semantic dependencies. A strict causal attention mechanism is implemented between modal blocks, that is, the subsequent generation block can pay attention to the preceding image block and cue word block, but the preceding block cannot see the subsequent block.

[0010] The data center-style adaptive length enhancement training strategy involves randomly perturbing the target response sequence of the training samples during the data preprocessing stage. This includes two aspects: first, random expansion, which appends a random number of special terminators [EOS] to the original response sequence with a certain probability to train the model to identify the endpoint of content generation; and second, random truncation, which truncates the original response with a certain probability, retaining only the preceding fragment as the training target. Simultaneously, attention isolation between samples is strictly implemented during multi-sample bundled training to ensure that adaptive length adjustment does not cause interference between different samples.

[0011] The technical effects to be achieved by the embodiments of the present invention are as follows: This invention constructs a hybrid diffusion framework, decoupling discrete text understanding and continuous image generation into independent experts who share an attention backbone. This effectively solves the problems of objective function mismatch and gradient interference during heterogeneous modality joint training, significantly improving the stability of model training and multi-task collaborative performance. Simultaneously, it utilizes an intra-modal bidirectional attention mechanism to achieve key-value cache reuse for fixed input conditions, avoiding repeated calculations of prefix information during iterative denoising and significantly reducing inference latency and computational overhead while ensuring generation quality. Furthermore, combined with an adaptive length enhancement strategy, it endows the mask diffusion model with the ability to handle open-ended variable-length sequences, achieving unified multimodal processing with high training stability, high inference efficiency, and flexible variable-length generation capabilities without modifying the model architecture. Attached Figure Description

[0012] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.

[0013] Figure 1 This is a schematic diagram illustrating the architecture of an information processing method based on a hybrid diffusion expert and an adaptive length mechanism according to an embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating a hybrid diffusion system architecture with modal decoupling but shared backbone according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram illustrating an intramodal bidirectional attention architecture according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating the generation result according to an embodiment of the present disclosure. Detailed Implementation

[0014] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0015] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0016] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0017] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0018] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0019] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0020] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0021] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0022] Figure 1 This is a schematic diagram illustrating an information processing method based on a hybrid diffusion expert and adaptive length mechanism according to an embodiment of the present disclosure.

[0023] To achieve the aforementioned objectives, the present invention employs the following technical framework: Figure 1 As shown.

[0024] The core innovation of this invention lies in proposing a "Mixture of Diffusion (MoD)" framework. This framework creatively decouples multimodal tasks into a "understanding expert" based on discrete mask diffusion and a "generative expert" based on continuous latent space diffusion. Although they share an attention backbone, they operate within their respective optimal state spaces, effectively solving the gradient interference and optimization conflict problems during heterogeneous modality joint training. Building upon this, this invention further designs an "intramodal bidirectional attention" mechanism and an "adaptive length enhancement" strategy: the former implements causal masks between modal blocks, enabling the reuse of key-value caches (KV Cache) for fixed input conditions (such as images and prompts), avoiding redundant computation of global attention in each denoising step, and significantly improving inference efficiency; the latter, through random truncation and extended training at the data level, endows the mask diffusion model with the ability to handle open-ended variable-length sequence generation, breaking through the limitations of traditional fixed-length generation.

[0025] This invention provides a hybrid diffusion modeling method for unifying multimodal understanding and generation, targeting multi-task scenarios involving both text and images (e.g., visual question answering, chart / document understanding, text-to-image generation, etc.). It achieves diffusion-based understanding of discrete text sequences and diffusion-based generation of continuous visual latent variables within the same model system. The core of this invention lies in addressing the fundamental differences between text and images in their state space and diffusion dynamics by employing a holistic scheme of "dual-expert diffusion decoupling + shared attention backbone + structured attention mask + adaptive length augmentation of the data layer + block-based inference," thereby balancing training stability, cross-modal interaction capabilities, and inference efficiency.

[0026] Mixture of Diffusion (MoD) dual-expert framework: This invention constructs a hybrid diffusion system with modal decoupling but shared backbone, the architecture of which is as follows: Figure 2 As shown.

[0027] This system decomposes the multimodal task into two parallel expert modules: an understanding expert and a generative expert. The understanding expert is specifically responsible for processing discrete modal data, including text input, text generation, and semantic feature extraction from images. Its specific structure includes a high-precision visual encoder (e.g., SigLIP), a multilayer perceptron (MLP) projection layer, and a discrete diffusion language model.

[0028] Discrete diffusion language models typically use a Transformer as their backbone (with input being a vector). This discrete diffusion model (DDP) outputs the word distribution at each position of the masked token sequence. Unlike most visual language models that use an autoregressive paradigm (generating tokens sequentially from left to right), this DDP model uses multi-step iterative denoising, predicting and filling the masked positions in parallel at each step, thus leveraging bidirectional context to achieve text generation and multimodal understanding. Its basic idea is: during the forward diffusion process, the real text sequence is... By time step Gradual "corrosion" is often achieved through probability. Replace part of the token with a special one. (mask) symbol, to obtain noisy sequence During the reverse generation process, the model learns to make conditional predictions about the masked positions based on the context, and gradually recovers the complete text from the "full mask" or high-noise state through multiple rounds of iterative denoising.

[0029] The formula for the forward diffusion process is as follows: .

[0030] The formula for the reverse generation process is as follows: .

[0031] In operation, the input image first undergoes a visual encoder to extract high-dimensional features, which are then mapped to the embedding space of the language model via an MLP projection layer, forming visual semantic tokens. Simultaneously, the text input is transformed into an integer index sequence by a tokenizer, mapped to a dense vector through a lookup table, and injected with positional encoding and a terminator. Finally, the visual tokens and text tokens are concatenated and fed into the model as a joint input sequence. The understanding expert employs a masked diffusion mechanism for training, randomly masking some tokens at time steps during the forward pass and learning to recover these masked tokens during the backward pass, thereby minimizing the cross-entropy loss. The cross-entropy loss formula is as follows: Correspondingly, the generative expert is specifically responsible for high-fidelity generation of continuous modal data. Its core components include a variational autoencoder (VAE) and a generative network based on a DiT architecture. The generative expert's conditional tokens can be directly derived from the unified semantic space of the understanding branch (visual encoder + MLP projection + discrete diffusion language model), aligning "understanding-generation" on the same conditional representation and improving the ability to follow complex instructions and bind fine-grained attributes.

[0032] Generative experts use pre-trained VAEs to compress images into a low-dimensional latent space to obtain continuous latent variables, and then train them using flow matching or continuous diffusion mechanisms to learn how to recover sharp image latent variables from Gaussian noise. The specific process is as follows: First, a pre-trained and frozen VAE encoder is used to train the image... Compressed into continuous latent variables Subsequently, noise was sampled from a standard Gaussian distribution. and from Uniform sampling time step Construct intermediate states based on the selected continuous diffusion / flow matching path. (e.g., linear interpolation path), will Time embedding and condition information Input DiT-based generative network Make predictions. The training objective is to make the network output consistent with the target direction (or noise / velocity field), and the mean squared error loss is used for optimization. Backpropagation is used to update only the generated experts and their time / conditional parameters.

[0033] The specific training loss formula for generating experts is as follows: Although the two experts deal with vastly different data types, this invention designs them to share the same Transformer-based self-attention backbone network. This means that the model is unified in its deep feature interaction and inference capabilities, differing only in the input / output layers and loss function calculations. This design preserves the deep interaction capabilities across modalities while effectively avoiding gradient interference and optimization conflicts arising from dense joint training of heterogeneous modalities.

[0034] Structured attention: intramodal bidirectional, intermodal causality: This invention proposes and implements an intra-modality bidirectional attention mechanism, the specific mechanism of which is as follows: Figure 3 As shown.

[0035] This mechanism employs a specially designed attention mask matrix (the specific form of which is shown in...). Figure 3 As shown, the input sequence is logically divided into different modal regions such as image blocks, cue word blocks, and generation blocks. Specifically, this invention implements a full attention mechanism within each modal block to ensure bidirectional perception between tokens to capture rich local semantic dependencies; while a strict causal attention mechanism is implemented between modal blocks, meaning that subsequent generation blocks can pay attention to preceding image blocks and cue word blocks, but preceding blocks cannot see subsequent blocks. A significant advantage of this design is the reuse of the key-value cache (KV Cache). During multi-step denoising inference, since the input image and cue words are fixed conditional information, the model only needs to calculate the KV features of these prefixes once at the initial stage of inference and store them in the cache. In each subsequent denoising time step, the model only needs to calculate the newly added KV features of the current generation block and concatenate them with the fixed prefixes in the cache. This mechanism significantly eliminates the repeated calculation of long sequence prefixes, thereby significantly improving inference throughput while ensuring generation quality.

[0036] Data layer adaptive length augmentation and inference methods: To address the limitation of traditional masked diffusion models by fixed sequence length, this invention introduces a data center-based adaptive length enhancement training strategy. This strategy randomly perturbs the target response sequence of the training samples during the data preprocessing stage. The specific implementation includes two aspects: first, random expansion, which appends a random number of special terminators ([EOS]) to the original response sequence with a certain probability, thereby training the model to identify the endpoint of content generation and enabling it to proactively stop when the content is complete; second, random truncation, which truncates the original response with a certain probability, retaining only the preceding fragment as the training target, thereby simulating intermediate states in the inference process and forcing the model to learn the ability to continue writing content using local context. Simultaneously, to ensure training stability, this invention strictly implements attention isolation between samples during multi-sample bundled training, ensuring that adaptive length adjustment does not cause interference between different samples.

[0037] During the inference phase, the model first performs a forward propagation on the input image and prompt words, calculates and caches their key-value features, and then initializes a fixed-length full mask block as the initial generation unit. After entering the iterative denoising loop, the model performs multi-step predictions on the current mask block and outputs the token probability distribution for each position at each step. A confidence threshold needs to be preset; only when the confidence of a predicted token at a certain position exceeds this threshold is it updated to a deterministic token; otherwise, the mask state is retained to proceed to the next step. During denoising, the model continuously checks whether a high-confidence terminator [EOS] has been generated. Once [EOS] is detected, the generation process immediately terminates, and subsequent content is truncated based on that position to output the final result. If [EOS] still does not appear after the current block is denoised, the model adds the valid text generated for the current block to the cache and automatically appends a new full mask block to continue the next round of generation.

[0038] Some of the generated results are shown below. Figure 4 As shown.

[0039] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. An information processing method based on hybrid diffusion experts and adaptive length mechanism, characterized in that, For multi-task scenarios involving both text and images, a hybrid diffusion system with modal decoupling but shared backbone is adopted for input text and image tasks. This system achieves dual-expert diffusion decoupling and a shared attention backbone. A structured attention mask is implemented through an intra-modal bidirectional attention mechanism, enabling block-based inference with adaptive length augmentation at the data layer. During the inference phase, a forward propagation is first performed on the input image and prompt words to calculate and cache their key-value features. Then, a fixed-length full mask block is initialized as the starting generation unit. After entering the iterative denoising loop, the current mask block is predicted in multiple steps, and the token probability distribution at each position is output at each step. Only when the confidence of the predicted token at a certain position exceeds the preset threshold is it updated to a deterministic token; otherwise, the mask state is retained and the next step is entered. During the denoising process, it is detected in real time whether a high-confidence terminator [EOS] has been generated. If [EOS] is detected, the generation process is terminated, and the subsequent content is truncated according to the position and the final result is output. If [EOS] still does not appear after the current block is denoised, the model adds the valid text generated for the current block to the cache and automatically appends a new full-mask block to continue the next round of generation; finally, the output is the text and image required by the task. During the training phase, training is performed using a data center-style adaptive length augmentation training strategy.

2. The information processing method based on hybrid diffusion experts and adaptive length mechanism as described in claim 1, characterized in that, The specific implementation of the hybrid diffusion system with decoupled modes but shared backbone is to decompose the multimodal task into two parallel expert modules: understanding experts and generating experts. The input from the understanding expert includes text input branches and image input branches; The text input branch first processes the input text through a tokenizer, then through embedding and positional encoding to form an input text vector; the image input branch first extracts high-dimensional features through a visual understanding encoder, then maps them to the embedding space of a language model through an MLP projection layer to form visual semantic tokens; these visual tokens are concatenated with text tokens to form a... The masked token sequence is fed as a joint input sequence into a discrete diffusion language model for text understanding and image understanding. Then, it passes through an intramodal bidirectional attention mechanism shared with the generative expert to output a text response with the vocabulary distribution at each position. The understanding expert is trained using a masking diffusion mechanism, which randomly masks some tokens according to time steps during the forward pass and learns to recover these masked tokens during the backward pass, thereby minimizing the cross-entropy loss; the cross-entropy loss function is: The input to the generative expert is the image generation branch. After the image passes through the visual generative encoder, it is mapped to the embedding space of the language model through the MLP projection layer to form a visual semantic token. For visual semantic tokens, the image is generated by a variational autoencoder and a generative network based on a DiT architecture, and then outputs the generated image through an intramodal bidirectional attention mechanism shared with understanding experts. The generative experts are trained using either flow matching or continuous diffusion mechanisms; The intramodal bidirectional attention mechanism is a Transformer-based self-attention backbone network.

3. The information processing method based on hybrid diffusion experts and adaptive length mechanism as described in claim 2, characterized in that, The specific implementation method of the discrete diffusion language model is as follows: through multi-step iterative denoising, the masked position is predicted and filled in parallel at each step, thereby realizing text generation and multimodal understanding by utilizing bidirectional context; During the forward diffusion process, the real text sequence By time step Gradual erosion, specifically, in terms of probability Replace some tokens with special ones Symbols are used to obtain noisy sequences. ; During the reverse generation process, the model learns to make conditional predictions about the masked position based on the context, and gradually recovers the complete text from the full mask or high-noise state through multiple rounds of iterative denoising. The formula for the forward diffusion process is as follows: The formula for the reverse generation process is as follows: 。 4. The information processing method based on hybrid diffusion experts and adaptive length mechanism as described in claim 2, characterized in that, The specific method for training the expert generation expert's flow matching or continuous diffusion mechanism is as follows: First, the training images are processed using a pre-trained and frozen VAE encoder. Compressed into continuous latent variables Subsequently, noise was sampled from a standard Gaussian distribution. and from Uniform sampling time step Construct intermediate states based on the selected continuous diffusion / flow matching path. ,Will Time embedding and condition information Input DiT-based generative network Make predictions. The training objective is to make the network output consistent with the target direction. Mean squared error loss is used for optimization, and backpropagation is used to update only the generated experts and their time / conditional parameters. The specific training loss formula for generating experts is as follows: 。 5. The information processing method based on hybrid diffusion experts and adaptive length mechanism as described in claim 1, wherein the specific implementation of the intramodal bidirectional attention mechanism is as follows: a special attention mask matrix is ​​designed to logically divide the input sequence into different modal regions such as image blocks, cue word blocks, and generation blocks. A full attention mechanism is implemented within each modal block to ensure that tokens can perceive each other bidirectionally to capture rich local semantic dependencies. A strict causal attention mechanism is implemented between modal blocks, that is, the subsequent generation block can pay attention to the preceding image block and cue word block, but the preceding block cannot see the subsequent block.

6. The information processing method based on hybrid diffusion experts and adaptive length mechanism as described in claim 1, characterized in that, The data center-style adaptive length enhancement training strategy involves randomly perturbing the target response sequence of the training samples during the data preprocessing stage. This includes two aspects: first, random expansion, which appends a random number of special terminators [EOS] to the original response sequence with a certain probability to train the model to identify the endpoint of content generation; and second, random truncation, which truncates the original response with a certain probability, retaining only the preceding fragment as the training target. Simultaneously, attention isolation between samples is strictly implemented during multi-sample bundled training to ensure that adaptive length adjustment does not cause interference between different samples.

Images