Input reordering and cache reuse method for multimodal decision models

Abstract

The present invention discloses an input rearrangement and cache reuse method for a multimodal decision model, comprising: placing a text input sequence before an image input sequence for sequence splicing to generate a text-first spliced ​​input sequence; during initial reasoning, using a multi-layer Transformer encoder to calculate and cache the text Key-Value vector corresponding to the text input sequence to a static cache area; during subsequent reasoning, performing inter-frame semantic consistency detection on the newly input current image frame, and performing on-demand or local updates to the Key-Value vector of the image portion based on the detection results; splicing and assembling the cached text Key-Value vector with the updated image Key-Value vector to complete the final decision. The present invention effectively avoids redundant calculations of unchanged instructions, realizes refined and intelligent reuse of cache, and improves the reasoning speed and throughput of the multimodal decision model in dynamic tasks.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computers, and in particular to an input rearrangement and cache multiplexing method for a multimodal decision model. Background Art

[0002] With the evolution of artificial intelligence technology, large multimodal decision-making models centered around the Transformer architecture have shown tremendous potential for application in areas such as intelligent agent games, human-computer interaction, and autonomous control, which require deep environmental understanding and real-time response. The successful execution of such tasks relies heavily on the model's ability to efficiently fuse and reason with high-frequency, dynamic visual perception information and relatively stable textual instruction information. Therefore, researching and developing technical solutions that can optimize the reasoning efficiency of such models in resource-constrained or latency-sensitive scenarios is of vital practical significance and holds broad market potential for promoting the engineering implementation and widespread application of advanced artificial intelligence systems.

[0003] Currently, the mainstream technical approach to solving such multimodal decision-making tasks is to employ end-to-end deep learning models. Specifically, existing technologies typically preprocess and serialize information from different modal sources. For example, text instructions are segmented into text token sequences, and image observations are segmented into image patches and projected into image token sequences. These token sequences are then concatenated to form a unified input sequence, which is then fed into a large, pre-trained Transformer encoder for processing. In common practice, to emphasize the dominance of visual perception, the input sequence is typically organized with image tokens first and text tokens last. Furthermore, to accelerate inference, existing technologies have explored various model optimization techniques. Key-value caching (KV-Cache) is widely used in tasks such as autoregressive language generation. This mechanism caches the key and value vectors of previously computed tokens during the Transformer's attention computation, allowing them to be reused in subsequent token generation, thus avoiding duplicate computation.

[0004] However, despite significant progress in model capabilities, existing solutions have exposed several underlying technical challenges in practical deployment, particularly in typical game-playing decision-making scenarios where textual instructions are stable over long periods of time while visual observations are frequently updated. First, existing technologies generally waste significant computing resources, creating efficiency bottlenecks. Second, conventional KV-Cache mechanisms lack the ability to perceive changes in image content at a fine-grained level, exacerbating resource waste. Summary of the Invention

[0005] The purpose of the invention is to provide an input reordering and cache reuse method for a multimodal decision model, in order to solve at least one technical problem existing in the prior art.

[0006] A technical solution for input reordering and cache reuse in a multimodal decision model, including:

[0007] Receive a text input sequence and an image input sequence, and perform sequence splicing in a text-first order to generate a spliced ​​input sequence;

[0008] The preset learnable position bias is fused with the original text and image position embeddings respectively to construct a reconstructed position embedding sequence that matches the order of the concatenated input sequence.

[0009] Combine the concatenated input sequence with the reconstructed position embedding sequence to form the total input representation of the model;

[0010] A multi-layer Transformer encoder is used to process the model's total input representation, extract and generate text key-value vectors from its text part, and store them in a static buffer for subsequent reasoning reuse. At the same time, the final attention output is generated based on the model's total input representation;

[0011] The final attention output is passed to the decision head to generate the final decision result.

[0012] Beneficial effect: The present invention effectively avoids redundant calculations of unchanged instructions, realizes refined and intelligent reuse of cache, and improves the reasoning speed and throughput of multimodal decision models in dynamic tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of the steps of an input reordering and cache reuse method for a multimodal decision model provided in an embodiment of the present application.

[0014] Figure 2 A flowchart of the steps for using a multi-layer Transformer encoder to process the reused text Key-Value vectors in subsequent reasoning provided by an embodiment of the present application.

[0015] Figure 3 A flowchart of the steps for generating an update decision signal provided in an embodiment of the present application.

[0016] Figure 4 A flowchart of the steps for generating a new image Key-Value vector provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0019] Research has found that the commonly used input structure of image first and text last, adopted by existing technologies, forces the model to re-encode the unchanged text instruction tokens at each inference time step. This results in significant computational resource waste, directly limiting the system's inference throughput and response speed, and creating an efficiency bottleneck. Conventional key-value cache (KV-Cache) mechanisms also exhibit structural inadequacies when dealing with such mixed-stability inputs. Firstly, this mechanism was originally designed to serve autoregressive tasks with unidirectional sequence growth, making it difficult to directly apply to input scenarios where the text segment is fixed and the image segment is completely replaced. Secondly, even when caching is designed for the image portion, existing methods lack the ability to perceive changes in the image content in a fine-grained manner. When only minor local changes occur in the image (e.g., only a single object in the scene moves), existing mechanisms still require a complete recalculation and refresh of the KV cache for the entire frame, failing to achieve deeper computational reuse and resulting in low cache utilization. This crude update strategy exacerbates resource waste, especially in scenes with gradual changes.

[0020] like Figure 1 As shown in FIG, a method for input rearrangement and cache reuse for a multimodal decision model is proposed, comprising the following steps:

[0021] A text input sequence and an image input sequence are received, and sequence splicing is performed on the text input sequence and the image input sequence in a text priority order to generate a spliced ​​input sequence.

[0022] In this embodiment, the text input sequence is recorded as I t , representing a sequence of tokens formed by tokenization of natural language text representing task instructions or descriptions. For example, the instruction "move to the red marked point on the map" may be processed into seven tokens [move, to, map, of, red, marked, point]. The tokenized text is then passed through the word embedding layer to generate a text vector encoding I t . Let the image input sequence be I v , refers to dividing the input image (such as the game screen) into multiple image patches (Patches), and then linearly projecting each image patch to form a token sequence. This is a standard operation in models such as Vision Transformer (ViT). Specifically, text priority refers to the order in which the text input sequence I is first placed when splicing two sequences. t Completely placed in the image input sequence I v Before. If I t =[I t 0 , I t 1 , I t 2 …I t L ], I v =[I v 0 , I v 1 , I v 2 …I v N ], then the input sequence I after splicing is [I t 0 ,...,I t L , I v 0 ,...,I v N ]. The text instructions in the game task usually remain unchanged throughout the interaction round, while the image observations are updated at a high frequency. t Placed at the front end of the sequence, it processes and caches the image input sequence I separately for subsequent steps t The information of creates a structural convenience, avoiding repeated feature encoding in each reasoning step and improving efficiency. The generated concatenated input sequence I serves as the direct object for the model to perform semantic understanding and position perception.

[0023] The preset learnable position bias is fused with the original text and image position embeddings respectively to construct a reconstructed position embedding sequence that matches the arrangement order of the concatenated input sequence.

[0024] In this embodiment, the original text position is embedded in P t and the original image position embedding P v It is a vector inherited from the pre-trained model that corresponds one-to-one to the token position. Since the input order has been changed before, directly using the original position embedding will cause the position information and token content to be misaligned. For example, although the image token is located in the second half of the spliced ​​input sequence I, its original position embedding P v It may still start from 0, which will seriously mislead the attention mechanism of the model. Therefore, a learnable position bias B is introduced. Specifically, by dividing the part of the bias B corresponding to the text and image modalities (respectively denoted as B t and B v ), and the original text position embedded P t and the original image position embedding P v Perform vector addition fusion to obtain the reconstructed position embedding: P' t = P t + B t and P' v = P v + B v . Among them B t and B v As a model parameter, it can be automatically learned through back propagation during training to optimally compensate for the position semantic offset caused by sequence rearrangement. The reconstructed position is embedded in P' t and P' v Splice according to the same text priority order as the spliced ​​input sequence I to form the reconstructed position embedding sequence P'= [P' t ;P' v ]. The reconstructed position embedding sequence P' provides accurate sequence order information that fully matches the current input structure for the subsequent Transformer encoding.

[0025] The concatenated input sequence and the reconstructed position embedding sequence are combined to form the total input representation of the model.

[0026] In this embodiment, content information (token word embeddings) and position information (reconstructed position embeddings) are integrated. Specifically, for each token in the concatenated input sequence I, its word embedding vector is added to the vector of the corresponding position in the reconstructed position embedding sequence P' to form the model's total input representation. This total input representation is a tensor that contains rich content and accurate temporal structure, and serves as the final input to the multi-layer Transformer encoder.

[0027] A multi-layer Transformer encoder is used to process the total input representation of the model, extract and generate text Key-Value vectors from its text part, and store them in a static cache for subsequent reasoning reuse. At the same time, the final attention output is generated based on the total input representation of the model.

[0028] In this embodiment, the multi-layer Transformer encoder is the core component for deep semantic understanding. When the total input representation of the model flows through the encoder: For the text part: the encoder specifically processes the image input sequence I t For the corresponding input part, calculate its key vector and value vector at each layer, which are respectively denoted as K t and V t This group (K t , V t ) vector pair (i.e., text Key-Value vector cache, recorded as KV text ) are extracted and stored in a dedicated static cache. The static cache is a memory area that remains unchanged between multiple inference time steps. text This represents the model's deep understanding of the unchanging text instructions and will be directly invoked during inference on subsequent frames, eliminating repeated computation. For a complete sequence: The encoder simultaneously processes the entire model input representation, performing global computation using the self-attention mechanism to ultimately generate the final attention output. This output incorporates all information from the text instructions and the current image observation.

[0029] The final attention output is passed to the decision head to generate the final decision result.

[0030] In this embodiment, the decision head is typically one or more linear layers or small neural networks that receive the highly condensed feature representation of the final attention output and map it to a specific action space to generate a decision result. For example, it outputs a discrete action representing upward movement or a continuous value representing the aiming coordinate (x, y). Specifically, assume a very simple scenario: a text instruction I t There are 2 tokens [T1, T2], image I vIt is divided into 4 tokens [P1, P2, P3, P4] and sequence splicing is performed. After splicing, the input sequence I is [T1, T2, P1, P2, P3, P4]. Position reconstruction is performed, assuming that the original absolute position embedding P is [0, 1, 2, 3, 4, 5]. Since the actual position of the image token P1 in the spliced ​​sequence is index 2, its original position embedding may come from index 0, which is misaligned. By introducing the learned bias B, a new position embedding P' is generated so that the position embedding of P1 can correctly reflect its position in the sequence I. Encoding and caching are performed, and the model processes the combined input of I and P': at time t=0: calculate and cache the KV corresponding to [T1, T2]. text ; At the same time, calculate the KV corresponding to [P1, P2, P3, P4] image_0 Based on [KV text ;KV image_0 ] performs attention calculation and generates a decision. At t=1: the text remains unchanged, and the image becomes [P1', P2', P3', P4']; at this time, there is no need to recalculate KV text , read directly from the static cache, the model only needs to calculate the new KV image_1 . Subsequent reasoning: cached KV text With the newly calculated KV image_1 Spliced ​​into [KV text ;KV image_1 ], and then perform attention calculation. Through the above steps, the problem of repeated calculation of static information and low reasoning efficiency caused by unreasonable input structure in the existing technology is solved. It provides infrastructure support for subsequent intelligent caching and advanced optimization. The calculation overhead of unchanging text information is reduced from once per time step to only once in the entire task cycle, which improves the reasoning efficiency and response speed in dynamic game scenarios. Among them, KV image_0 The Key and Value tensors obtained by forward calculation of the initial image token at time step t=0.

[0031] This embodiment creates the prerequisite for the separation of static and dynamic information at the input structure level by adjusting the text input sequence to form a spliced ​​input sequence with text priority before the image input sequence. Based on this structure, the model can deeply encode the unchanged part of the text input sequence at one time during the first inference, and store its calculation results, that is, the text Key-Value vector cache, in the static cache area. In the subsequent hundreds or thousands of inference time steps, the system can directly reuse the cache, bypassing the repeated encoding process of the text. This one-time calculation and persistent reuse mechanism directly reduces the text processing overhead, which is originally proportional to the number of inferences, to a constant. In gaming or human-computer interaction scenarios where text instructions remain unchanged for a long time, redundant calculations are eliminated, allowing computing resources to be concentrated on processing high-frequency changing image information, achieving a reduction in inference latency and an increase in throughput at the system level, and improving the performance of the model in real-time dynamic tasks.

[0032] like Figure 2 As shown, according to one aspect of the present application, when the text Key-Value vector is reused in subsequent reasoning, the process of processing using a multi-layer Transformer encoder further includes:

[0033] Performing inter-frame semantic consistency detection on the current image frame and the previous image frame to generate an update decision signal;

[0034] In response to the update decision signal indicating that an update is required, the current image frame is encoded to generate a new image Key-Value vector;

[0035] In response to the update decision signal indicating that no update is required, directly reusing the cached image Key-Value vector corresponding to the previous image frame;

[0036] Concatenate the text Key-Value vector with the new or reused image Key-Value vector to obtain the complete Key-Value set of the current frame;

[0037] The final attention is calculated using the complete Key-Value set of the current frame to generate a decision result.

[0038] In this embodiment, the core idea of ​​inter-frame semantic consistency detection is to no longer unconditionally calculate the image Key-Value vector for each newly input current image frame, but to first compare it with the previous image frame. The current image frame refers to the image input received by the model at the current reasoning time step (for example, time t+1); the previous image frame refers to the image input processed by the model at the previous reasoning time step (time t); the update decision signal is a Boolean signal, and when its value is yes, it indicates that the image content has changed significantly and the KV cache needs to be updated; when its value is no, it indicates that the change is negligible and the old KV cache can be reused. Specifically, high-level semantic features are extracted for the current image frame and the previous image frame respectively, denoted as F cur and F prev , which can be obtained from the intermediate layer output of the model or through a lightweight feature extraction network. cur and F prev The cosine similarity between the two images is calculated and compared with a preset threshold. If the similarity is lower than the threshold, the decision signal is updated to yes, otherwise it is no.

[0039] This embodiment introduces a similarity detection mechanism to effectively filter out inference frames with essentially unchanged image content, avoiding unnecessary encoding operations for nearly duplicate images and enabling on-demand allocation of computing resources. This addresses the issue of wasted computing resources caused by performing full KV cache updates on images with minimal changes when processing consecutive image frames. The system can dynamically determine the actual degree of change in image content and perform on-demand updates accordingly, improving the system's operational efficiency in dynamic scenarios.

[0040] like Figure 3 As shown, according to one aspect of the present application, generating an update decision signal includes:

[0041] From the feature sequence of historical image frames, a sliding window is defined to obtain a historical feature set within the window;

[0042] Based on the historical feature set within the window, a single historical stability feature representing the recent stable semantics is calculated and generated;

[0043] Compare the semantic differences between the features of the current image frame and the historical stability features;

[0044] Generate an update decision signal based on the semantic difference.

[0045] In this embodiment, to address the problem of fixed judgment thresholds being unable to adapt to the varying rates of change in different scenarios (e.g., static and highly dynamic), the current frame is no longer compared with the isolated previous frame, but rather with the stable state over a recent period. Specifically, the system maintains a historical image frame feature sequence—a queue that stores image features from the past N frames (e.g., N=5). A sliding window operates on this sequence, extracting features from the most recent M frames (e.g., M=3) to form the historical feature set within the window. By averaging or weighted averaging all feature vectors in the historical feature set within the window, a more stable and noise-resistant historical stability feature is obtained. The current image frame feature is compared with this historical stability feature, and an updated decision signal is generated. This embodiment, by introducing a sliding window for temporal smoothing, makes the semantic consistency judgment more reliable, effectively avoiding misjudgments caused by single noisy frames or minor jitter, and improving decision stability. This also addresses the problem of detection instability caused by simple frame-by-frame comparisons, which are susceptible to noise or transient jitter.

[0046] According to one aspect of the present application, a process of generating an update decision signal based on a semantic difference includes comparing with a dynamic update threshold, wherein generating the dynamic update threshold includes:

[0047] Based on the historical sequence composed of semantic differences of a predetermined number of time steps in the past, the historical change variance is calculated and obtained;

[0048] According to the historical change variance, the dynamic update threshold for current comparison is adaptively adjusted and determined.

[0049] In this embodiment, the threshold of judgment is made intelligent and adaptive. Specifically, the semantic difference measurement values ​​calculated in the past multiple time steps are recorded to form a historical inter-frame semantic difference sequence. Based on this sequence, its statistical variance is calculated to obtain the historical change variance σ 2 , directly reflects the activity level of the recent scene. Dynamic update threshold T dyn is set to σ 2 Functions such as T dyn = a*σ 2 + b, where a and b are hyperparameters. When the scene is stable for a long time, σ 2 It will be very small, T dyn Also reduced accordingly, making the detection more sensitive to small changes; when the scene is already in a highly dynamic change, σ 2 It will be big, T dyn The cache update strategy is also improved to tolerate normal changes and avoid overly frequent updates, making it possible for the cache update strategy to adapt to the dynamic characteristics of the scene.

[0050] This embodiment introduces a dynamic update threshold mechanism, that is, calculating the historical change variance based on the semantic difference between historical frames, and adaptively adjusting the judgment threshold accordingly, so that the update strategy of the KV cache has scene perception and self-regulation capabilities. In relatively static scenes, the historical change variance is small, and the dynamic threshold will be lowered accordingly, so that the system remains highly sensitive to small but potentially critical image changes. In highly dynamic scenes, the historical change variance is large, and the dynamic threshold will be increased to tolerate normal, large-scale picture changes and avoid too frequent cache refreshes. The allocation strategy of computing resources is optimized so that the system can operate at a near-optimal efficiency point in scenarios with different dynamic characteristics, avoiding the problem of fixed thresholds being too slow in some scenarios or too sensitive in other scenarios, and improving the average response speed and performance stability of the entire system in variable task processes.

[0051] like Figure 4 As shown, according to one aspect of the present application, generating a new image Key-Value vector includes:

[0052] Identifying a local change region of a current image frame relative to a previous image frame;

[0053] Determine the image token corresponding to the area of ​​the local change region in the serialized representation;

[0054] Only perform selective re-encoding on the image token corresponding to the region to generate a new local Key-Value vector;

[0055] The newly generated local Key-Value vector is replaced with the corresponding position in the cached image Key-Value vector to form a new image Key-Value vector.

[0056] In this embodiment, to solve the problem that even if an update is determined to be necessary, if the change is only local, a full update still wastes computing resources, the update operation itself is refined to achieve precise attack. Specifically, by performing pixel-level or feature-level differential calculations on the current image frame and the previous image frame, the local change area R where the content has changed significantly is identified; based on the coordinate information divided by the image token, the local change area R is mapped to the image token T corresponding to the area it covers. update ; The model only takes T update This part of the token is sent to the Transformer encoder to calculate their newly generated local Key-Value vectors; in the KV cache of the previous frame of the complete image, the newly generated local Key-Value vectors are used to replace the T updateThe old vector at the original position is replaced by the old vector, while the KV vectors of other unchanged areas remain unchanged. The final result is the updated KV vector set for the complete image. Maximizing the reuse of calculation results for unchanged areas is the ultimate and most effective means of conserving computing resources, especially in scenarios where the target object partially moves within the frame. Through a series of progressive optimizations, the system implements a highly intelligent, efficient, and robust dynamic cache management mechanism, reducing the model's computational overhead and response latency in continuous inference tasks without sacrificing decision accuracy.

[0057] This embodiment introduces a regional update mechanism that identifies localized change regions, determines the image tokens corresponding to these regions, and selectively re-encodes only these tokens when an update is necessary. This increases the granularity of cache updates from the entire frame to the token level, maximizing computational reuse. In many practical application scenarios, such as characters moving across a map or drones tracking specific targets, the vast majority of the image (such as the background, sky, and buildings) remains unchanged. By accurately identifying and isolating the changing regions, the model only needs to pay the computational cost for a small portion of the dynamic content in the image, while reusing existing computational results (cached KV vectors) for the vast static background area. In scenarios where image content changes locally, the actual computational effort per frame is reduced by orders of magnitude, resulting in an increase in inference frame rate and a reduction in system energy consumption. This efficiency optimization is particularly critical for decision models deployed on mobile devices or edge computing devices, as it determines application usability and user experience.

[0058] According to one aspect of the present application, constructing a reconstructed position embedding sequence includes:

[0059] The part of the learnable position bias corresponding to the text modality is fused with the original text position embedding to generate the reconstructed text position embedding;

[0060] The part of the learnable position offset corresponding to the image modality is fused with the original image position embedding to generate the reconstructed image position embedding;

[0061] According to the text priority order of the spliced ​​input sequence, the reconstructed text position embedding and the reconstructed image position embedding are spliced ​​to form a reconstructed position embedding sequence.

[0062] In this embodiment, in order to solve the problem of position semantic misalignment caused by the rearrangement of the input sequence and further improve the model's perception accuracy of the sequence structure and the cross-modal fusion effect, the general learnable position bias is conceptually divided into B t and B acting on the image v Two parts. Specifically, calculate the reconstructed text position embedding P' t = Pt + B t ; Calculate the reconstructed image position embedding P' v = P v + B v ; Splice to get the reconstructed position embedding sequence P' = [P' t ;P' v This embodiment clarifies and streamlines the general position reconstruction process, providing a basis for subsequent independent optimization of the position representations of the two modalities.

[0063] According to one aspect of the present application, the learnable position bias includes:

[0064] A set of first position bias parameters specific to the text modality, used to fuse with the original text position embedding;

[0065] and a second position bias parameter set specific to the image modality for fusion with the original image position embedding;

[0066] The first position bias parameter set and the second position bias parameter set are independent of each other during initialization and training optimization.

[0067] In this embodiment, in order to solve the problem that the universal bias parameters may not be able to fully model the position characteristics of different modes, resulting in inter-modal interference during the learning process, the parameters of the learnable position bias are structurally optimized. When designing the model, a unified bias parameter matrix B is no longer defined, but two completely independent parameter matrices are defined: the first position bias parameter set B t and the second position bias parameter set B v . Among them B t and B v Each modality has its own independent parameter space and gradient update path. When the model adjusts the position perception of text based on task requirements, its gradient update does not affect the position bias parameters of the image, and vice versa. This avoids parameter coupling and enables the model to more accurately learn the optimal position compensation strategy for each modality, improving the final fusion effect.

[0068] This embodiment explicitly decouples the learnable position bias in the model architecture into a first position bias parameter set (specific for text) and a second position bias parameter set (specific for images), ensuring their independence during training and optimization. This effectively addresses the learning interference caused by differences in positional characteristics between modalities. Text sequences have strict one-dimensional temporal order, while image token sequences are flattened from a two-dimensional space, with complex adjacency relationships. If a unified bias parameter is used, the gradient updates generated when the model learns one-dimensional sequence compensation for text may negatively impact the learning of two-dimensional spatial perception for images. The physical isolation of the parameter sets provides independent calibration space for the two modalities, allowing the model to learn the optimal position representation strategy for each modality in parallel without interference. This improves the accuracy and efficiency of the model's learning of structured information from both modalities, enhances the accuracy of cross-modal information fusion, and improves the accuracy of decision-making models in tasks that require precise spatial understanding and command integration (such as moving to an enemy in the upper left corner of the screen).

[0069] According to one aspect of the present application, the fusion process further includes:

[0070] In each layer or a specific layer of the multi-layer Transformer encoder, a lightweight inter-layer update rule is applied to the reconstructed position embedding sequence of the layer to generate the updated position embedding of the current layer;

[0071] The updated position embedding of the current layer is fused with the token representation of this layer, and the fusion result is sent to the self-attention module of this layer for calculation.

[0072] In this embodiment, to address the problem of position bias being corrected only once at the input, failing to adapt to the dynamic evolution of the semantic structure of information in deep networks, the position information fusion process is changed from static injection to dynamic evolution. Specifically, in the Lth layer of the Transformer, the corrected position embeddings from the input layer are no longer directly used; instead, a lightweight inter-layer update rule is first applied to the position embeddings from the L-1 layer. For example, this could be a small, learnable transformation matrix associated with the layer index L, or a simple interaction with the token representation of that layer. After obtaining the updated position embeddings for the current layer, they are added or fused with the token representation of that layer (i.e., the hidden state output from the L-1 layer). The representation incorporating the latest position information is fed into the self-attention module of the Lth layer. As the network deepens, the semantic representation of tokens also changes continuously. This embodiment allows the representation of position information to keep pace with time, fine-tuning it at each layer to better match the semantic structure of the current layer. This enhances the model's ability to represent complex spatial and structural relationships in deep networks. This enhances the model's robustness to sequence reordering, and effectively improves the accuracy and efficiency of cross-modal information fusion.

[0073] This embodiment applies lightweight inter-layer update rules to position embeddings in each layer or specific layers of a multi-layer Transformer encoder, transforming the perception of position information from a static, one-time injection to a dynamic process that evolves with the depth of the network. In the shallow layers of the Transformer, the model focuses more on local, geometric positional relationships; as the network deepens, the semantic representation of tokens becomes more abstract and advanced, and the corresponding positional relationships should also evolve into functional or conceptual relative relationships. Position embeddings are allowed to interact and fine-tune with the token representation of each layer, so that position information can keep pace with and adapt to the deepening semantic information. The model's ability to model complex, hierarchical spatial structures and long-distance dependencies is enhanced. In autonomous driving or complex strategy games that require deep scene understanding, the model can more accurately understand nested spatial relationships such as vehicles under a distant bridge, improving the decision-making quality and reliability of the model in complex scenarios.

[0074] According to one aspect of the present application, the process of generating the final attention output using a multi-layer Transformer encoder includes the following in its internal self-attention calculation:

[0075] Perform a linear transformation on the total input representation of the model to generate corresponding Query, Key, and Value vectors respectively;

[0076] The original attention score matrix is ​​calculated based on the query vector and the key vector;

[0077] Introducing the modal weight coefficient to adjust the relative attention influence between the tokens of the text input sequence and the image input sequence in the original attention score matrix to obtain the adjusted attention score matrix;

[0078] Normalize the adjusted attention score matrix to generate normalized attention weights;

[0079] The output of the self-attention calculation is obtained by using the normalized attention weight and the Value vector.

[0080] In this embodiment, the original attention score matrix S is calculated according to the standard process. Specifically, it is obtained by calculating the dot product of Q and K, that is, the original attention score matrix S = Q * K T ,in T is the transpose. Each element in the matrix S ijrepresents the original attention of the i-th token to the j-th token. Before the original attention score matrix S is fed into the Softmax function for normalization, it is adjusted. Specifically, for the elements in the original attention score matrix S that represent cross-modal interactions (for example, the score calculated by Q of the text token and K of the image token), it is multiplied by the modality weight coefficient W modality . The interactions within the modality (text-text, image-image) can be multiplied by 1 or other coefficients. In this way, the weighted attention score matrix S' is obtained. The weighted attention score matrix S' is fed into the Softmax function to obtain the normalized attention weight W = Softmax(S'). The value vector (V) is weighted and summed with the normalized attention weight W to obtain the final output. Among them, the query vector Query(Q), the key vector Key(K), and the value vector Value(V) are the standard inputs of the self-attention mechanism; the modal weight coefficient W modality W is a learnable scalar or small matrix whose core function is to act as a regulating valve. Due to the different characteristics of text and image modalities, in some tasks, the model may unreasonably over-focus on one modality. The modality weight coefficient is automatically learned through training and can amplify or reduce the cross-modal attention effect. For example, if the model finds that it needs to rely more on text instructions, W modality A value greater than 1 might be learned to increase the attention of image tokens over text tokens. This embodiment does not modify the positional embeddings at the input end, but instead intervenes directly at the source of attention allocation, within the self-attention module, the core computational unit of the Transformer. This provides a direct and efficient means to address the modality bias problem.

[0081] According to one aspect of the present application, a text update detection mechanism is also included, including:

[0082] When a new text input sequence is received, it is compared with the text input sequence used to generate the current text key-value vector;

[0083] If the comparison result shows that the two text input sequences are inconsistent, the text Key-Value vector cache in the static buffer area is determined to be invalid;

[0084] Clear all contents in the static buffer to trigger the subsequent re-encoding and caching process using a new text input sequence.

[0085] In this embodiment, the text update detection mechanism is a resident monitoring module whose responsibility is to monitor the content consistency of the static cache. text , and also saves the original text input sequence I used to generate it tBefore each inference step begins, when the model receives a new text input sequence I' t When first I' t With the old text sequence stored in memory I t Compare. The comparison method can be a simple string or a complete match of a token sequence. If I' t with I t If there is any inconsistency (for example, the game task changes from defending point A to attacking point B), the cache invalidation is triggered immediately. The invalidation operation includes all KV text The vector is completely cleared. After the cache is cleared, the system will recognize the KV text The new text is automatically entered into the sequence I' t Treat it as a new task, perform complete encoding and caching operations on it, and generate a new and correct KV text The text update detection mechanism adds a crucial fuse to the cache reuse strategy. It ensures that the model's decisions are always based on the latest task instructions, avoiding catastrophic decision-making errors caused by the use of outdated context. This is an indispensable guarantee for the completeness and reliability of this embodiment as it moves from the laboratory to practical application. It ensures its logical correctness and robustness in real, long-term, and dynamic application scenarios, and avoids the system from continuing to use outdated cache when static text instructions change.

[0086] In summary, this application proposes an input rearrangement and cache reuse method for a multimodal decision model, including: placing the text input sequence before the image input sequence for sequence splicing to generate a text-first spliced ​​input sequence; during initial reasoning, using a multi-layer Transformer encoder to calculate and cache the text Key-Value vector corresponding to the text input sequence to a static cache area; during subsequent reasoning, performing inter-frame semantic consistency detection on the newly input current image frame, and performing on-demand or local updates to the Key-Value vector of the image portion based on the detection results; splicing and assembling the cached text Key-Value vector with the updated image Key-Value vector to complete the final decision. This effectively avoids redundant calculations of unchanged instructions, realizes refined and intelligent reuse of cache, and improves the reasoning speed and throughput of the multimodal decision model in dynamic tasks.

[0087] According to another aspect of the present application, a method for input reordering and cache reuse for a multimodal decision model includes:

[0088] Step 1: Design and Implementation of a Text-First Input Method. We propose an input sequence design method that optimizes multimodal inference architectures. This method employs a text-first strategy, placing the language modality embedding at the beginning of the input sequence, followed by the visual modality, replacing the traditional image-first structure. This approach is particularly suitable for scenarios where images frequently change while text remains stable, such as in game decision-making. It effectively avoids the resource waste and latency associated with repeatedly computing the same text information for each frame. During the model's initial inference phase, the system simultaneously receives text and image inputs, extracts the query, key, and value representations of all tokens, and marks the text-specific keys and values ​​as static conditions, caching them hierarchically in a KV cache. Subsequent inferences only update the image input, and the system directly reuses the cached text KV representation and concatenates it to the current image token for attention calculation, reducing redundant computational overhead. Furthermore, the initial placement of text in the sequence facilitates its semantic involvement in image token attention allocation, improving modality fusion. Overall, this strategy is simple in structure and easy to integrate, effectively improving the inference efficiency and throughput of multimodal models in dynamic image input scenarios.

[0089] Step 2: Dynamic position encoding reconstruction algorithm. To adapt to the text-first input strategy, a dynamic position encoding reconstruction algorithm was constructed. In traditional multimodal pre-training models, the position encoding is strongly coupled to the input order of image first and text second. Directly adjusting the token arrangement will cause position information misalignment, destroy the contextual semantic structure, and affect the inference accuracy. By introducing a modality-decoupled position bias term and adding it to the original position encoding, a dynamic position encoding is obtained. This bias parameter is a learnable term and can be automatically adjusted through backpropagation to adapt to the position structure under the current input order, ensuring the stability of spatial relationship expression. At the same time, this mechanism works synergistically with KV-Cache to ensure that the position information of static text tokens in the cache is aligned with the current image input, avoiding attention bias caused by cache reuse. On the basis of maintaining pre-training compatibility, the smooth migration of position perception from static encoding to structural adaptation is completed, improving the system's temporal consistency and cross-frame semantic alignment capabilities.

[0090] In a specific embodiment of the present application, let the text input sequence be I t =[I t 0 , I t 1 , I t 2 …I t L ], the image input sequence is I v =[I v 0 , I v 1, I v 2 …I v N ], representing the language and visual modality input at the current moment. In most mainstream multimodal Transformers, the input is usually in the form of a concatenation of image tokens in front and text tokens in the back. v ;I t ]; This design facilitates the fusion modeling of image-dominant tasks during the training phase and improves the model’s ability to focus on visual content. However, a significant efficiency bottleneck is exposed during the inference phase: since text input usually remains unchanged for a period of time, while image input is frequently updated, the model needs to repeatedly process the same text at each time step, resulting in redundant computational overhead and limiting the throughput and real-time performance of inference. To address this issue, a text-first input order strategy is proposed, which places the text token at the front of the input sequence. The specific form is as follows: I=[I t ;I v ]. In this way, the model receives the complete [I t ;I v ] input, the encoder extracts the embedding of all tokens and calculates the corresponding Query, Key and Value representation. In this process, the text part I t The generated Key and Value vectors are marked as static context conditions and stored in a dedicated KV-Cache module for subsequent reuse. KV-Cache supports a multi-layer Transformer hierarchical cache structure, which can fully retain the K t (l) , V t (l) Avoid repeated inter-layer propagation and recalculation.

[0091] To further improve the reasoning efficiency across time steps, an inter-frame reuse mechanism is introduced into the design of KV-Cache to achieve cache inheritance and synthesis between different time steps. Specifically, in the first reasoning phase, the system clearly divides the Key and Value of each layer of Transformer into two types of cache areas: static KV area KV text t :From text input I t Generate, keep constant within the time period; dynamic image KV area KV rest t :The current frame image input I vWhen entering the subsequent inference phase (t=1, 2, ...), the model performs the following inter-frame KV reuse process: Static KV call: directly read the text KV cached in the previous time step from the KV-Cache, that is, KV text t , no need to recalculate; dynamic image coding: the new image observation I v 'Input model, only embed and Transformer encoding operations are performed on it to generate new KV rest t+1 ; KV splicing assembly: At each Transformer level, the system will KV text t With KV rest t+1 Splice in sequence order to form the complete Key and Value representation of the current frame: K t+1 =[KV text t ;KV rest t+1 ]; V t+1 =[V text t ; V rest t+1 ]; Unified attention normalization: The model uses an optimized normalization strategy in attention calculation to ensure the cached KV text t With dynamic image KV rest t Participate equally in the attention weight to maintain the continuous influence of text semantic information; cache replacement update: cache KV of the image part rest t After each frame is updated, the new KV rest t+1 Covered, and KV text t remains unchanged; Text update detection and failure management: If the system detects that the text input I t If there is a change, KV will be determined immediately text The cache is invalidated, all related level caches are cleared, and the encoding operation is re-executed to ensure context consistency and semantic accuracy.

[0092] In addition, to improve the robustness and scalability of the cache system, the KV-Cache module also integrates the following key mechanisms. A multi-level cache structure is introduced, and each layer of Transformer can maintain its own KV correspondence separately, enabling the model to efficiently call deep semantic information. A cache reference index system is set up to quickly locate the corresponding cache data through location mapping or token ID, reducing memory access latency. In practical applications, to prevent potential deviations caused by text updates, the cache module supports an active invalidation management strategy. Once a change in text input is detected, the cache will be immediately cleared and a new round of text embedding will be re-encoded to ensure context consistency. In summary, with the help of the above-mentioned text-first input sequence design and sophisticated KV cache strategy, efficient static reuse of stable text context in multimodal sequence modeling is achieved, which reduces the overhead of repeated calculations and improves the system's real-time response capability and overall throughput under dynamic image input tasks.

[0093] In the multimodal Transformer system, position encoding is the key means for the model to perceive sequence structure information. Let the original position encoding of the text token be P t =[P t 0 , P t 1 ,…,P t N ], the original position of the image token is encoded as P v =[P v 0 , P v 1 ,…,P v N ], both are absolute position embedding vectors inherited from the pre-trained model. In a typical multimodal input configuration, the input sequence is concatenated in the order of image tokens first and text tokens last, so the corresponding position encoding vector is: P=[P v ;P t ]; This order is consistent with the pre-training stage, allowing the model to fully utilize the existing attention mode and context-aware structure. However, under the text-first input strategy adopted in this embodiment, the order of tokens received by the actual model is adjusted to: P=[P t ;P v ], and under the text-first input strategy, the order of tokens received by the actual model changes and is updated to: P=[P t ;P v], that is, the text token is placed at the front of the sequence and the image token is placed at the back. Since the position encoding adopts the absolute position indexing method, the original embedding position of the image token no longer corresponds to its new semantic sequence position, resulting in context dislocation. For example, an image token that was originally at the beginning or middle of the sequence is forced to be mapped to the end of the sequence due to the change in token order. This structural dislocation will be accumulated and amplified in the multi-layer attention calculation, causing semantic dependency disorder and attention shift, seriously affecting the accuracy of reasoning. To address this problem, a dynamic position encoding reconstruction mechanism based on learnable bias is proposed to achieve structural adjustment of pre-trained absolute position embeddings. Before each input sequence splicing, this mechanism initializes an independent set of learnable position bias parameters for text tokens and image tokens respectively: B t ∈R L×d , B v ∈R N×d ; where d represents the embedding dimension, L and N are the maximum lengths of text and image tokens respectively. The position bias parameter is optimized end-to-end through backpropagation during training. Its role is to structurally reconstruct the pre-trained position embedding and generate a new position representation that is more suitable for the current token order: P t '=P t +B t ; P v '=P v +B v ; The final position perception used for the actual input sequence is: P'=[P t ' ;P v ' ]. In this way, the model retains the semantics of the original absolute position encoding initialization while introducing adaptive compensation for token position reordering, constructing a dynamic position embedding with structure-aware capabilities. This mechanism essentially completes the transition from static absolute position representation to learnable structured perception, effectively suppressing the negative impact of cross-modal position perturbations on context modeling and ensuring the model's robustness to changes in token order.

[0094] It is worth emphasizing that this dynamic position encoding mechanism not only acts on the embedding generation process of the input token, but also works closely with the aforementioned KV-Cache cache structure. In the multi-layer structure of Transformer, the Key and Value of each cache layer are essentially representation vectors containing position encoding. If the token position information in the cache is inconsistent with the dynamic image token currently being processed, it is very easy to cause attention shift or semantic coupling errors. Therefore, the position bias reconstruction module in this mechanism acts on the following: Static cache path: When the text token is first encoded and stored in KV-Cache, the system will be able to learn the bias B tApplied to position embedding to ensure that the cached KV vector is consistent with the current sequence structure; dynamic image encoding path: in the process of token encoding of each frame image, the position offset B v Synchronous addition ensures alignment with the cached content in spatial structure. By uniformly adjusting the position information in the cache path and the forward path, this mechanism ensures that the spliced ​​KV representation has continuous structural consistency in the attention calculation, effectively avoiding semantic drift caused by the movement of cache tokens or changes in the order of image tokens. At the same time, this mechanism also provides structural guarantees in sequence modeling across time steps, enabling the model to continue to utilize stable language context over a long period of time without the need to frequently adjust the attention structure. In summary, this dynamic position encoding reconstruction mechanism achieves flexible adaptation to traditional absolute position encoding, effectively solves the position perception conflict problem caused by cross-modal token order misalignment under the text-first input strategy, and provides solid support for the structural stability and context consistency of the KV cache mechanism.

[0095] This embodiment proposes an inference optimization mechanism based on a text-first input structure for multimodal game reasoning scenarios. By reorganizing the Transformer's input sequence organization, text instructions originally at the end are embedded and adjusted to the beginning of the sequence, so that they are encoded and a stable key-value representation is generated at the initial stage of reasoning. This static text representation is uniformly extracted and cached in a multi-level KV-Cache module, supporting repeated calls across frames in subsequent time steps, effectively avoiding repeated calculations of unchanged task descriptions in continuous reasoning tasks. It is particularly suitable for dynamic game tasks where visual observations frequently change and text input is long-term stable. In such scenarios, the computational complexity of each inference step can be reduced. By reallocating computing resources, the model's computing power can be concentrated on the processing of dynamic image tokens and the strategy generation process, comprehensively improving the system's response frame rate and throughput. To address the problem of position semantic shift caused by input structure adjustment, a dynamic position encoding reconstruction algorithm is further proposed to address the context-aware bias caused by the absolute position embedding in the pre-trained model under the new modal order. Specifically, based on the original position encoding, the system introduces a set of learnable position bias parameters that independently affect the initial position embedding of text and image tokens. During the training phase, this biased structure automatically learns the structural adjustment strategy that best matches the current token order, generating adaptive dynamic position encoding vectors. The reconstructed position embedding not only accurately reflects the actual order of the input sequence, effectively avoiding attention dislocation caused by token order changes, but also ensures that the static token representations cached in the KV-Cache are consistently spatially aligned with the current dynamic image input, thereby ensuring semantic consistency and attention stability across frames.

[0096] This application demonstrates technical advantages in improving reasoning efficiency, reducing redundant calculations, and maintaining semantic consistency through the joint design of a text-first input structure and a dynamic position encoding reconstruction algorithm. It provides an optimization path with engineering value for multimodal models in typical practical deployment scenarios such as high-frequency perception, resource constraints, and task stability.

[0097] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the scope of protection of the present invention.

Claims

1. An input reordering and cache reuse method for a multimodal decision model, characterized in that: include: Receive a text input sequence and an image input sequence, and perform sequence splicing in a text-first order to generate a spliced ​​input sequence; The preset learnable position bias is fused with the original text and image position embeddings respectively to construct a reconstructed position embedding sequence that matches the order of the concatenated input sequence. Combine the concatenated input sequence with the reconstructed position embedding sequence to form the total input representation of the model; A multi-layer Transformer encoder is used to process the model's total input representation, extract and generate text key-value vectors from its text part, and store them in a static buffer for subsequent reasoning reuse. At the same time, the final attention output is generated based on the model's total input representation; Pass the final attention output to the decision head to generate the final decision result; When the text key-value vector is reused in subsequent reasoning, the process of using the multi-layer Transformer encoder further includes: Performing inter-frame semantic consistency detection on the current image frame and the previous image frame to generate an update decision signal; In response to the update decision signal indicating that an update is required, the current image frame is encoded to generate a new image Key-Value vector; In response to the update decision signal indicating that no update is required, directly reusing the cached image Key-Value vector corresponding to the previous image frame; Concatenate the text Key-Value vector with the new or reused image Key-Value vector to obtain the complete Key-Value set of the current frame; The final attention is calculated using the complete Key-Value set of the current frame to generate a decision result.

2. The method according to claim 1, characterized in that Generate update decision signals, including: From the feature sequence of historical image frames, a sliding window is defined to obtain a historical feature set within the window; Based on the historical feature set within the window, a single historical stability feature representing the recent stable semantics is calculated and generated; Compare the semantic differences between the features of the current image frame and the historical stability features; Generate an update decision signal based on the semantic difference.

3. The method according to claim 2, characterized in that The process of generating an update decision signal based on the semantic difference includes comparing with a dynamic update threshold, wherein the generation of the dynamic update threshold includes: Based on the historical sequence composed of semantic differences of a predetermined number of time steps in the past, the historical change variance is calculated and obtained; According to the historical change variance, the dynamic update threshold for current comparison is adaptively adjusted and determined.

4. The method according to claim 1, wherein Generate a new image Key-Value vector, including: Identifying a local change region of a current image frame relative to a previous image frame; Determine the image token corresponding to the area of ​​the local change region in the serialized representation; Only perform selective re-encoding on the image token corresponding to the region to generate a new local Key-Value vector; The newly generated local Key-Value vector is replaced with the corresponding position in the cached image Key-Value vector to form a new image Key-Value vector.

5. The method according to claim 1, wherein Constructing the reconstructed position embedding sequence, including: The part of the learnable position bias corresponding to the text modality is fused with the original text position embedding to generate the reconstructed text position embedding; The part of the learnable position offset corresponding to the image modality is fused with the original image position embedding to generate the reconstructed image position embedding; According to the text priority order of the spliced ​​input sequence, the reconstructed text position embedding and the reconstructed image position embedding are spliced ​​to form a reconstructed position embedding sequence.

6. The method according to claim 5, characterized in that Learnable position biases include: A set of first position bias parameters specific to the text modality, used to fuse with the original text position embedding; and a second position bias parameter set specific to the image modality for fusion with the original image position embedding; The first position bias parameter set and the second position bias parameter set are independent of each other during initialization and training optimization.

7. The method according to claim 5, characterized in that The fusion process further includes: In each layer or a specific layer of the multi-layer Transformer encoder, a lightweight inter-layer update rule is applied to the reconstructed position embedding sequence of the layer to generate the updated position embedding of the current layer; The updated position embedding of the current layer is fused with the token representation of this layer, and the fusion result is sent to the self-attention module of this layer for calculation.

8. The method according to claim 1, characterized in that The process of generating the final attention output using a multi-layer Transformer encoder includes the following in its internal self-attention calculation: Perform a linear transformation on the total input representation of the model to generate corresponding Query, Key, and Value vectors respectively; The original attention score matrix is ​​calculated based on the query vector and the key vector; Introducing the modal weight coefficient to adjust the relative attention influence between the tokens of the text input sequence and the image input sequence in the original attention score matrix to obtain the adjusted attention score matrix; Normalize the adjusted attention score matrix to generate normalized attention weights; The output of the self-attention calculation is obtained by using the normalized attention weight and the Value vector.

9. The method according to claim 1, characterized in that Also included is a text update detection mechanism, including: When a new text input sequence is received, it is compared with the text input sequence used to generate the current text key-value vector; If the comparison result shows that the two text input sequences are inconsistent, the text Key-Value vector cache in the static buffer area is determined to be invalid; Clear all contents in the static buffer to trigger the subsequent re-encoding and caching process using a new text input sequence.

Images