Response information generation method and apparatus, computer device, and program product

Abstract

Embodiments of the present application are suitable for the technical field of natural language processing, and provide a response information generation method and device, computer equipment and program product. The method comprises: performing pre-padding processing on input information of a multi-modal large language model to obtain pre-padding key values and hidden state data, wherein the hidden state data comprises intermediate hidden state tensors corresponding to each network layer; determining proxy hidden state data based on the hidden state data, wherein the proxy hidden state data is used to construct a decoding-aware proxy query, and the decoding-aware proxy query is used to determine importance information of the pre-padding key values in a decoding process; determining target key values from the pre-padding key values according to the importance information; and performing decoding inference based on the target key values to generate response information. Through the above method, the key value cache occupancy and memory access overhead can be reduced without performing online rearrangement or dynamic elimination in the decoding phase, the response efficiency of the multi-modal large language model is improved, and the accuracy of the response information is ensured.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a method, apparatus, computer equipment and program product for generating response information. Background Technology

[0002] Multimodal large language models (MLLMs) can perform natural language processing on various input information such as text, images, audio, and video, and output information corresponding to user commands.

[0003] Currently, multimodal large language models can determine multiple key-value pairs (KV) based on the input multimodal information, and generate response information based on these determined key-value pairs. However, if each key-value pair is used for reasoning when generating response information, it requires a large amount of computational resources, reducing computational efficiency.

[0004] To conserve computing resources and improve computational efficiency, a certain number of key-value pairs can be selected for subsequent inference generation. However, in multimodal tasks such as image-text processing, document understanding, Optical Character Recognition (OCR) enhanced question answering, and long-context visual question answering, the input sequence often contains a large number of visual tokens and text tokens simultaneously. This leads to a rapid increase in the size of the prompt sequence key-value cache, resulting in high GPU memory usage and memory access overhead. To reduce the aforementioned resource consumption, existing technologies typically compress the historical prompt sequence key-value pairs after the pre-filling stage, retaining only a portion of the key-value pairs considered more important for use in subsequent decoding stages.

[0005] Most existing key-value caching methods in the pre-filling stage implicitly rely on the assumption that the query distribution, attention patterns, or statistical features observed in the pre-filling stage represent the actual access needs of the historical context in the subsequent decoding stage. This assumption holds some merit in some unimodal text scenarios; however, it often fails to hold true in multimodal reasoning scenarios. This is because the primary function of the pre-filling stage is to encode and align image regions, text prompts, and cross-modal information, while the primary function of the decoding stage is to perform answer generation, information extraction, and reasoning decisions around the generation objective. Due to the difference in functional objectives between the two stages, queries in the decoding stage are typically more concentrated in a few key visual regions, key text fragments, or embedded text regions in images, resulting in a significant shift in query distribution between the pre-filling and decoding stages. Summary of the Invention

[0006] In view of this, embodiments of this application provide a response information generation method, apparatus, computer equipment, and program product to balance the efficiency and accuracy of multimodal large language models.

[0007] A first aspect of this application provides a response information generation method, applied to a computer device, comprising: The input information of the multimodal large language model is pre-filled to obtain pre-filled key values ​​and hidden state data. The hidden state data includes intermediate hidden state tensors corresponding to each network layer. The hidden state data represents the context information used to generate the pre-filled key values. Based on the hidden state data, proxy hidden state data is determined. The proxy hidden state data is used to construct a decoding-aware proxy query. The decoding-aware proxy query is used to determine the importance information of the pre-filled key value in the decoding process. Based on the importance information, the target key value is determined from the pre-filled key values; Decoding and reasoning are performed based on the target key value to generate response information.

[0008] In one possible implementation, determining the proxy hidden state data based on the hidden state data includes: The mean and standard deviation of the hidden state data are calculated according to the feature dimension, and the mean and standard deviation are used to characterize the distribution characteristics of the pre-filling stage; Based on the mean and standard deviation, the distribution of hidden states corresponding to each network layer is sampled to obtain multiple proxy hidden state data.

[0009] In one possible implementation, sampling is performed on the hidden state data of each network layer based on the mean and standard deviation to obtain multiple proxy hidden state data, including: The standard deviation is expanded based on the variance expansion factor; The proxy hidden state data is obtained by performing Gaussian distribution random sampling based on the mean and the expanded standard deviation.

[0010] In one possible implementation, a decode-aware proxy query is constructed based on the proxy hidden state data in the following manner: The proxy hidden state data is mapped to the query vector space to obtain the decoded perception proxy query.

[0011] In one possible implementation, the importance information includes attention distribution, voting results, and a candidate pre-filled key-value set; obtaining the importance information based on the proxy query includes obtaining the importance information based on the decode-aware proxy query in the following manner: Calculate the attention distribution of each pre-filled key value for each decoded perceptual proxy query under the corresponding network layer and attention head; The multiple decoded perception agent queries are divided into multiple groups, and each group has a corresponding cumulative attention quality within the group; Based on the cumulative attention quality within each group, a set of candidate pre-filled key values ​​is determined for each network layer and each attention head. Based on the cumulative attention quality within the group, the voting results for each group are determined.

[0012] In one possible implementation, determining the target key value from the pre-filled key value based on the importance information includes: Based on the voting results of each group, the pre-filled key values ​​are sorted to obtain the sorting results; Based on the attention distribution and the candidate pre-filled key value set, sort the pre-filled key values ​​of the tied tickets in the sorting results; The target key value is determined from the pre-filled key values ​​in the order of sorting from front to back.

[0013] In one possible implementation, the step of decoding and reasoning based on the target key value to generate response information includes: Based on the target key value, the pre-filled key value is compressed and recombined to obtain the target key value cache; The response information is obtained by decoding and reasoning based on the target key-value cache.

[0014] A second aspect of this application provides a response information generation apparatus, comprising: The pre-filling module is used to pre-fill the input information of the multimodal large language model to obtain pre-filled key values ​​and hidden state data. The hidden state data includes intermediate hidden state tensors corresponding to each network layer. The hidden state data represents the context information used to generate the pre-filled key values. The decoding-aware proxy query module is used to determine proxy hidden state data based on the hidden state data. The proxy hidden state data is used to construct a decoding-aware proxy query. The decoding-aware proxy query is used to determine the importance information of the pre-filled key value in the decoding process. Based on the importance information, the target key value is determined from the pre-filled key value. The reasoning module is used to perform decoding reasoning based on the target key value to generate response information.

[0015] A third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect described above.

[0016] A fourth aspect of this application provides a computer program product that, when run on a computer device, causes the computer device to perform the method described in the first aspect.

[0017] Compared with the prior art, the embodiments of this application have the following advantages: The method described in this application embodiment can be used to compress the key values ​​of a multimodal large language model, thereby ensuring correctness while improving computational efficiency. In this application embodiment, the input information of the multimodal large language model can be pre-filled to obtain pre-filled key values ​​and hidden state data. The hidden state data includes intermediate hidden state tensors corresponding to each network layer, representing the context information used to generate the pre-filled key values. Based on the hidden state data, proxy hidden state data is determined, which is used to construct a decoder-aware proxy query. The decoder-aware proxy query is used to determine the importance information of the pre-filled key values ​​in the decoding process. Based on the importance information, a target key value is determined from the pre-filled key values. Decoding inference is performed based on the target key value to generate response information. This application embodiment selects the target key value with high importance to the response output through a simulated decoder-aware proxy query between pre-filling and the decoder, thereby reducing the number of key values ​​that need to be processed during inference and improving computational efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0019] Figure 1 This is a flowchart illustrating the steps of a response information generation method provided in an embodiment of this application; Figure 2 This is a data processing flowchart of a multimodal large language model provided in an embodiment of this application; Figure 3 This is a schematic diagram of an input information provided in an embodiment of this application; Figure 4 This is a schematic diagram of a response information generation device provided in an embodiment of this application; Figure 5 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] Existing multimodal large language models typically employ a key-value caching mechanism during inference to store historical keys and values ​​generated during the pre-filling stage of the prompt sequence. This allows for repeated access in subsequent decoding stages, avoiding redundant computation and improving inference efficiency. However, in multimodal tasks such as image-text processing, document understanding, OCR-enhanced question answering, and long-context visual question answering, the input sequence often contains a large number of visual and textual terms, causing the size of the prompt sequence key-value cache to increase rapidly, resulting in high GPU memory usage and memory access overhead. To reduce these resource consumptions, existing technologies typically compress the historical prompt sequence keys after the pre-filling stage, retaining only a subset of key-value pairs deemed more important for use in subsequent decoding stages.

[0022] Most existing key-value caching methods in the pre-filling stage implicitly rely on the assumption that the query distribution, attention patterns, or statistical features observed in the pre-filling stage represent the actual access needs of the historical context in the subsequent decoding stage. This assumption holds some merit in some unimodal text scenarios; however, it often fails to hold true in multimodal reasoning scenarios. This is because the primary function of the pre-filling stage is to encode and align image regions, text prompts, and cross-modal information, while the primary function of the decoding stage is to perform answer generation, information extraction, and reasoning decisions around the generation objective. Due to the difference in functional objectives between the two stages, queries in the decoding stage are typically more concentrated in a few key visual regions, key text fragments, or embedded text regions in images, resulting in a significant shift in query distribution between the pre-filling and decoding stages.

[0023] Under the aforementioned distribution mismatch conditions, if the importance of historical cue sequence keys is still evaluated solely based on local attention, activation statistics, or similarity information from the pre-filling stage, the evaluation results are likely to deviate from the actual needs of the decoding stage. Consequently, the compressed retained set cannot effectively cover the key contextual information upon which subsequent generation depends. This leads to the following technical problems: Key visual or textual terms are easily mistakenly deleted, leading to omissions of crucial information such as color attributes, object categories, field information, or image text in the generated results. Under strict budget constraints, the compressed results are highly sensitive to local noise and ranking fluctuations, causing model performance to degrade rapidly and making it difficult to balance compression ratio and generation quality. Using dynamic correction methods during the decoding stage to correct selection errors in the pre-filling stage introduces additional online computational overhead, latency jitter, and higher system implementation complexity.

[0024] Existing methods struggle to simultaneously meet the comprehensive requirements of multimodal inference tasks regarding accuracy, memory usage, latency stability, and engineering deployability. Therefore, it is necessary to propose a novel key-value cache selection method that can anticipate potential query distributions during the decoding phase in the pre-filling stage. Without modifying the main decoding path, this method performs on-budget stable selection and compression reconstruction of historical hint sequence keys, thereby improving the retention of critical contextual information and enhancing compression performance, inference stability, and engineering deployability in multimodal inference scenarios.

[0025] Existing technologies can perform key-value compression using window / heuristic methods. However, representative techniques in this category include key-value caching compression methods such as StreamingLLM, which rely on fixed windows and attention convergence point retention. The core idea is to directly prune historical key-values ​​based on preset positional rules without complex modeling of the actual contribution of each lexical element to subsequent generation. Specifically, these methods typically retain a small number of initial lexical elements at the beginning of the sequence that act as "attention sinks," while also retaining some lexical elements within the nearest window, and discarding most historical key-values ​​in the middle region, thus maintaining long-sequence inference with a near-constant cache size. These methods are simple to implement, have low inference overhead, and are easy to integrate with existing decoding processes. However, their retention decisions mainly rely on fixed positional assumptions rather than adaptive selection based on specific task semantics, visual region importance, or cross-modal relationships. This method has the following drawbacks: difficulty in identifying key information across positions and modalities: These methods mainly rely on fixed positional rules to retain key-value caches, failing to effectively capture key visual information, key text fragments, and long-distance dependencies distributed across positions, and are prone to mistakenly deleting context crucial for subsequent generation. Weak generalization ability across different tasks and input distributions: These methods rely on pre-defined heuristic rules and are typically only applicable to specific input structures or scenarios. Their retention performance tends to decline significantly when task type, input length, modal composition, or region of interest changes. Risk of key information loss under low-budget conditions: When the number of retainable keys is strictly limited, these fixed-rule methods struggle to balance local continuity with global importance, easily leading to insufficient keyword coverage and consequently reducing the robustness and accuracy of the overall inference results.

[0026] Existing technologies can also perform key-value compression using pre-filled statistical scoring methods. Representative techniques in this category include key-value cache selection methods like SnapKV, which are based on attention statistics during the pre-filling stage. The core idea is to estimate the importance of each lexical key by utilizing the attention distribution of real query statements on historical key values ​​during the prefilling stage, and then perform pruning all at once after prefilling. Specifically, these methods typically select an observation window at the end of the cue sequence, statistically analyze the attention response of the query statements to prefix key values ​​within this window, and sort the historical key values ​​according to their attention strength. Simultaneously, to improve the local continuity of the retained results, high-response positions can be aggregated, and several high-scoring items from the historical key values ​​can be selected and retained along with the key values ​​corresponding to the observation window. This achieves cache compression and reduced memory access overhead without interfering with the main path in the decoding stage. Compared to fixed-window or positional heuristic methods, these methods have stronger data-driven characteristics, but their importance assessment is still primarily based on the observation results during the prefilling stage. The main drawbacks of this type of method include: First, it fails to explicitly model the distribution offset between the pre-filling and decoding stages: these methods typically assume that the attention patterns observed at the end of the pre-filling stage represent the true attention distribution in the subsequent decoding stage, but this assumption is often unstable in actual generation. Second, it lacks representativeness of future attention regions in multimodal tasks: in scenarios such as image and text understanding and document question answering, the focus of the query statement in the pre-filling stage often differs significantly from the attention region oriented towards the generation target in the decoding stage. Therefore, relying solely on the pre-filling statistical results can easily miss truly crucial visual or textual terms. Third, it is highly sensitive to noise under low-budget conditions, resulting in poor stability of the retained set: when the number of retainable keys is strictly limited, the selection results based on one-time sorting are easily affected by local attention fluctuations, leading to unstable retention results between different samples or different layers. Fourth, it is difficult to guarantee keyword coverage: because this type of method mainly relies on the attention intensity of the pre-filling stage for selection and lacks explicit coverage constraints oriented towards the subsequent decoding target, it may still mistakenly delete key contextual information that plays a decisive role in the final generation quality during compression.

[0027] Existing technologies can also perform key-value compression through dynamic rearrangement / eviction during the decoding stage. Representative techniques of this type include H2O, which dynamically manages the key-value cache during the decoding stage. The core idea is to continuously estimate the importance of historical keys using the real decoding query vector during the autoregressive generation process, and dynamically perform retention, eviction, or window adjustment based on the latest estimation results, thereby preserving as much contextual information as possible that is more likely to be accessed in subsequent generation within a limited cache budget. Specifically, this type of method typically continuously calculates the contribution of historical terms to the current generation in several decoding steps, evictions keys considered to have low contributions in real time, and prioritizes the retention of recently generated relevant keys or frequently accessed important keys. The range of the accessible historical window can also be dynamically adjusted when necessary. Compared to schemes that only prune once during the pre-filling stage, this type of method can directly use query information from the real decoding stage to correct early estimation biases, thus theoretically being closer to the actual access needs of the subsequent generation stage. The main disadvantages of this type of method include: introducing additional online computational overhead: this type of method needs to continuously recalculate or update key-value importance during the decoding process and perform corresponding eviction and maintenance operations, increasing additional computational and memory access burdens. This approach can easily cause end-to-end latency jitter, which is detrimental to service stability: Because the cache adjustment overhead of different generation steps is not constant, online inference latency is prone to fluctuations, which is not conducive to stable deployment in service-oriented scenarios. The system implementation is complex, and the cost of coordinating with high-performance inference kernels is high: This type of method requires inserting additional cache management logic into the main decoding path, which is usually difficult to seamlessly integrate with existing high-performance attention kernels or high-throughput inference frameworks. Online changes to the pre-filled key-value structure increase the difficulty of engineering integration: Because historical key-value pairs are dynamically eliminated, rearranged, or reorganized during the decoding process, the implementation complexity of the system in terms of cache index maintenance, data layout adaptation, and operator interface design will be significantly increased.

[0028] Unlike the aforementioned prior art, the method in this application does not rely on preset rules such as fixed windows, head and tail retention, or attention sinks to heuristically prune the historical key-value cache. Nor does it simply score and sort historical key-values ​​based on the local attention statistics formed by the actual query statements in the pre-filling stage. Furthermore, it does not continuously perform online reordering, dynamic elimination, or window adjustment during decoding. Instead, after pre-filling and before decoding begins, it constructs a "decoding-aware proxy query" that can characterize the future decoding query distribution. Based on this decoding-aware proxy query, it estimates and stably selects the importance of historical prompt sequence key-values. Specifically, this application performs variance-expanded sampling on the hidden state distribution in the pre-filling stage, applies future position alignment to the decoding-aware proxy query, and combines group voting and stable sorting mechanisms to complete a one-time screening and compressed reconstruction of key historical key-values ​​under budget constraints without interfering with the main decoding path. Compared to window / heuristic methods, this application can more effectively identify key information across locations and modalities; compared to pre-filled statistical scoring methods, this application explicitly considers the distribution offset problem between the pre-filling stage and the decoding stage; compared to dynamic rearrangement / elimination methods in the decoding stage, this application does not need to introduce additional cache management overhead during online generation, thus simultaneously achieving compression effect, inference stability, and engineering deployability.

[0029] The technical solution of this application will be described below through specific embodiments.

[0030] Reference Figure 1 The diagram illustrates a flowchart of a response information generation method provided in an embodiment of this application, which may specifically include the following steps: S101, pre-fill the input information of the multimodal large language model to obtain pre-filled key values ​​and hidden state data. The hidden state data includes intermediate hidden state tensors corresponding to each network layer. The hidden state data represents the context information used to generate the pre-filled key values.

[0031] The input information of the aforementioned multimodal large language model can include information in various formats, such as text, video, audio, images, GIFs, documents, and other data in different formats.

[0032] Multimodal large language models can pre-fill input information. The pre-filling stage is the first forward computation stage in the inference stage of a multimodal large language model based on the Transformer architecture. It refers to the complete process in which the model performs a one-time full sequence encoding on the multimodal prompt sequence, generates and caches historical key-value pairs, and provides a contextual basis for subsequent autoregressive decoding. The key-value pair generation process can refer to existing technologies and will not be elaborated here.

[0033] S102, based on the hidden state data, determine the proxy hidden state data, the proxy hidden state data is used to construct the decoding-aware proxy query, the decoding-aware proxy query is used to determine the importance information of the pre-filled key value in the decoding process.

[0034] The mean and standard deviation of the hidden state data are calculated according to the feature dimension. The mean and standard deviation are used to characterize the distribution characteristics of the pre-filling stage. Based on the mean and standard deviation, samples are taken from the hidden state data of each network layer to obtain multiple proxy hidden state data.

[0035] Variance expansion can be performed on the mean and standard deviation; based on the expanded mean and standard deviation, random sampling with a Gaussian distribution is performed to obtain the agent hidden state data.

[0036] By mapping the proxy hidden state data to the query vector space, a decoder-aware proxy query is obtained. This decoder-aware proxy query is used to approximate the query distribution that may occur in subsequent decoding stages when the actual query vector in the decoding stage cannot be directly obtained, and to assess the potential importance of historical hint sequence key values ​​accordingly.

[0037] Importance information, including attention distribution, voting results, and candidate pre-filled key-value sets, is obtained based on decode-aware proxy queries in the following ways: calculating the attention distribution of each pre-filled key-value for each decode-aware proxy query under the corresponding network layer and attention head; dividing multiple decode-aware proxy queries into multiple groups, each group having a corresponding intra-group cumulative attention quality; determining the candidate pre-filled key-value sets under each network layer and attention head based on the intra-group cumulative attention quality; and determining the voting results for each group based on the intra-group cumulative attention quality.

[0038] For example, the hidden states or equivalent feature representations generated at each layer of the multimodal large language model during the pre-filling stage are read, and their distribution statistics are extracted to provide a basis for constructing a decoding-aware proxy query. The mean of each pre-filled hidden state Hl is calculated along the feature dimension. and standard deviation When the standard deviation is too small or the numerical distribution is too concentrated, a minimum variance lower bound can be introduced to improve the stability of subsequent sampling.

[0039] Based on the statistical results of the pre-filling stage, a decoding-aware proxy query that approximates the query distribution of the decoding stage is constructed to predict the historical context information that may be of interest before decoding begins.

[0040] Based on statistics at each level and Multiple proxy hidden state samples are generated by combining the variance expansion factor γ; the proxy hidden state is mapped to the query vector space through the query projection matrix WQ(l,h) of each layer and head to obtain the decoded perception proxy query; the future position rotation position encoding is applied to the decoded perception proxy query so that its position distribution is aligned with several positions at the beginning of the decoding stage.

[0041] Decode-aware proxy queries are used to assess the potential importance of historical pre-filled keys in subsequent decoding processes and form a scoring basis for budget selection. The attention distribution or equivalent response value of each decode-aware proxy query to each pre-filled key is calculated; multiple decode-aware proxy queries are divided into several groups, and the attention quality within each group is accumulated; based on the accumulated results within each group, a candidate set of key values ​​is determined, and the group voting results are generated.

[0042] S103, Determine the target key value from the pre-filled key value based on the importance information.

[0043] Based on the voting results of each group, sort the pre-filled key values ​​to obtain the sorting results; based on the attention distribution and the candidate pre-filled key value set, sort the pre-filled key values ​​with tied votes in the sorting results; determine the target key value from the pre-filled key values ​​in the order of sorting from front to back.

[0044] For example, under a given key-value cache budget constraint, the final set of key-value pairs to be retained is determined. The group voting results for each key are accumulated to form an overall importance ranking; keys with similar or tied scores are stably ranked using anchor terms to reduce selection jitter; under the budget constraint C(l,h), several keys with the highest scores are selected, and the keys corresponding to the nearest lexical are forcibly retained to ensure local continuity and generation stability.

[0045] S104, Decode and reason based on the target key value to generate response information.

[0046] Based on the target key value, the pre-filled key value is compressed and recombined to obtain the target query vector; the target query vector is then decoded by a decoder to obtain the response information.

[0047] For example, based on the retained index output by the budget selection module, the original prompt sequence key values ​​are compressed, recombined, or indexed for direct reading in the subsequent decoding stage. A continuous copy reconstruction of the original prompt sequence key values ​​is performed according to the retained index, forming a compressed key-value cache tensor; or a sparse index mapping is constructed based on the retained original prompt sequence key values, allowing access to corresponding key-value pairs by index during the decoding stage. The multimodal large language model performs autoregressive inference word-by-word based on the prompt sequence key-value cache generated in the pre-filling stage and generates output content. In this stage, queries for newly generated words continuously access the historical key-value cache; therefore, the size and retention quality of the prompt sequence key values ​​directly affect inference efficiency and generation quality.

[0048] This application is deployed in the inference process of a multimodal large language model, located between the pre-filling stage and the decoding stage. It is used to select, compress and reconstruct the key-value cache (pre-filled key-value) corresponding to the prompt sequence under budget constraints after pre-filling is completed and before decoding begins. This reduces cache usage and memory access overhead and improves the stability of key context information retention without interfering with the subsequent main decoding path.

[0049] The statistical results of each layer are output to the decoding-aware proxy query generation module.

[0050] Figure 2 This is a schematic diagram of the data processing flow of a multimodal large language model provided in an embodiment of this application. For example... Figure 2 As shown, the data processing flow for a multimodal large language model may include: Multimodal large language models receive inputs such as images, text, or documents, and perform pre-filling computations to generate hidden states for each layer and complete key-value caches of the cue sequences. After this step, the model has obtained the historical context representation required for subsequent decoding, but at this point, the number of original cue sequence key-values ​​is large, which will lead to high cache usage and memory access overhead.

[0051] The statistics module reads the hidden states or equivalent feature representations of each layer in the pre-filling stage and calculates their distribution statistics, including the mean, according to the feature dimensions. and standard deviation This step is used to characterize the overall representation distribution of the pre-filling stage and provides a foundation for the subsequent construction of decode-aware proxy queries.

[0052] The decoding-aware proxy query generation module is based on statistics at each layer. and Combined with variance expansion factor Generate multiple proxy hidden state samples and query the projection matrix. This is mapped to the query vector space to form decode-aware proxy queries corresponding to each layer and head. Subsequently, a rotational position encoding of the future position is applied to the decode-aware proxy query, aligning its position distribution with the starting position of the decoding stage, thereby approximately representing the query behavior of subsequent decoding stages.

[0053] The importance assessment module uses decode-aware proxy queries to calculate their attention distribution or equivalent response values ​​to each cue sequence key value, and divides multiple decode-aware proxy queries into several groups, performing cumulative analysis on the attention quality within each group. This step is used to identify important historical key values ​​that may be accessed in subsequent decoding from different decode-aware proxy query perspectives, thereby forming a more stable candidate set.

[0054] The budget selection module performs voting statistics and comprehensive ranking of historical key values ​​based on the cumulative results of each group, and combines anchor terms to stably rank key values ​​with similar scores. Under the given budget constraint `C(l,h)`, it selects several historical key values ​​with the highest scores, while forcibly retaining the keys corresponding to the most recent terms, resulting in the final set of pre-filled key value indices. This step improves the coverage of key contextual information while ensuring local continuity.

[0055] Based on the reserved index set, the original hint sequence key values ​​are compressed and recombined, or corresponding index mappings are generated, forming a compressed hint sequence key value cache. Subsequently, the model enters the regular decoding stage, directly performing autoregressive inference based on the compressed hint sequence key values, without performing online rearrangement or dynamic eviction of the hint sequence key values. This step reduces GPU memory usage and memory access overhead while helping to maintain the stability of inference latency.

[0056] The MM-Shift key-value selection algorithm is the core selection logic of this application. It is used to stably select key-value pairs in the hint sequence under budget constraints after the pre-filling phase and before the decoding phase. This algorithm constructs a decoding-aware proxy query to predict the potential importance of historical key-value pairs and, combined with group voting and a stable sorting mechanism, generates a final set of retained indexes to guide the compression and reconstruction of subsequent hint sequence key-value pairs. This application achieves efficient compression of the key-value cache without introducing online reordering overhead by adopting a "one-time selection in the pre-filling phase and direct reuse in the decoding phase" approach. The MM-Shift key-value selection algorithm aims to "improve keyword meta-coverage, enhance selection stability under low budget, and reduce memory access overhead in the decoding phase," and collaboratively optimizes the three processes of decoding-aware proxy query generation, importance assessment, and in-budget retention.

[0057] The input to the MM-Shift key-value selection algorithm includes the hidden states generated by each layer of the multimodal large language model during the pre-filling stage. Query projection matrix for each layer and each header The corresponding prompt sequence key-value and key-value cache budget and variance expansion factor proxy sampling number Number of groups Intra-group coverage threshold and anchor weights .in, For layer index, For attention head index; Indicates the first The pre-filled set of hidden states of the layer; Indicates the first Layer The query projection matrix of the head; Indicates the first Layer The key-value cache budget corresponding to each head; Used to adjust the variance expansion magnitude of the agent's hidden state; Indicates the number of samples taken by the proxy; Indicates the number of groups; This represents the cumulative attention quality threshold within the group; This represents the weight of the anchor item in a stable ranking. Furthermore, This represents the key corresponding to the t-th historical Prompt token under the h-th header at layer l. This represents the set of indices for all historical hint sequence key values. Indicates the first The set of indexes for group decoding-aware proxy queries. Indicates the first The set of key elements in the group that meet the cumulative attention quality threshold. This represents the last real query in the pre-filling stage under the h-th head of layer l, where d represents the attention head dimension. This indicates the index of the historical key corresponding to the most recent token.

[0058] Initialization: Pre-fill the hidden state for each layer. Calculate the mean along the feature dimension. and standard deviation This step initializes the candidate key set and voting statistics structure for each layer and head. It is used to extract the overall representation distribution features of the pre-filling stage, providing a foundation for constructing the decoding-aware proxy query. As shown in formula (1), the statistics can be expressed as:

[0059] Based on statistics at each level ` and Combined with variance expansion factor generate Each proxy hides a state sample, and the query projection matrix of each layer and head is used. Mapping this to the query space forms a decoder-aware proxy query; subsequently, future position alignment is applied to the decoder-aware proxy query to approximate the query distribution at several positions at the beginning of the decoding stage at the position encoding level. This step is used to construct an approximate representation of the future decoding query behavior in the pre-filling stage. As shown in formulas (2) and (3), the proxy hidden state sample and the decoder-aware proxy query can be expressed as follows:

[0060] Utilize each decoded perceptual agent query to compute its attention response to the key values ​​of each cue sequence under the current head in the current layer, and then... Each decode-aware proxy query is divided into For each group, the attention quality is accumulated. For each group, a threshold that meets the accumulated attention quality requirement is selected. The smallest set of keys is used as the key values ​​within the group. This step is used to identify important historical keys that are more likely to be accessed in subsequent decoding from multiple agent perspectives. As shown in Equations (4) to (5), the methods for determining the attention response of the decoding-aware agent query to historical keys, the cumulative attention quality within the group, and the key set within the group can be expressed as follows:

[0061] The key values ​​obtained from each group are voted on, and each key selected by the key set within the group receives one vote, thus obtaining the cumulative vote value for each key. Based on this, the voting results can be stabilized by combining anchor terms to reduce the ranking instability caused by single sampling fluctuations. This step improves the stability of the selection results under low budget conditions through the joint decision-making of multiple sets of decoded perception proxy queries. As shown in formulas (7) and (8), the cumulative voting value and stable ranking score can be expressed as follows:

[0062]

[0063] In a given budget Under constraints, based on the scores of each key value, select the highest-scoring historical key values ​​and forcibly retain the key values ​​corresponding to the most recent terms, thus obtaining the final set of retained indexes. This step improves the coverage of key contextual information while meeting budget constraints. As shown in formula (9), the final retained index set can be expressed as:

[0064] Based on the reserved index set The original cue sequence key values ​​are compressed and recombined or indexed and mapped to form a compressed cue sequence key value cache. Subsequently, the model enters the regular decoding phase, directly using the compressed cue sequence key values ​​for autoregressive inference, without performing dynamic rearrangement or online eviction during decoding. Through these steps, this application can perform decoding-aware estimation of the importance of historical cue sequence key values ​​after the pre-filling phase and before the decoding phase begins, and complete stable selection and compression reconstruction under budget constraints.

[0065] This embodiment uses a mixed text and image input scenario as an example to illustrate the specific implementation process of this application. Figure 3 This is a schematic diagram illustrating one type of input information provided in this application. For example... Figure 3 As shown, the input information is a sequence of mixed image and text prompts, including image-encoded terms and text instruction terms. The task type is image description or visual question answering. The input image contains visual attribute information and embedded text information that significantly influence the final generated result, such as clothing color, object category, packaging label, or readable text in the image. After the multimodal large language model completes pre-filling, it obtains the hidden states corresponding to the input at each layer and the complete prompt sequence key-value cache. Now, it needs to be cached in each head. The key-value cache selection is completed under the constraints. The specific implementation process is as follows: The statistics module reads the hidden states generated at each layer of the image-text hybrid hint sequence during the pre-filling stage. And calculate the mean according to the feature dimension. and standard deviation This characterizes the overall distribution of the input at the current layer. If the standard deviation is too small, a minimum variance lower bound can be set to prevent degradation in subsequent surrogate sampling.

[0066] The decoding-aware proxy query generation module generates a query based on the statistics corresponding to the input. and And combined with variance expansion factor Multiple proxy hidden state samples are sampled at each layer; then, the samples are mapped to decode-aware proxy queries through the query projection matrix of each layer and each head, and future position alignment is applied so that the decode-aware proxy queries can approximately represent the query distribution that the image input may generate in the subsequent decoding stage.

[0067] The importance assessment module uses the decoded perception proxy query to calculate its attention distribution to the input historical prompt sequence key value, and divides multiple decoded perception proxy queries into several groups. It accumulates the attention quality of each group of decoded perception proxy queries, thereby identifying the historical key values ​​corresponding to the key visual regions and key text regions that are more likely to affect the subsequent generation results in the image and text scene.

[0068] The budget selection module performs voting statistics and stable sorting based on the cumulative results of each group, obtains the importance ranking results of historical key values ​​under the current layer and current head, and then applies the ranking to the budget. Under constraints, a final set of retention items is selected, while retaining the keys corresponding to the most recent terms. In this embodiment, the budget selection process can increase the probability of retaining key contextual information related to color attributes, object categories, and image text.

[0069] The index reconstruction module compresses and reassembles the original prompt sequence key value corresponding to the input based on the reserved set, generating a compressed prompt sequence key value cache, or establishes an index mapping corresponding to the reserved set for direct access in the subsequent decoding stage.

[0070] The model enters the regular decoding stage, directly accessing the compressed prompt sequence key values ​​for autoregressive generation. In this mixed text and image input scenario, if a fixed window method or a selection method based on pre-filled local statistics is used, key information such as color attributes, object categories, or embedded text may be missed under low budget conditions. However, this application constructs a decoding-aware proxy query in the pre-filling stage, which can more stably retain important visual and text regions related to subsequent generation, thereby improving the completeness and accuracy of the generated results under compression conditions.

[0071] Furthermore, in specific tasks such as describing clothing, recognizing product packaging, recognizing advertising logos, or document image question answering, if the image contains both visual attribute information and readable text information, existing methods tend to retain only the subject outline information while omitting attribute or text information. This application, however, can more stably retain historical key-value information corresponding to the above-mentioned key information under strict budget constraints, thus making it more conducive to maintaining the complete expression of visual attributes and image text in the generated results.

[0072] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0073] Reference Figure 4 The diagram illustrates a response information generation device provided in an embodiment of this application, which may specifically include a pre-filling module 41, a decoding perception proxy query module 42, and an inference module 43, wherein: The pre-filling module 41 is used to pre-fill the input information of the multimodal large language model to obtain pre-filled key values ​​and hidden state data. The hidden state data includes intermediate hidden state tensors corresponding to each network layer. The hidden state data represents the context information used to generate the pre-filled key values. The decoding-aware proxy query module 42 is used to determine proxy hidden state data based on the hidden state data. The proxy hidden state data is used to construct a decoding-aware proxy query. The decoding-aware proxy query is used to determine the importance information of the pre-filled key value in the decoding process. Based on the importance information, the target key value is determined from the pre-filled key value. The reasoning module 43 is used to perform decoding reasoning based on the target key value to generate response information.

[0074] In one possible implementation, determining the proxy hidden state data based on the hidden state data includes: The mean and standard deviation of the hidden state data are calculated according to the feature dimension, and the mean and standard deviation are used to characterize the distribution characteristics of the pre-filling stage; Based on the mean and standard deviation, the distribution of hidden states corresponding to each network layer is sampled to obtain multiple proxy hidden state data.

[0075] In one possible implementation, determining the proxy hidden state data based on the hidden state data includes: The mean and standard deviation of the hidden state data are calculated according to the feature dimension, and the mean and standard deviation are used to characterize the distribution characteristics of the pre-filling stage; Based on the mean and standard deviation, the distribution of hidden states corresponding to each network layer is sampled to obtain multiple proxy hidden state data.

[0076] In one possible implementation, sampling is performed on the hidden state data of each network layer based on the mean and standard deviation to obtain multiple proxy hidden state data, including: The standard deviation is expanded based on the variance expansion factor; The proxy hidden state data is obtained by performing Gaussian distribution random sampling based on the mean and the expanded standard deviation.

[0077] In one possible implementation, a decode-aware proxy query is constructed based on the proxy hidden state data in the following manner: The proxy hidden state data is mapped to the query vector space to obtain the decoded perception proxy query.

[0078] In one possible implementation, the importance information includes attention distribution, voting results, and a candidate pre-filled key-value set; obtaining the importance information based on the proxy query includes obtaining the importance information based on the decode-aware proxy query in the following manner: Calculate the attention distribution of each pre-filled key value for each decoded perceptual proxy query under the corresponding network layer and attention head; The multiple decoded perception agent queries are divided into multiple groups, and each group has a corresponding cumulative attention quality within the group; Based on the cumulative attention quality within each group, a set of candidate pre-filled key values ​​is determined for each network layer and each attention head. Based on the cumulative attention quality within the group, the voting results for each group are determined.

[0079] In one possible implementation, determining the target key value from the pre-filled key value based on the importance information includes: Based on the voting results of each group, the pre-filled key values ​​are sorted to obtain the sorting results; Based on the attention distribution and the candidate pre-filled key value set, sort the pre-filled key values ​​of the tied tickets in the sorting results; The target key value is determined from the pre-filled key values ​​in the order of sorting from front to back.

[0080] In one possible implementation, the step of decoding and reasoning based on the target key value to generate response information includes: Based on the target key value, the pre-filled key value is compressed and recombined to obtain the target key value cache; The response information is obtained by decoding and reasoning based on the target key-value cache.

[0081] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0082] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 5 As shown, the computer device 500 of this embodiment includes: at least one processor 50 ( Figure 5 (Only one is shown in the diagram), memory 51, and computer program 52 stored in memory 51 and executable on at least one processor 50, wherein processor 50 executes computer program 52 to implement the steps in any of the above method embodiments.

[0083] Computer device 500 may be a server, cloud server, personal terminal, etc. This computer device may include, but is not limited to, processor 50 and memory 51. Those skilled in the art will understand that... Figure 5 This is merely an example of computer device 500 and does not constitute a limitation on computer device 500. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0084] The processor 50 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0085] In some embodiments, memory 51 may be an internal storage unit of computer device 500, such as a hard disk or memory of computer device 500. In other embodiments, memory 51 may be an external storage device of computer device 500, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on computer device 500. Furthermore, memory 51 may include both internal and external storage units of computer device 500. Memory 51 is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. Memory 51 may also be used to temporarily store data that has been output or will be output.

[0086] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0087] This application provides a computer program product that, when run on a computer device, enables the computer device to perform the steps described in the above-described method embodiments.

[0088] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for generating response information, characterized in that, include: The input information of the multimodal large language model is pre-filled to obtain pre-filled key values ​​and hidden state data. The hidden state data includes intermediate hidden state tensors corresponding to each network layer. The hidden state data represents the context information used to generate the pre-filled key values. Based on the hidden state data, proxy hidden state data is determined. The proxy hidden state data is used to construct a decoding-aware proxy query. The decoding-aware proxy query is used to determine the importance information of the pre-filled key value in the decoding process. Based on the importance information, the target key value is determined from the pre-filled key values; Decoding and reasoning are performed based on the target key value to generate response information.

2. The method as described in claim 1, characterized in that, Determining the proxy hidden state data based on the hidden state data includes: The mean and standard deviation of the hidden state data are calculated according to the feature dimension, and the mean and standard deviation are used to characterize the distribution characteristics of the pre-filling stage; Based on the mean and the standard deviation, the distribution of hidden states corresponding to each network layer is sampled to obtain multiple proxy hidden state data.

3. The method as described in claim 2, characterized in that, The hidden state distribution corresponding to each network layer is sampled based on the mean and the standard deviation to obtain multiple proxy hidden state data, including: The standard deviation is expanded based on the variance expansion factor; The proxy hidden state data is obtained by performing Gaussian distribution random sampling based on the mean and the expanded standard deviation.

4. The method according to any one of claims 1-3, characterized in that, The following method is used to construct a decoding-aware proxy query based on the proxy hidden state data: The proxy hidden state data is mapped to the query vector space to obtain the decoded perception proxy query.

5. The method according to any one of claims 1-3, characterized in that, The importance information includes attention distribution, voting results, and a set of candidate pre-filled key-value pairs; the importance information is obtained based on the decoded perception proxy query in the following manner: Calculate the attention distribution of each pre-filled key value for each decoded perceptual proxy query under the corresponding network layer and attention head; The multiple decoded perception agent queries are divided into multiple groups, and each group has a corresponding cumulative attention quality within the group; Based on the cumulative attention quality within each group, a set of candidate pre-filled key values ​​is determined for each network layer and each attention head. Based on the cumulative attention quality within the group, the voting results for each group are determined.

6. The method as described in claim 5, characterized in that, Determining the target key value from the pre-filled key values ​​based on the importance information includes: Based on the voting results of each group, the pre-filled key values ​​are sorted to obtain the sorting results; Based on the attention distribution and the candidate pre-filled key value set, sort the pre-filled key values ​​of the tied votes in the sorting results; The target key value is determined from the pre-filled key values ​​in the order of sorting from front to back.

7. The method as described in claim 5, characterized in that, The step of decoding and reasoning based on the target key value to generate response information includes: Based on the target key value, the pre-filled key value is compressed and recombined to obtain the target key value cache; The response information is obtained by decoding and reasoning based on the target key-value cache.

8. A response information generation device, characterized in that, include: The pre-filling module is used to pre-fill the input information of the multimodal large language model to obtain pre-filled key values ​​and hidden state data. The hidden state data includes intermediate hidden state tensors corresponding to each network layer. The hidden state data represents the context information used to generate the pre-filled key values. The decoding-aware proxy query module is used to determine proxy hidden state data based on the hidden state data. The proxy hidden state data is used to construct a decoding-aware proxy query, and the decoding-aware proxy query is used to determine the importance information of the pre-filled key value in the decoding process. Based on the importance information, the target key value is determined from the pre-filled key values; The reasoning module is used to perform decoding reasoning based on the target key value to generate response information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes a computer program to implement the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When a computer program product is run on a computer device, it causes the computer device to perform the method as claimed in any one of claims 1-7.

Images