Segmented mixed reasoning method based on uncertain driving large language model

Abstract

This invention relates to a segmented hybrid inference method for large language models based on uncertainty-driven approaches. The method includes acquiring current text data and historical state features to estimate the uncertainty index of the current segment; minimizing a unified scheduling objective function based on the uncertainty index to obtain a target inference pattern; performing inference calculations based on the target inference pattern to generate information contribution values ​​corresponding to key-value pairs; calculating the corresponding dynamic merging control probabilities based on the uncertainty index and information contribution values, and performing weighted merging or pruning on the key-value pairs to be merged to obtain compressed key-value pairs; defining a deviation metric and limiting the deviation metric to not exceed a preset upper bound determined by the dynamic merging control probability set and the uncertainty index; triggering a rollback process when the deviation exceeds this limit; otherwise, feeding back the compressed key-value pair state to the next segment for iterative iteration until the inference of all segments is completed; thereby reducing memory usage and inference latency while ensuring accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a segmented hybrid reasoning method based on an uncertain-driven large language model, a computer-readable storage medium, and a computer device. Background Technology

[0002] As the application of large language models in natural language processing and generative artificial intelligence continues to expand, model inference typically includes a "pre-filling stage with contextual clues (full inference)" and a "token-by-token generation stage (incremental inference)". In long context scenarios, the pre-filling stage can significantly increase waiting time and memory pressure, thereby affecting inference throughput and service stability.

[0003] In the Transformer decoding model, the self-attention mechanism stores a key-value cache (KV cache) during inference. Its usage increases approximately linearly with the sequence length, becoming the main memory bottleneck for long sequence inference.

[0004] To alleviate the bottleneck of video memory and the waiting problem of long text inference, a type of "segmented inference scheme based on preset length" has emerged in the existing technology: the long text is divided into multiple segments according to the preset length, and full inference, hybrid inference and / or incremental inference are performed on each segment in batches to reduce peak video memory usage and improve throughput.

[0005] For example, in a common implementation, long requests are divided into multiple segments of fixed length (e.g., a request of about 8000 characters is split into four segments of about 2000 characters each). The first segment is inferred and historical results are generated. Then, the subsequent segments are merged with the newly arrived request information and hybrid inference is performed, so as to process long inputs while serving short inputs in parallel as much as possible.

[0006] However, the aforementioned segmentation strategy based on "preset length / fixed rules" has significant drawbacks: First, it lacks content awareness. Fixed segmentation ignores the semantic density and logical dependencies of the input text, easily causing critical context breaks at logically complex or strongly dependent boundaries, thus inducing inference drift or illusions. Second, it wastes resources. In low-difficulty, low-entropy segments, fixed strategies often cannot further reduce resource overhead, and there is still considerable room for improvement in inference efficiency. Third, it lacks closed-loop control. Existing solutions mostly focus on "how to segment / how to perform full and hybrid inference orchestration between segments," but usually do not incorporate the compression or merging of the underlying key-value cache into a unified decision-making framework, resulting in uncontrollable errors when the two are combined.

[0007] Meanwhile, another type of existing technology attempts to compress or merge KV caches. For example, it discards some key-value pairs of tokens based on a fixed threshold, a fixed ratio, or a random mask before performing mean / weighted mean merging to reduce the cache size.

[0008] On the other hand, for KV caching compression, existing technologies typically decide whether to retain or discard key-value pairs based solely on local attention scores. For example, by calculating the attention level of the current query to historical keys, a fixed threshold is set, and KV pairs with attention levels below the threshold are directly deleted.

[0009] However, this compression strategy based on "local attention" has an inherent flaw: it is a "locally greedy" strategy that ignores the global confidence (i.e., uncertainty) generated by the model. When the model is dealing with highly ambiguous and difficult text passages (where the model is in a state of high uncertainty), even "marginal information" with low attention scores can play a crucial role in correcting errors in subsequent inference. If aggressive compression is still mechanically applied based on the attention threshold at this time, it is very easy to destroy the robustness of semantics, causing the model to produce illusions or logical collapse. Conversely, when the model is extremely confident (low uncertainty), this strategy is often too conservative and cannot maximize the release of memory. Summary of the Invention

[0010] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a segmented hybrid inference method for large language models based on uncertainty-driven approaches. This method integrates "segmented inference mode selection" and "KV cache compression / merging" into the same closed-loop scheduling framework, enabling the system to dynamically adjust the inference mode and compression intensity based on the current uncertainty of the model, and to back off promptly when errors exceed limits. This reduces memory usage and inference latency while maintaining accuracy.

[0011] A second objective of this invention is to provide a computer-readable storage medium.

[0012] The third objective of this invention is to provide a computer device.

[0013] To achieve the above objectives, the first embodiment of this invention proposes a segmented hybrid inference method for large language models based on uncertainty-driven inference, comprising the following steps: acquiring the text data and historical state features of the current input request, and estimating the uncertainty index of the current segment based on the historical state features, wherein the historical state features include the attention distribution, hidden state statistical features, and / or system operating state features of the previous inference time step or historical window, or feature vectors calculated by a lightweight prior network; constructing a unified scheduling objective function including memory usage cost, inference latency cost, and uncertainty risk, and minimizing the unified scheduling objective function based on the uncertainty index to select the target inference mode for the current segment from a preset inference mode set, wherein the inference mode set includes full inference, hybrid inference, and incremental inference; and according to... The target inference mode performs inference calculations on the text data of the current segment, generates key-value pairs, and calculates the information contribution of each generated key-value pair. Based on the uncertainty index and the information contribution, it calculates the dynamic merging control probability of each key-value pair, and performs weighted merging or pruning on the key-value pairs to be merged according to the sampling or deterministic selection results to obtain compressed key-value pairs. It defines a deviation metric between the compressed output distribution and the full inference output distribution, and limits the deviation metric to not exceed a preset upper bound determined by the dynamic merging control probability set and the uncertainty index. When the deviation metric exceeds the preset upper bound, it triggers a rollback process to adjust the merging strength and / or scheduling weight. When the deviation metric does not exceed the preset upper bound, it feeds back the state of the compressed key-value pairs to the next segment for iterative iteration until the inference of all segments is completed.

[0014] The segmented hybrid inference method for large language models based on uncertainty-driven principles proposed in this application has the following advantages: it can incorporate "segmented inference mode selection" and "KV cache compression / merging" into the same closed-loop scheduling framework, enabling the system to dynamically adjust the inference mode and compression intensity according to the current uncertainty of the model, and to back off in time when the error exceeds the limit, thereby reducing memory usage and inference latency while ensuring accuracy.

[0015] In addition, the uncertain-driven segmented hybrid reasoning method for large language models proposed in the above embodiments of the present invention may also have the following additional technical features:

[0016] Optionally, the uncertainty index is one or more of attention entropy, logit entropy, top-k margin, or self-consistency divergence, and is estimated by the attention distribution, hidden state statistical features, and / or lightweight prior network of the previous inference time step.

[0017] Alternatively, the unified scheduling objective function can be obtained from the following formula:

[0018] Where m represents the candidate inference mode, VRAM(m) represents the GPU memory usage when executing mode m, Latency(m) represents the inference latency, and Risk(m) represents the risk measure caused by low precision or low context coverage. , Represents the normalization constant. , , Indicates the weighting coefficient. Indicators representing uncertainty.

[0019] Optionally, the information contribution is calculated by accumulating the attention weight, gradient approximation sensitivity, or information gain of the i-th token in the current segment and subsequent segments, in order to characterize the degree of influence of the key-value pair on the subsequent inference output.

[0020] Optionally, the dynamic merging control probability can be obtained according to the following formula:

[0021]

[0022] in, This represents the Sigmoid function. Indicates the degree of information contribution. Indicators representing uncertainty This prevents positive numbers with a denominator of zero. This indicates an adjustable parameter; and, The symbols and values ​​are used to set and The monotonous relationship.

[0023] Optionally, the Kullback–Leibler divergence or Jensen–Shannon divergence can be used to measure the deviation between the compressed output distribution and the full inference output distribution.

[0024] Optionally, when a deviation metric is detected to exceed a preset threshold, the weight of the uncertainty risk term in the unified scheduling objective function is increased. This triggers a switching of inference modes to favor the more robust full-scale inference mode.

[0025] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing an uncertain-driven large language model segmented hybrid inference program, which, when executed by a processor, implements the uncertain-driven large language model segmented hybrid inference method as described above.

[0026] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described uncertain-driven large language model segmented hybrid reasoning method. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the segmented hybrid reasoning method based on an uncertain-driven large language model according to an embodiment of the present invention.

[0028] Figure 2 This is a flowchart illustrating a segmented hybrid reasoning method based on an uncertain-driven large language model according to an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the architecture of a segmented hybrid reasoning system based on an uncertain-driven large language model according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the core decision-making logic according to an embodiment of the present invention, wherein, Figure 4 a illustrates the change in the dominance of the risk term in mode selection within the unified objective function under low and high uncertainty scenarios; Figure 4 b illustrates the relationship between the dynamic retention probability curve based on U and S and the shift of U, and provides a reference line for the upper bound constraint of the error. Detailed Implementation

[0031] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0032] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0033] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0034] Figure 1This is a flowchart illustrating the segmented hybrid reasoning method based on an uncertain-driven large language model according to an embodiment of the present invention, as shown below. Figure 1 As shown, this uncertain-driven segmented hybrid reasoning method for large language models includes the following steps:

[0035] S110, obtain the text data and historical state features of the current input request, and estimate the uncertainty index of the current segment based on the historical state features. The historical state features include the attention distribution, hidden state statistical features and / or system operation state features of the previous inference time step or historical window, or feature vectors calculated by a lightweight prior network.

[0036] As an example, the uncertainty metric employs one or more of attention entropy, logit entropy, top-k margin, or self-consistent divergence, and is estimated through the attention distribution, hidden state statistics, and / or lightweight prior network of the previous inference time step.

[0037] S120: Construct a unified scheduling objective function that includes memory usage cost, inference latency cost, and uncertainty risk. Minimize the unified scheduling objective function according to the uncertainty index to select the target inference mode for the current segment from a preset inference mode set. The inference mode set includes full inference, hybrid inference, and incremental inference.

[0038] As an example, the unified scheduling objective function is obtained according to the following formula:

[0039] Where m represents the candidate inference mode, VRAM(m) represents the GPU memory usage when executing mode m, Latency(m) represents the inference latency, and Risk(m) represents the risk measure caused by low precision or low context coverage. , Represents the normalization constant. , , Indicates the weighting coefficient. Indicators representing uncertainty.

[0040] Specifically, in the unified objective function , , Configuration can be tailored to business objectives: In latency-sensitive scenarios, it can improve... Precision-sensitive scenarios can improve In scenarios with limited video memory, performance can be improved. VRAM(m) can be estimated online, for example, based on the current cache pool size, the size of the temporary active cache required by the candidate mode, and the number of concurrent requests; Latency(m) can be obtained by using a sliding window to count the execution time of the previous batch of requests; Risk(m) can be obtained through offline calibration or empirical setting.

[0041] It should be noted that the target inference pattern is selected from candidate inference patterns by minimizing Score(m). Specifically, this selection is "dynamically matched": when U is low, the system tends to choose the more resource-efficient hybrid or incremental pattern; when U is high, the system automatically increases the impact of risk items and switches to the more robust full-scale pattern (see [link to relevant documentation]). Figure 4 a).

[0042] S130, perform inference calculations on the text data of the current segment according to the target inference pattern, generate key-value pairs, and calculate the information contribution of each generated key-value pair.

[0043] As an example, the information contribution is calculated by accumulating the attention weight, gradient approximation sensitivity, or information gain of the i-th token in the current segment and subsequent segments, and is used to characterize the degree of influence of the key-value pair on the subsequent inference output.

[0044] S140: Calculate the dynamic merging control probability of each key-value pair based on the uncertainty index and information contribution, and perform weighted merging or pruning on the key-value pairs to be merged based on the sampling or deterministic selection results to obtain compressed key-value pairs.

[0045] As an example, the dynamic merging control probability is obtained according to the following formula:

[0046]

[0047] in, This represents the Sigmoid function. Indicates the degree of information contribution. Indicators representing uncertainty This prevents positive numbers with a denominator of zero. This indicates an adjustable parameter; and, The symbols and values ​​are used to set and The monotonic relationship is such that, in one embodiment, key-value pairs with higher contributions have a lower tendency to merge or a higher tendency to retain.

[0048] Specifically, the parameter k in the dynamic merge control probability formula can be used to adjust the sensitivity of the contribution to the retention / merger decision: the larger the absolute value of k, the steeper the curve; the smaller the absolute value of k, the flatter the curve. The bias term b is used to set the base retention rate; Its function is to improve numerical stability and avoid unstable amplification effects when U is close to 0.

[0049] It should be noted that the above probability mapping is only an example and does not require the use of Sigmoid. Without changing the idea of ​​"the intensity is jointly modulated and combined by U and S", equivalent probability or threshold generation mechanisms can also be achieved by Tanh, Softplus or piecewise linear functions.

[0050] It should be noted that, to avoid ambiguity, in one embodiment, P(i) can be defined as the "retention probability". The corresponding "merging probability" can be expressed as:

[0051]

[0052] Thus, when the contribution level The higher the value, the higher the retention probability; when the uncertainty index U is higher, the system retains cache more conservatively for the same contribution (see [reference]). Figure 4 (A diagram illustrating the upward / leftward shift of the curve in b as U moves).

[0053] S150 defines a deviation metric between the compressed output distribution and the full inference output distribution, and limits the deviation metric to a preset upper bound determined by the dynamic merging control probability set and the uncertainty index.

[0054] As an example, the Kullback–Leibler divergence or Jensen–Shannon divergence is used to measure the deviation between the compressed output distribution and the full inference output distribution.

[0055] S160, when the deviation metric exceeds the preset upper bound, a rollback process is triggered to adjust the merging strength and / or scheduling weight; when the deviation metric does not exceed the preset upper bound, the compressed key-value pair status is fed back to the next segment for iterative iteration until the inference of all segments is completed.

[0056] As an example, when the deviation metric is detected to exceed a preset threshold, the weight of the uncertainty risk term in the unified scheduling objective function is increased. This triggers a switching of inference modes to favor the more robust full-scale inference mode.

[0057] In one embodiment, the error constraint defines the offset metric D (e.g., KL divergence or JS divergence) of the compressed output distribution relative to the full inference output distribution, and establishes an upper bound constraint relationship for the error:

[0058]

[0059] When the upper bound of the error is detected to exceed the preset threshold δ, the system triggers a rollback process: for example, reducing the compression ratio (increasing the retention probability or reducing the merging probability), adjusting the weight coefficient to make the risk item more dominant, and / or switching to a more robust inference mode, thereby forming a closed-loop control of "detection - adjustment - re-execution / continue".

[0060] It should be noted that although existing schemes segment the input according to a preset length and perform batch inference, these schemes are usually characterized by "fixed segmentation rules, fixed pattern matching, or weak adaptability." This application introduces an uncertainty index to enable the system to have dual adjustment capabilities of "dynamic pattern matching + dynamic compression intensity," and forms a closed-loop backoff through an error constraint module, thereby avoiding the uncontrollable error problem caused by the superposition of segmentation and compression.

[0061] In other words, this application does not treat "segmented reasoning" and "cache compression" as two separate strategies, but rather integrates them into the same scheduling semantics: U determines the overall conservative / aggressive level of the system, S determines the retention / merging priority of local entries, and error constraints determine whether rollback and re-decision are needed.

[0062] In addition, as a specific embodiment, such as Figure 3 As shown, the system includes: a feature acquisition module, an uncertainty estimation module, a scheduling decision module, an inference execution module, a cache merging control module, an error constraint / rollback detection module, and a KV cache pool.

[0063] The feature acquisition module receives the input text and acquires historical state features related to inference. These historical state features can come from the model side (e.g., attention statistics and hidden state statistics from the previous segment) or the system side (e.g., runtime status such as memory utilization, bandwidth utilization, and latency). In an optional implementation, system runtime status features can be collected in real-time by sensors or a monitoring module and used to form a state vector.

[0064] The uncertainty estimation module is used to predict the generation uncertainty index U of the current segment based on feature vectors or statistical features. The module can use uncertainty quantification methods such as attention entropy, logit entropy, top-k margin, and self-consistent divergence; it can also use a lightweight prior network to predict U, thereby avoiding the logical paradox of having to complete a full inference to obtain U.

[0065] The scheduling decision module is used to construct a unified scheduling objective function and make dynamic decisions among the candidate inference mode set. Unlike schemes based on fixed rules, this embodiment emphasizes "on-demand matching": when U is low, the system favors a resource-friendly mode; when U is high, the system favors a robust mode, and can accept backoff signals for re-decision when the error exceeds the limit (see...). Figure 2The top dashed line indicates the "Undo / Adjust" path.

[0066] The inference execution module performs inference computation on the current segment according to the target inference pattern m, generating the model output and original key-value pairs for the current segment. This module reads historical key-value cache from the key-value cache pool and reuses it, thereby reducing redundant computation.

[0067] The cache merging control module receives the original key-value pairs and contribution S from the inference execution module, and generates a dynamic merging control probability by combining it with the uncertainty index U. Then, it performs selective merging / pruning to obtain the compressed key-value pairs and writes them back to the key-value cache pool.

[0068] The error constraint / rollback detection module is used to verify the output deviation caused by cache compression. If the detected deviation exceeds the threshold, a rollback / adjustment signal is sent to the scheduling decision module and the cache merging control module, such as reducing the compression ratio, increasing the weight of risk items, or switching the inference mode to form a closed-loop control.

[0069] In addition, as a specific embodiment, such as Figure 2 As shown, the method starts from "Start / Receive Inference Request", executes steps S101 to S107 in sequence, and enters the backoff branch when the error exceeds the limit.

[0070] S101: Acquire Input Text and System Operating Status Features. The feature acquisition module receives the input text and can simultaneously acquire system operating status features as auxiliary information, such as remaining video memory, GPU utilization, average queuing latency, and bandwidth usage. The purpose of this step is to provide observable data for subsequent uncertainty estimation and scheduling decisions.

[0071] In one embodiment, the input text can be organized into a token sequence; the system can maintain the statistical features of the most recent steps using a sliding window approach. For the first segment, the key-value cache pool can be initialized to empty or a preset hint cache can be loaded.

[0072] S102: Estimating the uncertainty index U based on historical characteristics. Uncertainty estimation does not necessarily rely on "the true uncertainty after the complete reasoning of the current segment," but can be predicted using attention statistics or hidden state statistics of the previous segment, thereby enabling scheduling decisions to occur in advance.

[0073] As one example, entropy can be calculated on the attention distribution of the previous time step to obtain a component of the uncertainty index; as another example, entropy or top-k probability difference can be calculated on the logits distribution; as a third example, the degree of divergence of results generated from multiple samplings can be measured. The above examples can be used individually or in combination.

[0074] S103 & S104: Select the target inference mode m* based on the unified objective function Score(m). In particular, this application does not simply perform mechanical segmentation based on a fixed length and then fix the matching inference mode, but combines the uncertainty index U estimated in step S102 to achieve the linkage scheduling of "on-demand segmentation" and "dynamic mode matching": when U shows that the current segment is more prone to errors or more dependent on the context, the system tends to select the mode with more complete context coverage; when U shows that the current segment is relatively stable, the system tends to select the mode with lower resource consumption.

[0075] The unified objective function can be:

[0076]

[0077] The first term in the above formula describes the memory pressure. VRAM(m) can be obtained through offline profiling, online estimation, or table lookup; Used for normalization to make scores comparable across different hardware environments.

[0078] The second term characterizes latency pressure. Latency(m) can be taken as the end-to-end inference time, or as a weighted combination of the pre-filling stage time and the average incremental stage time; Used for normalization.

[0079] The third term characterizes the uncertainty risk. Risk(m) can be a mode-related risk coefficient or function, representing the degree to which mixed / incremental modes are more likely to cause output deviations under conditions of insufficient context coverage, quantization error, or reuse error; and this risk is amplified or reduced by U, making the system automatically more conservative under high uncertainty.

[0080] like Figure 4 As shown in a: In low uncertainty scenarios, the risk factor contributes less to the score, and the system is more likely to choose incremental or hybrid mode; in high uncertainty scenarios, the risk factor increases significantly, and the system is more likely to switch to full mode to suppress error propagation.

[0081] S105: Perform inference computation and generate original key-value pairs (KV). The inference execution module performs inference on the current segment according to the selected target inference mode m*. The full inference mode performs a full attention computation on the current segment; the hybrid inference mode allows some tokens to reuse historical KV and some tokens to perform full computation; the incremental inference mode mainly performs computation on newly added tokens and maximizes the reuse of historical states.

[0082] In one embodiment, to avoid position misalignment caused by the reuse of historical key-value pairs, an offset can be added to the position encoding of historical key-value pairs or an alignable index mapping strategy can be adopted to ensure that the concatenated sequence maintains the same relative position semantics.

[0083] S106-A: Calculate the information contribution S of key-value pairs. The contribution evaluation module calculates the information contribution S_i for each key-value entry corresponding to each token. The higher the contribution, the more critical the entry is to subsequent output, and in principle, it should be retained more conservatively.

[0084] In one alternative implementation, cumulative attention weights can be used as a contribution approximation: the intensity of attention access to a KV item in subsequent steps is counted and accumulated; in another alternative implementation, gradient approximation sensitivity estimation can be used: the magnitude of output distribution change after applying a small perturbation to the item is evaluated; in yet another alternative implementation, information gain or mutual information approximation can be used as the contribution.

[0085] S106-B: Generating a dynamic merging control probability p based on U and S. The cache merging control module treats the uncertainty index U as "global conservatism" and the information contribution S as "local importance," and generates a dynamic merging control probability through a nonlinear function. It is important to note that the "dynamic merging control probability" generation mechanism in this step is fundamentally different from traditional attention pruning methods based on fixed thresholds. Traditional methods (such as those based solely on attention scores)... Set threshold ,when (Time-based deletion) is equivalent to the default model having a constant dependence on information at any given time. This application, however, introduces... As a denominator (or negatively correlated term), a "floating threshold" is constructed: when the model uncertainty... As the threshold increases, the "soft threshold" automatically decreases, allowing low-contribution key-value pairs that might otherwise be deleted to be retained, thus providing the model with more contextual cues to aid in error correction; conversely, the same applies. This non-linear adaptive adjustment is not available in existing technologies.

[0086] As a concrete example, the following calculation formula can be used:

[0087]

[0088] in, This represents the contribution of the i-th KV entry; U represents the uncertainty index of the current segment. The value is a very small positive number to enhance numerical stability; k is the scaling factor and b is the bias term. The function of Sigmoid is to map the result to the (0,1) interval to satisfy the probability definition.

[0089] To align with the intuition that "higher contributions lead to greater conservatism, while lower contributions lead to greater aggressiveness," this embodiment allows P(i) to be defined as the "retention probability," from which the merging probability is derived as 1-P(i). Furthermore, the sign of k can be used to define the monotonic direction: when "greater contributions lead to greater retention," P(i) can be set to follow... It monotonically increases; when it is necessary to "the greater the contribution, the less likely to merge", P(i) can be directly used for retention decisions or k can be assigned the corresponding sign to achieve equivalent effects.

[0090] S106-C: Performs selective merging / pruning of the KV cache. Based on the output of the dynamic control probability, the system can use Bernoulli sampling to generate a mask or a deterministic top-r selection strategy to determine the merge / retain sets. For entries that need to be merged, weighted averaging, grouping aggregation, or other merging operators can be used to generate new KV entries; for retained entries, they are directly written back to the cache pool.

[0091] In an optional implementation, a "cumulative contribution minimum constraint" can be set: for example, to ensure that the total contribution of the final retained set is not less than a preset ratio, so as to ensure that the critical context is not over-compressed.

[0092] S107: Error Upper Bound Constraint Detection (D ≤ δ?). After merging, the error constraint / back-off detection module estimates the deviation D of the compressed output distribution relative to the full inference output distribution. D can be a measure such as KL divergence or JS divergence.

[0093] In one embodiment, the error constraint can be expressed in the following form:

[0094]

[0095] Where f is a non-decreasing function used to limit the upper bound of the maximum error caused by the merging probability set and the level of uncertainty.

[0096] If D is detected to exceed the threshold δ, then proceed to... Figure 2 The shown rollback branch triggers a rollback strategy (reduce compression ratio / switch mode). Reducing the compression ratio can be achieved by increasing the retention probability, decreasing the size of the merge group, or increasing the minimum cumulative contribution guarantee ratio; switching modes can be achieved by adding... Alternatively, you can switch directly to a more robust full-scale inference implementation.

[0097] If D is detected to be within the threshold, the KV cache pool is updated and the status is fed back to the next segment. Then it is determined whether the inference has ended. If it has not ended, it returns to S101 to enter the next iteration.

[0098] Specifically, such as Figure 4 As shown, this application achieves dynamic dual adjustment of inference mode and cache compression by introducing an uncertainty index U.

[0099] like Figure 4 As shown in a, when the model is in a low uncertainty state, the risk cost term has a weaker influence on the score, and the system tends to choose the incremental mode or hybrid mode that consumes less resources; when the model is in a high uncertainty state, the risk term increases significantly and dominates the decision, and the system automatically switches to the more robust full mode, thereby avoiding the accuracy collapse caused by forced segmentation or low coverage reuse.

[0100] like Figure 4 As shown in b, during the cache compression stage, the uncertainty index U determines the direction and steepness of the retention probability curve. High uncertainty corresponds to a more conservative curve shape, meaning a higher retention probability for the same contribution; low uncertainty corresponds to a more aggressive curve shape, meaning only high-contribution entries are retained and low-contribution entries are merged more aggressively. This U-based nonlinear adjustment mechanism adaptively aligns the cache compression intensity with the model perplexity level.

[0101] Figure 4 Figure b also illustrates the upper bound reference line for the error constraint. The system does not blindly pursue the compression rate, but rather releases as much video memory as possible while ensuring that the error constraint is met; when the error detection exceeds the limit, the system uses a rollback strategy to "pull" the curve back to a more conservative region, thereby forming a closed-loop quality control.

[0102] In summary, this application proposes a segmented hybrid inference method for large language models driven by uncertainty. It introduces an uncertainty index as a "global control valve" and information contribution as a "local retention criterion." Under a unified objective function, it dynamically selects the inference mode. Simultaneously, during the cache merging stage, it adaptively generates merging control probabilities based on the uncertainty index and information contribution, and forms a closed-loop verification and backoff adjustment through an error constraint module. Therefore, this application offers the following advantages: First, by introducing the uncertainty index as the core control variable for unified scheduling, it can automatically reduce compression intensity and favor robust inference modes when the model is "confused," thereby significantly reducing the risk of accuracy collapse caused by fixed strategies and improving robustness and stability. Second, by constructing a unified objective function that includes memory, latency, and risk, it places "segmented inference mode selection" and "cache compression intensity" under the same decision semantics, avoiding the error explosion caused by the simple superposition of existing segmentation and compression schemes. Through error upper bound constraints and a backoff mechanism, an interpretable closed-loop quality control path is formed in long-sequence inference, enabling the system to adaptively achieve a balance between "lower memory usage, lower latency, and controllable accuracy loss" under different hardware environments and different business objectives. Fourth, a joint driving mechanism of information contribution and uncertainty index is introduced into cache merging. Compared with existing techniques that rely solely on attention score (contribution) for compression, this application uses the uncertainty index to dynamically adjust the threshold for determining information contribution (e.g., increasing the retention probability curve under high uncertainty). This makes the compression strategy no longer "blind" but able to perceive the current "level of confusion" of the model, thereby achieving safer and more efficient memory compression while ensuring that key context is not lost.

[0103] To implement the above embodiments, this invention also proposes a computer-readable storage medium storing a large language model segmented hybrid inference program based on uncertainty-driven inference. When executed by a processor, this large language model segmented hybrid inference program based on uncertainty-driven inference implements the large language model segmented hybrid inference method based on uncertainty-driven inference as described above.

[0104] In addition, this invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described uncertain-driven large language model segmented hybrid reasoning method.

[0105] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0110] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0111] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0112] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0113] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0114] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0115] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0116] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A segmented hybrid reasoning method based on an uncertain-driven large language model, characterized in that, Includes the following steps: The text data and historical state features of the current input request are obtained, and the uncertainty index of the current segment is estimated based on the historical state features. The historical state features include the attention distribution, hidden state statistical features and system operation state features of the previous inference time step, or feature vectors calculated by a lightweight prior network. A unified scheduling objective function is constructed, which includes memory usage cost, inference latency cost, and uncertainty risk. The unified scheduling objective function is minimized according to the uncertainty index to select the target inference mode for the current segment from a preset set of inference modes. The set of inference modes includes full inference, hybrid inference, and incremental inference. Based on the target reasoning pattern, perform reasoning calculations on the text data of the current segment, generate key-value pairs, and calculate the information contribution of each generated key-value pair; Based on the uncertainty index and the information contribution, the dynamic merging control probability of each key-value pair is calculated, and weighted merging or pruning is performed on the key-value pairs to be merged according to the sampling or deterministic selection results to obtain compressed key-value pairs. Define a deviation metric between the compressed output distribution and the full inference output distribution, and limit the deviation metric to no more than a preset upper bound determined by the dynamic merging control probability set and the uncertainty index; When the deviation metric exceeds the preset upper bound, a rollback process is triggered to adjust the merging strength and / or scheduling weight; when the deviation metric does not exceed the preset upper bound, the compressed key-value pair status is fed back to the next segment for iterative iteration until the inference of all segments is completed. The uncertainty metric is one or more of attention entropy, logit entropy, top-k margin, or self-consistency divergence. The unified scheduling objective function is obtained according to the following formula: Where m represents the candidate inference mode, VRAM(m) represents the GPU memory usage when executing mode m, Latency(m) represents the inference latency, and Risk(m) represents the risk measure caused by low precision or low context coverage. , Represents the normalization constant. , , Indicates the weighting coefficient. Indicators representing uncertainty.

2. The segmented hybrid reasoning method based on an uncertain-driven large language model as described in claim 1, characterized in that, The information contribution is calculated by accumulating the attention weight, gradient approximation sensitivity, or information gain of the i-th token in the current segment and subsequent segments, and is used to characterize the degree of influence of key-value pairs on subsequent inference output.

3. The segmented hybrid reasoning method based on an uncertain-driven large language model as described in claim 1, characterized in that, The dynamic merging control probability is obtained using the following formula: in, This represents the Sigmoid function. Indicates the degree of information contribution. Indicators representing uncertainty This prevents positive numbers with a denominator of zero. This indicates an adjustable parameter; and, The symbols and values ​​are used to set and The monotonous relationship.

4. The segmented hybrid reasoning method based on an uncertain-driven large language model as described in claim 1, characterized in that, The Kullback–Leibler divergence or Jensen–Shannon divergence is used to measure the deviation between the compressed output distribution and the full inference output distribution.

5. The segmented hybrid reasoning method based on an uncertain-driven large language model as described in claim 1, characterized in that, When a deviation metric is detected to exceed a preset threshold, the weight of the uncertainty risk term in the unified scheduling objective function is increased. This triggers a switching of inference modes to favor the more robust full-scale inference mode.

6. A computer-readable storage medium, characterized in that, It stores a large language model segmented hybrid inference program based on uncertainty-driven approach. When the processor executes the large language model segmented hybrid inference program based on uncertainty-driven approach, it implements the large language model segmented hybrid inference method based on uncertainty-driven approach as described in any one of claims 1-5.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the segmented hybrid reasoning method based on an uncertain-driven large language model as described in any one of claims 1-5.

Images