Long context reasoning query vector adaptive regulation and control method and system
By introducing an attention degradation awareness mechanism and adaptive query vector control into the large language model, the problem of attention degradation in long context reasoning is solved, the visibility of key information and reasoning efficiency are improved, and more stable reasoning results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING KNOWLEDGE ATLAS TECHNOLOGY CO LTD
- Filing Date
- 2026-01-19
- Publication Date
- 2026-04-28
AI Technical Summary
Existing large language models suffer from attention degradation in long context reasoning scenarios, lacking a mechanism for perceiving and judging key information, resulting in unstable reasoning performance and low efficiency.
An attention degradation perception mechanism is introduced. By analyzing the attention distribution state, it is determined whether degradation has occurred. When degradation is detected, the query vector is adaptively adjusted to adjust the model's attention to long contextual information.
It improves the ability to focus key information and the efficiency of reasoning in long contexts, and ensures the stability and reliability of reasoning results, making it suitable for application scenarios that are sensitive to latency and computing power.
Smart Images
Figure CN121936600A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology and relates to an adaptive control method and system for long context reasoning query vectors, particularly an adaptive control method and system for long context reasoning query vectors based on attention degradation perception. Background Technology
[0002] With the widespread application of Large Language Models (LLMs) in tasks such as long document understanding, multi-document question answering, code analysis, and complex reasoning, LLMs need to accurately locate and effectively utilize key content in scenarios containing a large amount of contextual information. As a core component supporting the contextual modeling and reasoning capabilities of LLMs, the effectiveness of self-attention mechanisms under long context conditions directly determines the upper limit of LLM reasoning performance. However, in practical applications, as the length of the input context continues to increase, existing reasoning mechanisms still face many challenges in attention allocation and focusing on key information.
[0003] In recent years, research on the performance degradation of large language models in long context reasoning scenarios can be mainly summarized into the following technical approaches: 1. Methods based on context expansion or window optimization.
[0004] These methods enable the model to handle longer input sequences by increasing the context length supported by large language models and employing strategies such as sliding windows or segmented inference. Although they formally expand the model's context capacity, they still rely on a fixed self-attention allocation mechanism and fail to solve the problem of key information being overwhelmed by a large amount of irrelevant or weakly relevant information in large-scale contexts. Under long context conditions, the model is still prone to "key information not visible" or inference failure.
[0005] 2. Enhancement methods based on repeated decoding or multi-round inference.
[0006] These methods attempt to improve a model's ability to understand complex problems by increasing the number of inference steps, introducing multiple rounds of inference, or generating more intermediate inference content. However, these methods mainly rely on repeated use of the same attention distribution. When the model's attention has already become scattered or degraded in the initial stage, subsequent repeated decoding cannot fundamentally improve the model's ability to focus on key information. On the contrary, it may amplify the influence of noisy information, leading to a lengthy and inefficient inference process.
[0007] 3. Methods based on static optimization of model structure or parameters.
[0008] These methods statically optimize the model through model pruning, parameter rearrangement, or attention structure adjustment to improve its performance in long-context scenarios. However, these methods typically perform optimization offline, lacking real-time awareness of the attention distribution during inference. They struggle to dynamically adjust the model's focus based on the contextual features of the specific input, potentially leading to attention degradation issues during actual inference.
[0009] 4. Adaptation method based on parameter updates during the inference phase.
[0010] In recent years, some studies have attempted to update model parameters in a limited manner during the inference phase to improve the model's adaptability to the current input. Although such methods have improved the model's inference performance in specific scenarios to some extent, they generally lack an explicit mechanism for determining the state of attention distribution. They usually default to performing parameter adjustments in all input scenarios, making it difficult to distinguish "whether adjustment is needed" and "how adjustment should be made," resulting in unnecessary parameter perturbations and insufficient inference stability.
[0011] In summary, while existing long-context reasoning enhancement schemes have alleviated the problem of model performance degradation to some extent, they still generally suffer from the following key technical shortcomings: (1) Lack of perception and judgment mechanism for attention degradation state: Existing methods mostly perform inference or parameter adjustment under fixed strategies, and fail to actively detect the degradation state of attention distribution inside the model, making it difficult to identify the situation where key information is buried in time.
[0012] (2) Lack of refined control methods for attention degradation: Even if some methods introduce parameter updates during the inference stage, their control objects and methods are still relatively crude, failing to make targeted control on key components such as query vectors, making it difficult to effectively restore attention discrimination without changing the overall structure of the model.
[0013] (3) Insufficient stability and reliability of long context reasoning: In ultra-long context or complex reasoning tasks, the model is prone to problems such as unstable reasoning results and omission of key information due to distraction, which affects the controllability and credibility of the model in practical applications.
[0014] Given the aforementioned shortcomings of existing technologies, there is an urgent need to research a novel long-context reasoning enhancement scheme. Summary of the Invention
[0015] To overcome the shortcomings of existing technologies, this invention proposes an adaptive adjustment method and system for query vectors in long context reasoning. It introduces an attention degradation perception mechanism during the model reasoning stage to analyze and determine the attention distribution state within the model. When attention degradation is detected, the query vector is adaptively adjusted in a targeted manner to restore the model's ability to focus on key information and improve the performance of long context reasoning.
[0016] To achieve the above objectives, the present invention provides the following technical solution: An adaptive adjustment method for long-context reasoning query vectors, characterized by the following steps: S1: Obtain the inference task input containing long context information, and send the inference task input into the large language model to perform inference; S2: During the reasoning process of the large language model, analyze the attention weight distribution of the self-attention module to obtain the attention distribution state of the current reasoning stage; S3: Based on the attention distribution state, determine whether the attention in the current reasoning stage has degenerated; S4: If attention does not degenerate during the current inference phase, continue inference; if attention degenerates during the current inference phase, trigger the query vector adaptive control mechanism. S5: After triggering the query vector adaptive adjustment mechanism, based on the attention distribution state, the relevant parameters used to generate the query vector are subject to constrained adaptive adjustment to guide the large language model to adjust its attention to long contextual information. S6: Based on the adjusted query vector, continue to perform inference of the large language model and generate the corresponding inference results.
[0017] Preferably, step S2 specifically includes: S21: Calculate the attention score and obtain the attention weight based on the attention score; S22: Calculate the quantitative index of attention distribution state based on the attention weights; S23: Perform cross-attention head and cross-layer summarization on the quantitative indicators of the attention distribution state to obtain the attention distribution state of the current inference stage.
[0018] Preferably, step S21 specifically includes: S211: Calculate the first... of the input to the reasoning task. The token is paired with the first Attention score per token: , In the formula, It is the first The first layer The first one in the attention. The token for the first Attention score for each token It is the first The first layer The first one in the attention. A query vector for each token. It is the first The first layer The first one in the attention. The key-value pair of the token. yes and dot product operation, It is the subspace dimension of the attention head. It is the first The token for the first The relative position offset of each token; S212: Obtaining attention weights based on attention scoring: , In the formula, It is the first The first layer The first one in the attention. The token is assigned to the first Attention weight of each token Yes Perform exponentiation. The length of the input to the reasoning task, that is, the total number of inputs to the reasoning task. A token.
[0019] Preferably, the attention distribution state quantification index in step S22 includes distribution entropy, Top-order distribution, and other metrics. Concentration and peak weight, and specifically includes: S221: Calculate the distribution entropy: , In the formula, It is the first The first layer The first one in the attention. The distribution entropy of each token. Furthermore, it is a numerically stable term; S222: Calculate Top- Concentration: , In the formula, For the first The first layer The first one in the attention. Top- of tokens Concentration To get the largest indivual The set of indices; S223: Calculate peak weight: , In the formula, For the first The first layer The first one in the attention. Peak weight of each token.
[0020] Preferably, the attention distribution state in the current inference stage of step S23 includes , and / or ,in, , , , In the formula, For the first The overall distribution entropy of each token For the first Top-total number of tokens Concentration For the first The overall peak weight of each token This represents the total number of layers in the self-attention module. The number of attention heads per layer.
[0021] Preferably, in step S3, attention degradation in the current reasoning stage is determined when at least one of the following conditions is met: Exceeding the preset threshold Below the preset threshold and It is below the preset threshold.
[0022] Preferably, in step S5, the constrained adaptive adjustment of the relevant parameters used to generate the query vector specifically involves applying an amplification factor to the original query vector to enhance its role in attention calculation.
[0023] Furthermore, the present invention also provides an adaptive control system for long context reasoning query vectors, characterized in that it includes: The reasoning task input acquisition module is used to acquire the reasoning task input containing long contextual information and send the reasoning task input into the large language model to perform reasoning. The attention distribution state acquisition module is used to analyze the attention weight distribution of the self-attention module and obtain the attention distribution state of the current inference stage during the reasoning process of the large language model. The attention degradation perception and judgment module is used to determine whether attention degradation has occurred in the current reasoning stage based on the attention distribution state. The query vector adaptive control mechanism trigger module is used to trigger the query vector adaptive control mechanism when attention degrades in the current inference stage. The query vector adaptive control module based on the degenerate state is used to perform constrained adaptive control of the relevant parameters used to generate the query vector according to the attention distribution state after the query vector adaptive control mechanism is triggered, so as to guide the large language model to adjust its attention to long context information. The continued reasoning module is used to continue the reasoning of the large language model based on the regulated query vector and generate the corresponding reasoning results.
[0024] Furthermore, the present invention also provides a long context reasoning query vector adaptive adjustment device, characterized in that it includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the long context reasoning query vector adaptive control method as described above. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the steps of the long context reasoning query vector adaptive control method as described above.
[0025] Compared with existing long-context reasoning enhancement methods that rely solely on context expansion, repeated decoding, static parameter optimization, or unified parameter updates during the inference phase, the long-context reasoning query vector adaptive adjustment method and system proposed in this invention, based on attention degradation awareness, has the following significant advantages in terms of key information focusing ability, inference efficiency, and inference stability: 1. Adaptive regulation based on attention degradation perception fundamentally alleviates the problem of loss of key information in long contexts.
[0026] Existing methods typically employ fixed inference strategies during long-context reasoning, lacking real-time awareness of the model's internal attention distribution. Even when attention is clearly scattered, reasoning continues in a predetermined manner, resulting in key information being overwhelmed by a large amount of irrelevant context. This invention introduces an attention degradation awareness mechanism to analyze the concentration and discriminative power of attention distribution during reasoning. Upon detecting attention degradation, it proactively triggers an adaptive adjustment mechanism for the query vector, guiding the model to refocus on contextual information highly relevant to the reasoning objective. This improves the visibility and utilization efficiency of key information from a mechanism perspective.
[0027] 2. Efficient control without adding reasoning steps significantly improves the efficiency of reasoning in long contexts.
[0028] Unlike enhancement methods that rely on adding inference steps or generating more intermediate inference content, this invention directly improves the model's attention to contextual information by adaptively adjusting the query vector without increasing the number of decoding steps or introducing additional inference chains. This effectively reduces computational complexity and inference latency while ensuring inference accuracy, avoids invalid computation caused by repeated decoding, and enables the model to maintain high inference efficiency and better energy efficiency in ultra-long context scenarios. It is suitable for application scenarios that are sensitive to latency and computing power.
[0029] 3. The stability of the reasoning process and the consistency of the results are significantly enhanced.
[0030] In long-context or complex reasoning tasks, existing methods are prone to problems such as fluctuating reasoning results, missing key information, or logical incoherence due to attentional distraction, affecting the stability and reliability of the reasoning results. This invention uses a collaborative mechanism of "attention degradation perception - query vector regulation" to intervene in the reasoning process in a targeted manner, reducing reasoning deviations caused by attentional anomalies, improving the consistency and robustness of the model in multi-stage reasoning processes, and enabling the model to maintain stable and controllable reasoning performance under different context scales and task conditions. Attached Figure Description
[0031] Figure 1 This is a flowchart of the long context reasoning query vector adaptive adjustment method of the present invention.
[0032] Figure 2 This is a schematic diagram of the adaptive control system for long context reasoning query vectors of the present invention.
[0033] Figure 3 This is a structural block diagram of the long context reasoning query vector adaptive control device of the present invention. Detailed Implementation
[0034] Before detailing any embodiment of the invention, it should be understood that the invention, in its application, is not limited to the details of the construction and arrangement of the components set forth in the following description or illustrated in the following figures. The invention can have other embodiments and can be practiced or carried out in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising” or “having” and variations thereof in this invention is intended to cover the items set forth below and their equivalents, as well as any additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “linkage,” and variations thereof are used broadly and cover both direct and indirect installation, connection, support, and linking. Moreover, “connection” and “linkage” are not limited to physical or mechanical connections or links. Furthermore, firstly, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention. Secondly, the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. The term "a" should not be construed as a limitation on the quantity.
[0035] As large language models are increasingly used in long-context understanding and complex reasoning tasks, these models need to accurately locate and utilize key content in scenarios containing a large amount of contextual information. However, limited by existing model reasoning mechanisms and attention allocation methods, the following prominent technical problems still exist in long-context reasoning: (1) Attention allocation degrades severely in long context scenarios.
[0036] As the scale of the input context continues to increase, the model's internal attention mechanism needs to globally distribute among a large number of context labels. Key labels are easily overwhelmed by a large number of semantically similar or irrelevant labels, resulting in a scattered attention distribution. The model struggles to effectively focus on information that is highly relevant to the reasoning goal, leading to problems such as "key information not visible" or "intermediate information lost".
[0037] (2) Existing reasoning methods lack the ability to perceive and judge attention degradation.
[0038] Existing large language models typically employ fixed reasoning processes and parameter configurations during the reasoning phase, making it impossible to perceive and evaluate the distribution of attention in real time. When attention has significantly deteriorated, the model continues to reason in the predetermined manner, lacking means to identify and intervene in abnormal attention states, resulting in a significant decrease in reasoning performance as the length of the context increases.
[0039] (3) The reasoning stage lacks an adaptive regulation mechanism for attention degradation.
[0040] Even when attention degradation has already occurred, existing technologies still mainly rely on increasing inference steps or repeating decoding to try to improve inference performance. However, such methods cannot fundamentally improve the model's ability to focus on key information. On the contrary, they may further amplify the impact of noise information. There is a lack of direct control measures for attention degradation, making it difficult to guarantee the stability and reliability of long context inference.
[0041] To address the aforementioned issues, this invention proposes an adaptive adjustment method and system for query vectors in long context reasoning based on attention degradation perception. This method introduces an attention degradation perception mechanism during the model reasoning stage to analyze and determine the attention distribution state within the model. Upon detecting attention degradation, it adaptively adjusts the query vector accordingly, thereby restoring the model's ability to focus on key information and improving the performance of long context reasoning.
[0042] Specifically, the present invention achieves an effective improvement in the performance of long context reasoning by: when the attention distribution is concentrated and the reasoning state is normal, the original reasoning mechanism of the model remains unchanged to avoid unnecessary parameter perturbations; when a significant degradation feature of the attention distribution is detected, the query vector adaptive adjustment mechanism is activated to adjust the focus of the model and guide the attention to refocus on the context information related to the reasoning target.
[0043] This invention achieves performance breakthroughs primarily through the following three aspects: (1) Introduce the attention degradation perception mechanism.
[0044] During the reasoning process, the internal attention distribution of the model is analyzed to identify degradation features such as distraction and reduced discrimination, thereby enabling proactive perception and judgment of abnormal attention states in long-context reasoning.
[0045] (2) Adaptive control of query vector based on degenerate state.
[0046] After detecting attention degradation, only the relevant parameters used to generate query vectors are adaptively adjusted to adjust the model's attention to contextual information, effectively improving the visibility and discriminability of key information without changing the overall structure of the model.
[0047] (3) Improvement of stability and effectiveness of long context reasoning.
[0048] By leveraging the synergistic effect of attention degradation perception and query vector manipulation, the risk of key information being overwhelmed in long context scenarios is reduced, improving the accuracy and stability of model inference results and enabling the model to exhibit higher reliability and robustness in complex, ultra-long context reasoning tasks.
[0049] Figure 1 A flowchart of the long context reasoning query vector adaptive adjustment method of the present invention is shown. Figure 1 As shown, the adaptive adjustment method for long context reasoning query vectors of the present invention includes the following steps: S1: Input acquisition for inference task.
[0050] Obtain the inference task input containing long contextual information, and send the inference task input into the large language model to perform inference.
[0051] The inference task input is represented as a sequence of lengths of... The input representation consists of a sequence, that is, the inference task input has a total of Each token. During the reasoning process of the large language model, the input sequence is mapped to the hidden representation of each layer of the model, which serves as the basic input for subsequent attention calculations, thereby providing the internal representation of the model for attention activation and state analysis in the reasoning stage.
[0052] S2: Attention distribution state acquisition.
[0053] During the reasoning process of the large language model, the attention weight distribution of the self-attention module is analyzed to obtain the attention distribution state of the current reasoning stage, which is used to evaluate the large language model's attention to long contextual information.
[0054] In the specific implementation process, let the length of the input sequence be... That is, the inference task input has a total of There are [number] tokens, and the self-attention module of the large language model has [number] tokens. Layers and each layer has One point of attention, It is the next level (that is, the first) The output of the layer has passed through the first layer. The result after layer self-attention + FFN.
[0055] .
[0056] The hidden dimension of each layer is: , No. The first layer A query vector with attention heads. Key Sum (Value) for: .
[0057] in, , and They are the first The first layer The query vector parameter matrix, key parameter matrix, and value parameter matrix of the attention head are used to... Layer output ( (Dimension) projected onto the subspace of the attention head ( (dimensional), that is, to dimensional vector mapping to Dimensional vector. , That is to say, The dimension is ,and The dimension is The essence of the above formula is linear projection, that is, each attention head is projected through an independent parameter matrix. , and , convert the original sequence of 3D feature vector , respectively mapped to Vie , , .
[0058] For example, if the length of the input sequence is... Hidden Dimensions Attention count Then the subspace of each attention head ;at this time yes The matrix, multiplied by ( After that, the result was... yes The matrix, that is, each sequence (token) corresponds to a 16-dimensional query vector.
[0059] 1. Calculate the attention score and obtain the attention weight based on the attention score.
[0060] Calculate the current position of the input to the inference task, for example, the position of the input. The token is paired with other locations, such as the first one. Attention score per token: , In the formula, It is the first The first layer The first one in the attention. The token for the first Attention score for each token; It is the first The first layer The first one in the attention. The query vector of each token is The OK; It is the first The first layer The first one in the attention. The key-value pair of the token is The OK; yes and Dot product operation; It is the subspace dimension of the attention head; It is the first The token for the first The relative position offset of each token functions similarly to a masking mechanism, primarily to exclude invalid contexts. For example, if the first token's position is offset... If the token is an "illegal context" (such as a "future token" in a generative task, or a PAD token used for padding), then... Set it to a very small negative number so that the score at that position is ignored by softmax.
[0061] Then, the attention weights (i.e., attention distribution) are obtained through softmax: , In the formula, It is the first The first layer The first one in the attention. The token is assigned to the first Attention weight of each token Yes Perform exponentiation. The length of the input to the reasoning task, that is, the total number of inputs to the reasoning task. A token.
[0062] Will It is considered as a probability distribution over context positions and used for subsequent state analysis.
[0063] 2. Calculate the quantitative index of attention distribution state based on the attention weights.
[0064] To evaluate the attention paid to contextual information by a large language model, one or more of the following quantitative indicators of attention distribution state can be calculated for the attention weights: (1) Distribution entropy: , In the formula, It is the first The first layer The first one in the attention. The distribution entropy of each token. Furthermore, it is a numerically stable term.
[0065] Distribution entropy is used to measure whether attention is scattered; the higher the distribution entropy, the more scattered the attention.
[0066] (2) Top- Concentration: , In the formula, For the first The first layer The first one in the attention. Top- of tokens Concentration To get the largest indivual The set of indices, that is, all of them Arrange from largest to smallest and take the first few. indivual Adding them together gives .
[0067] Top- Concentration is used to measure whether attention is focused, Top- The higher the concentration, the more focused the attention is on a few key locations.
[0068] (3) Peak weight: , In the formula, For the first The first layer The first one in the attention. Peak weight of each token.
[0069] Peak weight is used to measure whether the most important point is prominent enough. The smaller the peak weight, the less obvious the focus, and the more likely key information is to be buried.
[0070] 3. The attention distribution state quantification index is summarized across attention heads and layers to obtain the attention distribution state of the current inference stage.
[0071] The above-mentioned quantitative indicators of attention distribution can be summarized at the attention head and layer to form the overall state quantity of the inference stage, that is, the attention distribution state of the current inference stage: , , , In the formula, For the first The overall distribution entropy of each token For the first Top-total number of tokens Concentration For the first The overall peak weight of each token This represents the total number of layers in the self-attention module. The number of attention heads per layer.
[0072] Can , , One or more of these are used as the attention distribution state in the current reasoning phase for attention degradation determination.
[0073] S3: Perception and judgment of attentional degradation.
[0074] Based on the attention distribution state, determine whether attention has degenerated in the current reasoning stage.
[0075] Specifically, regarding the current position, i.e., the [number]th position... Attention weight distribution for each token The quantitative index of attention distribution state calculated based on its distribution characteristics , , One or more of the following are used to identify attention degradation. Attention degradation in the current reasoning stage is determined to occur when at least one of the following conditions is met: (1) Dispersion index of attention distribution Exceeding the preset threshold indicates that attention is excessively scattered across contextual locations and lacks effective focus; (2) Concentration index of attention distribution A reading below a preset threshold indicates that attention has not been focused on a small number of key contextual locations; (3) Peak weights of attention distribution If the value is below the preset threshold, it indicates that the location corresponding to the key information has not received sufficient attention weight.
[0076] Each preset threshold is a hyperparameter that can be set and adjusted according to the actual situation.
[0077] S4: Query vector adaptive control mechanism triggered.
[0078] If attention does not degrade during the current inference phase, query vector manipulation is not performed. The original inference parameters of the large language model remain unchanged, and subsequent inference results are generated according to the established inference process.
[0079] When attention degrades during the current inference phase, an adaptive adjustment mechanism for the query vector is triggered to adjust and prepare the relevant parameters used to generate the query vector.
[0080] S5: Adaptive adjustment of query vectors based on degenerate state.
[0081] After the query vector adaptive adjustment mechanism is triggered, the relevant parameters used to generate the query vector are subject to constrained adaptive adjustment based on the attention distribution state, so as to guide the large language model to adjust its attention to long contextual information.
[0082] Specifically, the query vector adaptive modulation updates the parameters used to generate query vectors in the large language model, and determines the modulation range and intensity based on the degree of attention degradation. The modulation can be selectively updated across different network layers and / or different attention heads to enhance the attention weights on contextual information relevant to the inference goal, thereby improving the visibility and discriminative power of key information.
[0083] In one specific implementation, adaptive query vector control is achieved by applying a scaling operation to the query vector related to the degree of attention degradation. When attention degradation is detected, an amplification factor is applied to the original query vector to enhance its role in attention calculation; when attention degradation is not detected, the query vector remains unchanged, thereby improving the ability to focus on key information while ensuring inference stability. For example, , in, It is the control coefficient (hyperparameter, the specific value of which can be set and adjusted as needed).
[0084] When attention is not degraded, let ,but = (The query vector remains completely unchanged); when attention degradation is detected, let ,but =(1+ ) (The query vector is enhanced).
[0085] S6: Continue reasoning.
[0086] Based on the adjusted query vector, the large language model continues to perform inference, generating the corresponding inference results.
[0087] After completing the adaptive adjustment of the query vector in step S5, the reasoning process of the large language model continues to be executed based on the adjusted query vector, attention calculation and information fusion are completed, and the corresponding reasoning result is generated. The reasoning result is used as the output result of the current long context reasoning task.
[0088] Figure 2 A schematic diagram of the adaptive control system for long context reasoning query vectors of the present invention is shown. Figure 2 As shown, the long context reasoning query vector adaptive control system of the present invention includes: 1. Input Acquisition Module for Inference Tasks.
[0089] The inference task input acquisition module is used to acquire inference task input containing long contextual information and send the inference task input to the large language model to perform inference.
[0090] 2. Attention distribution state acquisition module.
[0091] The attention distribution state acquisition module is used to analyze the attention weight distribution of the self-attention module during the reasoning process of the large language model, and obtain the attention distribution state of the current reasoning stage.
[0092] 3. Attention deterioration perception and judgment module.
[0093] The attention degradation perception and determination module is used to determine whether attention degradation has occurred in the current reasoning stage based on the attention distribution state.
[0094] 4. Query vector adaptive control mechanism trigger module.
[0095] The query vector adaptive control mechanism trigger module is used to trigger the query vector adaptive control mechanism when attention degrades during the current inference phase.
[0096] 5. Query vector adaptive adjustment module based on degenerate state.
[0097] The query vector adaptive control module based on the degradation state is used to perform constrained adaptive control of the relevant parameters used to generate the query vector according to the attention distribution state after the query vector adaptive control mechanism is triggered, so as to guide the large language model to adjust its attention to long contextual information.
[0098] 6. Continue with the reasoning module.
[0099] The continued reasoning module is used to continue the reasoning of the large language model based on the adjusted query vector and generate the corresponding reasoning results.
[0100] Furthermore, this invention also provides an adaptive adjustment device for long contextual reasoning query vectors. For example... Figure 3 As shown, the long context reasoning query vector adaptive control device of the present invention includes: a memory 11 for storing one or more programs; one or more processors 12; when the one or more programs are executed by the one or more processors 12, the one or more processors 12 implement the long context reasoning query vector adaptive control method of the present invention. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the long context reasoning query vector adaptive control method of the present invention.
[0101] The computer-readable storage medium includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in this invention, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0102] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0103] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this invention can be implemented in hardware, software modules executed by a processor, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.
[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention based on the concept of the present invention, without departing from the essence and scope of the technical solutions of the present invention.
Claims
1. A method for adaptive adjustment of query vectors in long contextual reasoning, characterized in that, Includes the following steps: S1: Obtain the inference task input containing long context information, and send the inference task input into the large language model to perform inference; S2: During the reasoning process of the large language model, analyze the attention weight distribution of the self-attention module to obtain the attention distribution state of the current reasoning stage; S3: Based on the attention distribution state, determine whether the attention in the current reasoning stage has degenerated; S4: If attention does not degenerate during the current inference phase, continue inference; if attention degenerates during the current inference phase, trigger the query vector adaptive control mechanism. S5: After triggering the query vector adaptive adjustment mechanism, based on the attention distribution state, the relevant parameters used to generate the query vector are subject to constrained adaptive adjustment to guide the large language model to adjust its attention to long contextual information. S6: Based on the adjusted query vector, continue to perform inference of the large language model and generate the corresponding inference results.
2. The adaptive adjustment method for long context reasoning query vectors according to claim 1, characterized in that, Step S2 specifically includes: S21: Calculate the attention score and obtain the attention weight based on the attention score; S22: Calculate the quantitative index of attention distribution state based on the attention weights; S23: Perform cross-attention head and cross-layer summarization on the quantitative indicators of the attention distribution state to obtain the attention distribution state of the current inference stage.
3. The adaptive adjustment method for long context reasoning query vectors according to claim 2, characterized in that, Step S21 specifically includes: S211: Calculate the first input of the reasoning task. The token is paired with the first... Attention score per token: ， In the formula, It is the first The first layer The first one in the attention. The token for the first Attention score for each token It is the first The first layer The first one in the attention. A query vector for each token. It is the first The first layer The first one in the attention. The key-value pair of the token yes and dot product operation, It is the subspace dimension of the attention head. It is the first The token for the first The relative position offset of each token; S212: Obtaining attention weights based on attention scoring: ， In the formula, It is the first The first layer The first one in the attention. The token is assigned to the first Attention weight of each token Yes Perform exponentiation. The length of the input to the reasoning task, that is, the total number of inputs to the reasoning task. A token.
4. The adaptive adjustment method for long context reasoning query vectors according to claim 3, characterized in that, The attention distribution state quantification indicators in step S22 include distribution entropy, Top-level... Concentration and peak weight, and specifically includes: S221: Calculate the distribution entropy: ， In the formula, It is the first The first layer The first one in the attention. The distribution entropy of each token. Furthermore, it is a numerically stable term; S222: Calculate Top- Concentration: ， In the formula, For the first The first layer The first one in the attention. Top- of tokens Concentration To get the largest indivual The set of indices; S223: Calculate peak weight: ， In the formula, For the first The first layer The first one in the attention. Peak weight of each token.
5. The adaptive adjustment method for long context reasoning query vectors according to claim 4, characterized in that, The attention distribution state in the current inference stage in step S23 includes , and / or ,in, ， ， ， In the formula, For the first The overall distribution entropy of each token For the first Top-total number of tokens Concentration For the first The overall peak weight of each token This represents the total number of layers in the self-attention module. The number of attention heads per layer.
6. The adaptive adjustment method for long context reasoning query vectors according to claim 5, characterized in that, In step S3, attention degradation in the current reasoning stage is determined when at least one of the following conditions is met: Exceeding the preset threshold Below the preset threshold and It is below the preset threshold.
7. The adaptive adjustment method for long contextual reasoning query vectors according to any one of claims 1-6, characterized in that, In step S5, the constrained adaptive adjustment of the relevant parameters used to generate the query vector specifically involves applying an amplification factor to the original query vector to enhance its role in attention calculation.
8. A long context reasoning query vector adaptive control system, characterized in that, include: The reasoning task input acquisition module is used to acquire the reasoning task input containing long contextual information and send the reasoning task input into the large language model to perform reasoning. The attention distribution state acquisition module is used to analyze the attention weight distribution of the self-attention module and obtain the attention distribution state of the current inference stage during the reasoning process of the large language model. The attention degradation perception and judgment module is used to determine whether attention degradation has occurred in the current reasoning stage based on the attention distribution state. The query vector adaptive control mechanism trigger module is used to trigger the query vector adaptive control mechanism when attention degrades in the current inference stage. The query vector adaptive control module based on the degenerate state is used to perform constrained adaptive control of the relevant parameters used to generate the query vector according to the attention distribution state after the query vector adaptive control mechanism is triggered, so as to guide the large language model to adjust its attention to long context information. The continued reasoning module is used to continue the reasoning of the large language model based on the regulated query vector and generate the corresponding reasoning results.
9. A device for adaptive adjustment of long contextual reasoning query vectors, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the long context reasoning query vector adaptive control method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the long context reasoning query vector adaptive control method as described in any one of claims 1-7.