A memory strategy optimization method, device, medium and program product of a long sequence intelligent agent

Abstract

The application belongs to the technical field of agent training, and particularly relates to a memory strategy optimization method, device, medium and program product of a long-time sequence agent. The application solves the problem that a memory compression link is wrongly rewarded or punished during end-to-end memory compression training, realizes fine-grained and low-entanglement credit distribution for each memory write, greatly improves credit distribution quality while maintaining high efficiency of compressed context, and has extremely small additional calculation overhead.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent agent training, and specifically relates to a method, device, medium and program product for optimizing the memory strategy of a long-term intelligent agent. Background Technology

[0002] With the development of Large Language Models (LLMs) and agent technologies, long-horizon agents have demonstrated outstanding performance in complex tasks such as web search, multi-hop question answering, embodied planning, and tool-enhanced inference. These agents complete tasks gradually through multiple interleaved rounds of "reasoning-tool invocation-environment feedback." However, if the entire interaction history is fully preserved, the context length of the prompt words increases linearly with the number of interaction rounds, significantly increasing the cost of inference tokens, exceeding the fixed context window length limit, and reducing model performance under long context conditions. Therefore, memory compression has become a key capability for efficient training and inference of long-horizon agents.

[0003] However, existing technologies suffer from causal entanglement between memory credit and errors in tools, observation, and reasoning during end-to-end memory compression training, leading to the problem of incorrect rewards and punishments in the memory compression process. Summary of the Invention

[0004] This application addresses the problem in existing technologies where memory credits are causally entangled with errors in tools, observations, and inferences during end-to-end memory compression training, leading to incorrect rewards and penalties in the memory compression process. It provides a method, device, medium, and program product for optimizing the memory strategy of long-term intelligent agents. This achieves fine-grained, low-entanglement credit allocation for each memory write, significantly improving the quality of credit allocation while maintaining high efficiency in the compression context, with minimal additional computational overhead.

[0005] To address the aforementioned technical problems, this application proposes four aspects.

[0006] In a first aspect, this application proposes a memory strategy optimization method for a long-term intelligent agent, comprising: during the process of the target intelligent agent answering a target question, performing the following operations for each step of the memory writing action in each attempt path: calculating the local counterfactual score between the current memory and the historical memory, wherein the current memory is the memory generated in the current step of the current attempt path, and the historical memory is the memory written in the previous step of the current step; calculating the current log-likelihood difference between the current memory and the sibling memory, wherein the sibling memory is the memory generated in other attempt paths corresponding to the step number of the current step; determining the original memory score of the current step based on the local counterfactual score and the current log-likelihood difference; optimizing the memory strategy of the target intelligent agent based on the original memory scores of each step in the completion path, wherein the completion path is the attempt path that ends the task of answering the target question, and the original memory score of each step is the original memory score of each step of the memory writing action.

[0007] In some embodiments, calculating the local counterfactual score between the current memory and the historical memory includes: obtaining the current prior state of the current step in the current attempt path, the current memory, the historical memory, and the correct answer to the target question; calculating a current answer score based on the current prior state, the current memory, and the correct answer, wherein the current answer score is used to characterize the probability that the target agent obtains the correct answer based on the current memory; calculating a historical answer score based on the current prior state, the historical memory, and the correct answer, wherein the historical answer score is used to characterize the probability that the target agent obtains the correct answer based on the historical memory; and determining the local counterfactual score based on the current answer score and the historical answer score, wherein the local counterfactual score is used to characterize the degree of closeness between the current memory and the correct answer compared to the historical memory.

[0008] In some embodiments, calculating the current log-likelihood difference between the current memory and its sibling memory includes: determining the current log-likelihood value of the current memory based on the current preceding state, the current memory, and the correct answer, wherein the current log-likelihood value is the average log-likelihood value of each word in the current memory; obtaining the average log-likelihood value of the sibling memory in each parallel path, wherein the parallel path is another attempted path that is in the same group as the current attempted path; and determining the current log-likelihood difference based on the current log-likelihood value and the average log-likelihood value of the sibling memory.

[0009] In some embodiments, determining the raw memory score for the current step based on the local counterfactual score and the current log-likelihood difference includes: determining an adaptive sharpness of the gating based on all log-likelihood differences, wherein all log-likelihood differences are all log-likelihood differences generated by the target agent during the training process of the current batch; determining the gating opening based on the adaptive sharpness, the current log-likelihood difference, and the local counterfactual score; and determining the raw memory score based on the gating opening and the local counterfactual score.

[0010] In some embodiments, optimizing the memory strategy of the target agent based on the original memory scores of the memory actions written at each step in the completion path includes: updating the original memory scores of each step in the successful path to obtain a completion memory score, wherein original memory scores less than zero are updated to zero, the completion path includes a successful path and a failed path, the successful path being an attempt path to obtain the correct answer to the target question; using the completion memory score of the last step in each completion path as a smoothed memory score; determining the smoothed memory score of the current step based on the smoothed memory score of the next step and the completion memory score of the current step; and optimizing the memory strategy based on the smoothed memory scores of each step in each completion path.

[0011] In some embodiments, optimizing the memory strategy based on the smooth memory score of each write memory operation in each of the completion paths includes: obtaining the final reward / penalty for the completion path; determining the trajectory-level score of each memory lexicon based on the final reward / penalty and the smooth memory score of each step; and optimizing the memory strategy based on the trajectory-level score of each memory lexicon.

[0012] In some embodiments, optimizing the memory strategy based on the trajectory-level score of each memory lexicon includes: determining candidate memory strategies based on the trajectory-level score of each memory lexicon; determining a pruning strategy for the candidate memory strategies based on the candidate memory strategies and the old memory strategies; and pruning the candidate memory strategies according to the pruning strategy to obtain an optimized memory strategy.

[0013] In a second aspect, this application proposes an electronic device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the methods described in the first aspect.

[0014] Thirdly, this application proposes a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0015] Fourthly, this application proposes a computer program product, including a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the method described in any one of the first aspects.

[0016] This application discloses a method for optimizing the memory strategy of a long-term intelligent agent, comprising: during the process of the target intelligent agent answering a target question, performing the following operations for each step of the memory writing action in each attempt path: calculating the local counterfactual score between the current memory and the historical memory, wherein the current memory is the memory generated in the current step of the current attempt path, and the historical memory is the memory written in the previous step of the current step; calculating the current log-likelihood difference between the current memory and the sibling memory, wherein the sibling memory is the memory generated in other attempt paths corresponding to the step number of the current step; determining the original memory score of the current step based on the local counterfactual score and the current log-likelihood difference; optimizing the memory strategy of the target intelligent agent based on the original memory scores of each step in the completion path, wherein the completion path is the attempt path that ends the task of answering the target question, and the original memory score of each step is the original memory score of each step of the memory writing action. This invention addresses the problem in existing technologies where memory credits are causally entangled with errors in tools, observations, and inferences during end-to-end memory compression training, leading to incorrect rewards and penalties in the memory compression process. It achieves fine-grained, low-entanglement credit allocation for each memory write, significantly improving the quality of credit allocation while maintaining high efficiency in the compression context, with minimal additional computational overhead. Attached Figure Description

[0017] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0018] Figure 1 The main flowchart of a memory strategy optimization method for a long-term intelligent agent provided in the embodiments of this application; Figure 2 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] With the development of Large Language Models (LLMs) and Agent technologies, Long-horizon Agents have demonstrated outstanding performance in complex tasks such as web search, multi-hop question answering, embodied planning, and tool-enhanced reasoning. These agents complete tasks step by step through multiple rounds of interleaved "reasoning-tool invocation-environment feedback" processes. However, if the entire interaction history is fully preserved, the context length of the prompt words will increase linearly with the number of interaction rounds, thus significantly increasing the cost of inference tokens, exceeding the upper limit of the fixed context window length, and reducing the model's performance under long context conditions.

[0020] Currently, existing technologies used to solve the above-mentioned technical problems can be mainly divided into the following three categories: 1) Memory-based methods using external retrieval. Examples include MemGPT, MemoryBank, Mem0, and A-MEM, which store historical interactions in external storage (vector databases, etc.) and retrieve relevant fragments based on semantic similarity when needed, incorporating them into the context. However, their access is based on the similarity between the question and the memory fragment, making it difficult to align with task-level goals, and the retrieval process often relies on a fixed workflow.

[0021] 2) Non-end-to-end memory methods based on summary compression. For example, ReSum and other schemes use independent summary models to compress the accumulated interaction history into compact reasoning states. These methods can reduce context length, but the optimization objective of the summary model is decoupled from the downstream task objective, making it impossible to determine whether a specific memory update should be responsible for the final result.

[0022] 3) Self-memory compression methods based on end-to-end policy learning. For example, SUPO integrates summary-based context management into multi-round reinforcement learning; MEM1 studies the synergy between memory and reasoning; MemPO explicitly introduces [this technology] into trajectories. <mem>The writing action employs a joint optimization strategy at the trajectory and memory levels. Such methods can learn memory compression for downstream performance, but their supervision signal remains primarily focused on the final result, making it impossible to distinguish whether a failure is caused by memory, tools, observation, or the reasoning process.

[0023] Therefore, existing technologies for memory compression have the following problems: 1) Causal Entanglement of Memory Credit. In end-to-end memory compression, the final trajectory reward is jointly determined by memory updates, tool invocation, environmental observation, and downstream inference. Directly propagating trajectory-level rewards to memory tokens leads to the problem of "causally entangled memory credit allocation." For example, when the tool returns incorrect evidence but the memory provides an accurate summary (the error originates from the tool), and when the tool returns correct evidence but the memory omits key facts (the error originates from the memory), both scenarios ultimately fail, but the required memory credit directions are exactly opposite, which existing outcome-oriented reward mechanisms cannot distinguish.

[0024] 2) Static evaluation lacks an incremental perspective. Existing memory reward methods mostly use the "compressed whole memory state / prefix" as the scoring object, giving a uniform signal to the whole memory, which cannot characterize the marginal information gain of a single write, making the training signal too coarse.

[0025] 3) Lack of counterfactual correction from a retrospective perspective. Relying solely on local scores of forward trajectories makes it difficult to eliminate confounding factors induced by tools / reasoning; existing post-credit methods are usually geared towards general actions or reasoning steps and do not construct specific post-relevance metrics and directional consistency gating mechanisms for memory writing. Therefore, they cannot suppress false credit induced by tools / reasoning while retaining sensitivity to real memory failures.

[0026] To address the aforementioned technical problems, this invention proposes a memory strategy optimization method for long-term intelligent agents. The implementation details of this method are described below. The following content is for illustrative purposes only and is not essential for implementing this solution.

[0027] Example 1: like Figure 1 As shown, this application provides a method for optimizing the memory strategy of a long-term intelligent agent. The method is applicable to electronic devices, such as servers, mobile terminals, computers, and cloud platforms. The data processing functionality of the production equipment provided in this application embodiment can be implemented by the processor of the electronic device calling program code, wherein the program code can be stored in a computer storage medium. The method for optimizing the memory strategy of the long-term intelligent agent includes: When a long-time agent is asked a question, it first considers the question, then uses tools to search for information, checks the search results, records the results, and then considers the question again based on the recorded results. After multiple rounds of this process, the long-time agent can finally arrive at the answer to the target question. In this process, because the long-time agent needs to record multiple search results, and the agent's contextual limit is limited, too many search results can lead to poor performance. Therefore, existing technologies avoid this problem by compressing and storing each search result, i.e., extracting key information. However, whether the extracted key information is relevant to answering the target question requires training.

[0028] Since the memory compression step cannot be trained separately, current technologies rely on the entire process—the "think-search-look-think again" process—to train the memory compression step. A result is then compared to the correct answer, and rewards or penalties are applied to the memory compression step. However, obtaining a correct or incorrect result involves multiple steps. For example, the memory compression step might correctly extract key points, but an error in the search tool leads to an incorrect result. In this case, the memory compression step should not be penalized. Similarly, if the memory compression step extracts incorrect key points, but the search step performs well, resulting in a correct result, the memory compression step should not be rewarded. Furthermore, the memory compression step performs multiple compression operations to obtain the result. How should rewards or penalties be allocated for each compression operation? These are technical problems that current technologies cannot solve.

[0029] Therefore, this application proposes a memory strategy optimization method for long-term intelligent agents. Since this application aims to clearly define the reward and penalty for each memory compression operation, the following operations are performed for each step of the memory writing action in each attempt path during the target agent's response to the target question: Step S1: Calculate the local counterfactual score between the current memory and the historical memory, wherein the current memory is the memory generated in the current step of the current attempt path, and the historical memory is the memory written in the previous step of the current step.

[0030] When a long-term intelligent agent searches based on its thinking results, multiple search results will inevitably appear. Each search result forms a memory, and new thinking can be generated based on a memory. Therefore, in the process of answering the target question, there will inevitably be multiple trial paths. Step S1 in this application is to execute the memory writing action at each step of each trial path. For ease of description, this application refers to the target trial path and the memory writing action in the trial path as the current trial path and the current step, respectively.

[0031] Since the current retrieval result of a long-term agent is generated based on the memory formed in the previous step, theoretically, the memory extracted from the current retrieval result will be closer to the correct answer to the target question than the memory formed in the previous step. Therefore, this application aims to solve the problem of how to allocate rewards and penalties to each step. It is necessary to first clarify the degree of progress of the memory between two adjacent steps. Of course, in reality, the current memory is not necessarily more advanced than the memory of the previous step, but may be further away from the correct answer. Therefore, this application needs to calculate the local counterfactual score between the current memory and the previous historical memory. The degree of progress between the current memory and the historical memory is determined by the local counterfactual score. In this application, for the sake of convenience, many descriptions of the memory writing action are omitted. However, those skilled in the art should know that the previous step, the next step, each step, and the current step mentioned in this application all refer to the memory writing action of the corresponding step number. That is, the previous step refers to the memory writing action of the previous step in terms of time sequence for the memory writing action of the current step, the next step refers to the next memory writing action in terms of time sequence for the memory writing action of the current step, and each step refers to each memory writing action.

[0032] In some embodiments, step S1, "calculating the local counterfactual score between the current memory and the historical memory," includes: Step S11: Obtain the current previous state of the current step in the current attempt path, the current memory, the historical memory, and the correct answer to the target question.

[0033] When calculating local counterfactual scores, it is necessary to first determine the probability of obtaining the correct answer based on current memory and historical memory, respectively. Therefore, the correct answer needs to be the calculation target. Similarly, to obtain the correct answer, in addition to memory, the preconditions for answering are also required. In this application, these preconditions are referred to as the prior state, which is used to represent the answering progress and is obtained from the results of multiple rounds of memory. The current prior state consists of the target question, the historical memory written in the previous step, and the reasoning, tool invocation, and tool observation of the previous round; the historical memory itself is a compressed representation of the previous interaction history.

[0034] Step S12: Calculate the current answer score based on the current prior state, the current memory, and the correct answer, wherein the current answer score is used to characterize the probability that the target agent obtains the correct answer based on the current memory.

[0035] Step S13: Calculate the historical answer score based on the current prior state, the historical memory, and the correct answer, wherein the historical answer score is used to characterize the probability that the target agent obtains the correct answer based on the historical memory.

[0036] In this application, the current memory and the historical memory are respectively placed in a calculation formula constructed from the memory, the previous state and the correct answer, and the current answer score of the current memory and the historical answer score of the historical memory are calculated respectively under the current previous state.

[0037] In this application, the preceding state of step t in the i-th attempted path is denoted as:

[0038] in, This represents the i-th attempt at the target problem. This represents the historical memory of the previous step. , , These represent the previous round of reasoning, tool invocation, and tool observation, respectively. If this is the first memory write operation of the i-th attempt path, i.e., the first step, then empty memory is used. And omit the previous round of interaction.

[0039] The write operation in the current step is represented as:

[0040] in, Represents the current memory strategy.

[0041] In this application, the correct answer will be recorded as: ; Therefore, this application uses the following formula to calculate the answer score. By using both current and historical memory for 'm' in the formula, the current answer score and the historical answer score can be obtained respectively. Wherein, This indicates that the score has been given.

[0042]

[0043] In the formula, Indicates the prediction of the first When there are 1 target words, the condition includes the 1 preceding it. One target word.

[0044] Step S14: Determine the local counterfactual score based on the current answer score and the historical answer score, wherein the local counterfactual score is used to characterize the degree of closeness between the current memory and the historical memory and the correct answer.

[0045] After obtaining the current answer score from the current memory and the historical answer score from the historical memory, the local counterfactual score can be determined based on the difference between the current answer score and the historical answer score.

[0046] Step S14 can be represented by the following formula:

[0047] Of course, if it's the first step, then

[0048] In this application, to eliminate scale differences at different steps, the Δ of the same cue word group g at the same step t is standardized, where the cue word group refers to N paths sampled from the same query in GRPO training, and the calculation formula is as follows:

[0049] In the formula, This represents the standardized local counterfactual score. This represents the mean within the group. The superscript (g, t) indicates that this statistic is the average of Δ calculated over all paths in the same prompt group g where a memory write also occurred at step t. This represents the within-group standard deviation, calculated similarly over the path set of "same group, same step index t". A value greater than 0 indicates that the current memory is closer to the correct answer than the historical memory.

[0050] Step S2: Calculate the current log-likelihood difference between the current memory and the sibling memory, wherein the sibling memory is the memory generated in other attempted paths that corresponds to the current step number.

[0051] Since the local counterfactual score in this application can characterize which memory, current memory or historical memory, is more likely to yield the correct answer, theoretically, the local counterfactual score obtained in steps S1-S14 can be used as the basis for reward or punishment in the current step. However, its authenticity cannot be guaranteed. That is, even if the local counterfactual score is greater than 0, it cannot be guaranteed that the current memory will necessarily yield the correct answer more easily than the historical memory. Therefore, other parameters are needed to corroborate this. In the process of corroboration, this application directly adds the correct answer to the current preceding state and proves the degree of closeness between the current memory and the correct answer through reverse deduction.

[0052] Besides differentiating the degree of progress between two adjacent memories on the same path, i.e., the local counterfactual score, the means of proving the closeness between the current memory and the correct answer can also be supported by the closeness between the current path and other sibling paths regarding the memory corresponding to the same step and the correct answer. This is the current log-likelihood difference value in this application, where the current log-likelihood value characterizes the probability that the current memory itself is generated (written out) when the current preceding state and the correct answer are both conditions. It is calculated by averaging the log-likelihood of each word in the current memory. Intuitively, if a memory writing is indeed necessary to obtain the correct answer, then the probability of that memory being generated should be amplified after adding the correct answer as a condition.

[0053] In some embodiments, step S2, "calculating the current log-likelihood difference between the current memory and the sibling memory", includes: Step S21: Determine the current log-likelihood value of the current memory based on the current prior state, the current memory, and the correct answer, wherein the current log-likelihood value is the average log-likelihood value of each word in the current memory.

[0054] Since each memory entry contains many lexical units, when calculating the current log-likelihood value of the current memory, we need to calculate the log-likelihood value of each lexical unit in the current memory, and then average the log-likelihood values ​​of multiple lexical units to obtain the current log-likelihood value of the current memory. The specific formula for calculating the log-likelihood value of the current memory is as follows:

[0055] In the formula, It is the j-th word in the current memory. This represents the current log-likelihood value of the current memory.

[0056] Step S22: Obtain the average likelihood of the number of sibling pairs in each parallel path, wherein the parallel path is another attempted path that is in the same group as the current attempted path.

[0057] When calculating the average log-likelihood of sibling paths, the log-likelihood value of each sibling path is calculated in the same way as the above formula. Then, the log-likelihood values ​​of each sibling path are averaged to obtain the average log-likelihood of sibling paths.

[0058] Step S23: Determine the current log-likelihood difference based on the current log-likelihood value and the average of the sibling log-likelihood values.

[0059] By subtracting the average log-likelihood of the siblings from the current log-likelihood value, we can obtain the difference between the current memory and the memory of the corresponding step in the same group. The specific expression is as follows:

[0060] In the formula, This represents the log-likelihood value of each sibling's memory. This represents the set of sibling memories within the same group that underwent memory writing at step t. Where, when A value >0 indicates that the current memory has a higher extrinsic relevance relative to its sibling memories in the same group, and vice versa.

[0061] To limit the influence of outcast brothers, this application addresses the issue of centralized... Perform the trimming; the trimming range is... ,in =0.1, =10 is an empirically determined hyperparameter that is symmetric in the logarithmic field (±log10≈±2.3).

[0062] Step S3: Determine the original memory score for the current step based on the local counterfactual score and the current log-likelihood difference.

[0063] This application aims to verify the correctness of local counterfactual scores using the log-likelihood difference. Specifically, when the log-likelihood difference and the local counterfactual score move in the same direction (e.g., both are greater than 0 or both are less than 0), the local counterfactual score is considered correct. If their directions are inconsistent, it indicates a problem in the result-obtaining process, but the exact stage of the problem cannot be determined. In this case, the penalty for the compressed memory stage is suppressed to prevent it from being penalized due to errors in other stages. Therefore, this application designs a gating system to control the reward and penalty criteria for the current step.

[0064] In some embodiments, step S3, "determining the original memory score of the current step based on the local counterfactual score and the current log-likelihood difference," includes: Step S31: Determine the adaptive sharpness of the gate based on all log-likelihood differences, where all log-likelihood differences are all log-likelihood differences generated by the target agent during the training process of the current batch.

[0065] Step S32: Determine the gating opening of the gate based on the adaptive sharpness, the current log-likelihood difference, and the local counterfactual score.

[0066] Step S33: Determine the original memory score based on the gating degree and the local counterfactual score.

[0067] To control the reward or punishment for the current step based on the direction of the log-likelihood difference and the local counterfactual score, it is necessary to first determine the gating opening. The gating opening is influenced by the gating sharpness. Therefore, this application first determines the gating sharpness, then determines the gating opening based on the sharpness, and finally determines the reward or punishment basis for the current step—that is, the original memory score—based on the gating opening and the local counterfactual score of the current memory. Considering that for a fixed gating sharpness, when... Scale drift can cause sharpness loss, so this application uses adaptive sharpness.

[0068] Specifically, in this application, the adaptive sharpness of the gating is calculated using the following formula:

[0069] The adaptive sharpness of the gating is represented by c = ln(0.8 / 0.2) ≈ 1.39, which is the sharpness calibration constant. Choosing it ensures that about 30%–50% of the samples can pass the σ(±c) threshold and the gating remains in the effective working area. Represents the sample standard deviation of all log-likelihood differences generated during the training process of the current batch, where... =0.5, =20 is the upper and lower limits of sharpness clipping, to prevent... When the gating is too wide or too narrow, sharpness becomes uncontrolled—too small a gating degenerates into a constant (signal is flattened), while too large a gating becomes a hard switch (saturation), thus preventing a fixed sharpness. The problem of failure during scale drift makes... Heng was established.

[0070] After calculating the adaptive sharpness, the gating opening can be calculated. The formula for calculating the gating opening in this application is as follows:

[0071] in, Indicates the opening degree of the gate. This is the preset gate threshold.

[0072] After obtaining the gating opening, the raw memory score for the current step can be calculated based on the gating opening and the local counterfactual score, as shown in the following formula:

[0073] In the formula, This represents the original memory score at the current step. The gating in this application allows when... , When all values ​​are positive, positive credit is amplified, i.e., local counterfactual scores are amplified. When all values ​​are negative, negative scores are retained. When there is a conflict in direction, credit can be suppressed to prevent incorrect causes from being misassigned.

[0074] Step S4: Optimize the memory strategy of the target agent based on the original memory scores of the memory writing actions at each step in the completion path, wherein the completion path is the attempt path to end the task of answering the target question, and the original memory score of each step is the original memory score of the memory writing action at each step.

[0075] The raw memory scores obtained in steps S1-S3 above can only serve as a basis for memory strategy optimization, but do not represent the final optimization result, i.e., the final reward / penalty result. Therefore, in some embodiments, step S4, "optimizing the memory strategy of the target agent based on the raw memory scores of the memory actions written in each step of the completion path," includes: Step S41: Update the original memory scores of each step in the successful path to obtain the completed memory score, wherein the original memory scores that are less than zero are updated to zero. The completed path includes the successful path and the failed path, and the successful path is the attempt path to obtain the correct answer to the target question.

[0076] Since the training process of a model should include both successful and failed cases, this application requires the optimization of the policy by relying on both successful and failed paths. In this application, the failed path refers to the path that answers the target question but is incorrect, or that fails to output an answer to the target question for other reasons.

[0077] In this application, a successful memory masking operation is performed on the completed path for which the correct answer has been obtained. Specifically, the original memory scores of negative values ​​in the successful path are updated to 0. This prevents negative memory signals at the microscopic level from overwriting the overall success evidence. For failed paths, the original memory scores at each step are not updated. To easily distinguish between the updated and unupdated original memory scores, the updated original memory score is referred to as the "complete memory score." This complete memory score also includes unupdated original memory scores; some original memory scores do not need to be updated, so the complete memory score at this point is numerically equivalent to the original memory score. This step can also be understood as updating the original memory scores of each step in the completed path to the complete memory score. For original memory scores less than zero in the successful path, these are updated to zero as the corresponding complete memory score. For other steps where no update is needed, the complete memory score remains numerically the same as the corresponding original memory score. This will continue to be used in subsequent expressions. To indicate, only the subsequent This represents the updated original memory score, i.e., the completed memory score.

[0078] Step S42: Use the completion memory score of the last step in each completion path as the smooth memory score.

[0079] Step S43: Determine the smooth memory score of the current step based on the smooth memory score of the next step and the completion memory score of the current step.

[0080] In theory, since the current memory in the last step is closest to the correct answer, the current memory will inevitably get a higher score. However, this score is also inseparable from the memory of the previous steps. Therefore, this application adopts a smoothing method to smooth the score of the later steps to the earlier steps.

[0081] Specifically, the final step's completion memory score is used as the smoothed memory score, i.e. .

[0082] In the formula, This represents the smooth memory score for the final step. This represents the original memory score of the final step, i.e., the completed memory score. This indicates the last step in completing the path.

[0083] Then, the smooth memory score for each step is calculated step by step from back to front, as shown in the following formula:

[0084] In the formula, α∈[0,1] represents the EMA smoothing coefficient. Through gradual smoothing, the scores of all memory write operations in the final path are updated to smoothed memory scores. Since some memory writes are preparatory (pre-storing clues whose value only becomes apparent after subsequent observations), these writes do not receive a score if only local utility is considered. Reverse EMA allows the utility of downstream steps to flow back in reverse time to earlier preparatory writes, resolving the attribution problem of delayed utility.

[0085] Step S44: Optimize the memory strategy based on the smooth memory score of each step in each of the completion paths.

[0086] The smooth memory score only refers to the score of each memory writing operation in terms of memory. In reality, after the task of answering the target question is completed, there will be a final reward or penalty given by the system. Therefore, in some embodiments, step S44, "optimizing the memory strategy based on the smooth memory score of each step in each completion path," includes: Step S441: Obtain the final reward or penalty for the completed path.

[0087] The final reward or punishment refers to the final reward or punishment given by the system when the task ends. The task can end because of a correct or incorrect answer, or because of other reasons, such as various restrictions that prevent the task from continuing to answer the target question.

[0088] Step S442: Determine the trajectory-level score of each memory lexicon based on the final reward / penalty and the smooth memory score of each step.

[0089] After obtaining the endgame reward and penalty for the completed path, it is necessary to convert the endgame reward and penalty into a trajectory-level score. The specific calculation formula is as follows:

[0090] In the formula, Indicates the final reward or penalty for the completed path. , These represent the within-group mean and standard deviation of the final reward / penalty for multiple paths during the response to the target question. The trajectory-level score measures how much better / worse this completed path is compared to its sibling paths in the same group. It provides coarse-grained but stable global supervision to all terms along the path; the smoothed memory score is a fine-grained incremental signal added only to the remembered terms, on top of this.

[0091] After obtaining the trajectory-level score, the trajectory-level score is then assigned to different lexical units. In this application, for lexical units in the memory field, the source of their trajectory-level score, in addition to the transformation from the final reward / penalty, also includes the smoothing memory score. Therefore, the final trajectory-level score assignment expression for each lexical unit in this application is as follows:

[0092] In the formula, Indicates which memory write operation the k-th word belongs to; This represents the lexical position range occupied by the memory m written in the current step within the path sequence. The above formula allows the trajectory-level score of the memory lexical to be... The trajectory-level scores for other word terms are... This ensures that the memory-specific score (smooth memory score) in this application is used only for memorizing word units and does not affect other word units.

[0093] Step S443: Optimize the memory strategy based on the trajectory-level scores of each memory lexicon.

[0094] In some embodiments, step S443, "optimizing the memory strategy based on the trajectory-level scores of each memory lexicon," includes: Step S4431: Determine candidate memory strategies based on the trajectory-level scores of each memory lexicon.

[0095] Step S4432: Determine the pruning strategy for the candidate memory strategy based on the candidate memory strategy and the old memory strategy.

[0096] Step S4433: After pruning the candidate memory strategy according to the pruning strategy, the optimized memory strategy is obtained.

[0097] After determining the trajectory-level scores of each memory unit, the memory strategy of the target agent can be optimized based on the trajectory-level scores of each unit. However, during optimization, the training may collapse due to the large update span. Therefore, this application also needs to restrict the update of the memory strategy. Thus, this application first determines the candidate memory strategy based on the trajectory-level scores of each memory unit. Since this memory strategy may have a large update span compared to the old memory strategy, this application needs to determine the pruning strategy for the candidate memory strategy based on the candidate memory strategy and the old memory strategy. Then, the candidate memory strategy is pruned according to the pruning strategy to obtain the optimized memory strategy and prevent the training from collapsing due to the large update span.

[0098] The update expression for the memory strategy in this application is as follows, which includes the entire process of steps S4431-S4433:

[0099] In the formula, the KL term is regularized relative to the reference policy to prevent the model from drifting too far. The reference policy is the initial policy preset for the model at the beginning of training, and is expressed as follows: , The importance sampling ratio, , Representing a path The k-th word element, Indicate candidate memory strategies, Representing old memory strategies, Indicates the area to be clipped.

[0100] Since the training data is collected using the old strategy, while the update uses the new strategy, their distributions differ, requiring an importance sampling ratio for correction—a ratio greater than 1 indicates that the new strategy makes that word more likely to appear. Then, combined with the pruning range, the deviation between the old and new strategies on each word in a single update is limited, preventing a single, drastic update from causing training failure. The KL term further constrains the strategy to the vicinity of the reference strategy, allowing the smooth memory score to be incorporated through the trajectory-level score of the word.

[0101] Compared with existing technologies, this application, while maintaining the same preconditions, provides a local counterfactual score for the current memory by comparing the probability of correct answers between "historical memory" and "current memory," thus solving the problem of static evaluation lacking an incremental perspective. It calculates the log-likelihood of the current memory under the premise of the current preconditions and the correct answer, and obtains the log-likelihood difference logρ by centering it with sibling trajectories in the same group. A directional consistency filter is applied to the local counterfactual score, allowing credit to pass only when the two directions are consistent, suppressing false credit induced by tools / reasoning and alleviating causal entanglement. Smoothed memories are distributed to each memory unit, while the remaining units retain trajectory-level scores to ensure global training stability. Simultaneously, a memory success mask is used to avoid applying erroneous negative signals to memory writes on the completion path, and a reverse exponential moving average (EMA) is used to backpropagate the delay utility to the preceding preparatory memory writes, improving the stability of fine-grained supervision. This invention addresses the problem in existing technologies where multiple rounds of interaction can lead to increased model inference costs, exceeding the fixed context window length limit, and decreased model performance. It achieves efficient memory compression and significantly improves the quality of credit allocation while minimizing additional computational overhead.

[0102] Example 2: Secondly, this application provides an electronic device, such as Figure 2 As shown, it includes: at least one processor 100; and a memory 200 communicatively connected to the at least one processor 100; wherein the memory 200 stores instructions executable by the at least one processor 100, the instructions being executed by the at least one processor 100 to enable the at least one processor 100 to execute a memory strategy optimization method for a long-term intelligent agent in the above embodiments.

[0103] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0104] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0105] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0106] The processor can communicate with external devices via the I / O bus through wired or wireless networks.

[0107] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0108] Example 3: Thirdly, this application proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0109] Computer-readable storage media can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, and computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0110] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0111] Example 4: Fourthly, this application proposes a computer program product, including a computer program / instructions, characterized in that, when the computer program is executed by a processor, it implements the steps of the method described in any one of the first aspects.

[0112] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0114] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0116] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0117] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a controller to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0118] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein.

[0119] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.< / mem>

Claims

1. A method for optimizing the memory strategy of a long-term intelligent agent, characterized in that, include: During the process of the target agent answering the target question, the following operations are performed for each step of the write-to-memory action in each attempt path: Calculate the local counterfactual score between the current memory and the historical memory, where the current memory is the memory generated in the current step of the current attempt path, and the historical memory is the memory written in the previous step of the current step; Calculate the current log-likelihood difference between the current memory and the sibling memory, wherein the sibling memory is a memory generated in other attempted paths and corresponding to the current step number; The original memory score for the current step is determined based on the local counterfactual score and the current log-likelihood difference. The memory strategy of the target agent is optimized based on the original memory scores of each step in the completion path, wherein the completion path is the attempt path to end the task of answering the target question, and the original memory score of each step is the original memory score of the memory writing action in each step.

2. The method according to claim 1, characterized in that, The calculation of the local counterfactual score between current memory and historical memory includes: Obtain the current preceding state of the current step in the current attempt path, the current memory, the historical memory, and the correct answer to the target question; The current answer score is calculated based on the current prior state, the current memory, and the correct answer, wherein the current answer score is used to characterize the probability that the target agent obtains the correct answer based on the current memory; The historical answer score is calculated based on the current prior state, the historical memory, and the correct answer, wherein the historical answer score is used to characterize the probability that the target agent obtains the correct answer based on the historical memory; The local counterfactual score is determined based on the current answer score and the historical answer score, wherein the local counterfactual score is used to characterize how close the current memory is to the correct answer compared to the historical memory.

3. The method according to claim 2, characterized in that, The calculation of the current log-likelihood difference between the current memory and the sibling memory includes: The current log-likelihood value of the current memory is determined based on the current prior state, the current memory, and the correct answer, wherein the current log-likelihood value is the average log-likelihood value of each word in the current memory; Obtain the average likelihood of the number of sibling pairs of the sibling memory in each parallel path, wherein the parallel path is another attempted path that is in the same group as the current attempted path; The current log-likelihood difference is determined based on the current log-likelihood value and the average of the sibling log-likelihood values.

4. The method according to claim 1, characterized in that, Determining the original memory score for the current step based on the local counterfactual score and the current log-likelihood difference includes: The adaptive sharpness of the gate is determined based on all log-likelihood differences, where all log-likelihood differences are all log-likelihood differences generated by the target agent during the training process of the current batch. The gating opening is determined based on the adaptive sharpness, the current log-likelihood difference, and the local counterfactual score. The original memory score is determined based on the gating degree and the local counterfactual score.

5. The method according to claim 1, characterized in that, The optimization of the target agent's memory strategy based on the original memory scores of the memory writing actions at each step in the completion path includes: The original memory scores at each step of the successful path are updated to obtain the completed memory score, wherein the original memory scores that are less than zero are updated to zero. The completed path includes a successful path and a failed path, and the successful path is the attempt path to obtain the correct answer to the target question. The completion memory score of the last step in each of the completion paths is used as the smooth memory score. The smooth memory score of the current step is determined based on the smooth memory score of the next step and the completion memory score of the current step. The memory strategy is optimized based on the smooth memory score of each step in each of the completion paths.

6. The method according to claim 5, characterized in that, The optimization of the memory strategy based on the smoothed memory score of each step in each of the completion paths includes: Obtain the final rewards and penalties for the completed path; The trajectory-level score of each memory lexicon is determined based on the final reward / penalty and the smooth memory score of each step, wherein the memory lexicon is a lexicon in the memory field; The memory strategy is optimized based on the trajectory-level scores of each memory lexicon.

7. The method according to claim 6, characterized in that, The optimization of the memory strategy based on the trajectory-level scores of each memory lexical unit includes: Candidate memory strategies are determined based on the trajectory-level scores of each memory lexical unit; A pruning strategy for the candidate memory strategy is determined based on the candidate memory strategy and the old memory strategy; After pruning the candidate memory strategies according to the pruning strategy, the optimized memory strategy is obtained.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7.

Images