Methods, devices, electronic devices and storage media for intelligent agent self-training optimization

By constructing a reflection mechanism in a virtual simulation space, the intelligent agent can autonomously trigger reflection and generate multi-dimensional correction strategies, solving the problems of insufficient utilization of failure experience and single optimization dimension in traditional training. This achieves efficient and stable training results and is suitable for complex, long-term, and security-sensitive scenarios.

CN123117166APending Publication Date: 2026-11-17CHINA UNITED NETWORK COMM GRP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610995513.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-06
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In complex, long-term, high-security-constraint, and multi-step task scenarios, traditional AI agent training suffers from insufficient utilization of failure experience, lack of autonomous review and correction capabilities, passive and unretrievalable experience playback mechanisms, and a single optimization dimension. This results in high training costs, poor stability, and weak generalization ability, making it difficult to meet the training requirements of high reliability and high efficiency.

Method used

A virtual simulation space matching the target training task is constructed, basic reflection parameters are configured, and the agent is driven to perform online training and trigger reflection in the virtual space. Training data and historical memory data are extracted, and a structured reflection report is generated. The reflection memory bank supports retrieval by error type and optimization direction, generates multi-dimensional correction strategies, and performs retraining to form an autonomous closed-loop optimization.

Benefits of technology

It enables the agent to autonomously trigger a reflection mechanism during training, perform structured attribution analysis, generate multi-dimensional correction strategies, reduce training costs, and improve training stability and generalization ability. It is suitable for high-reliability and high-efficiency training in security-sensitive scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117166A_ABST
    Figure CN123117166A_ABST
Patent Text Reader

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for intelligent agent self-training optimization, relating to the field of virtual simulation technology. The method includes: constructing a virtual simulation space, initializing the basic model of the intelligent agent, and configuring reflection parameters; driving the intelligent agent to train online, triggering reflection when preset reflection conditions are met; extracting training data and historical memory data within the reflection window, inputting them into a large language model to generate a reflection report; parsing the reflection report, generating reflection memory units and storing them in a reflection memory bank; performing semantic parsing and multi-source data fusion attribution verification on the reflection memory units, identifying core root causes, and generating correction strategies; injecting each correction strategy into the training architecture of the intelligent agent and performing retraining; and updating the reflection memory bank based on the effect verification data of the retraining. This invention can effectively reduce the training cost of intelligent agents in virtual space, improve training stability and generalization ability, and meet the high reliability and high efficiency training requirements in security-sensitive scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of virtual simulation technology, and in particular to a method, apparatus, electronic device, and storage medium for intelligent agent self-training optimization. Background Technology

[0002] Currently, the training of artificial intelligence (AI) agents in virtual simulation environments generally adopts deep reinforcement learning and imitation learning as core technical approaches. Through repeated interaction and trial and error in digital twins and virtual training scenarios, the agent gradually learns decision-making strategies, and this approach has been widely applied in fields such as surgical simulation, high-risk operation training, game-theoretic competition, and autonomous driving simulation. Traditional training methods mainly rely on environmental reward signals to drive policy updates, using an experience replay pool to randomly sample and reuse historical trajectories, and relying on extensive trial and error to achieve model convergence. However, in complex, long-term, high-safety-constraint, and multi-step task scenarios, these methods generally suffer from high training costs, poor stability, and weak generalization ability, especially in safety-sensitive scenarios where they cannot meet the requirements for highly reliable and efficient training.

[0003] To address the aforementioned issues, the industry has proposed several improvement solutions: 1) Agent reflection solution: By determining the failure diagnosis results corresponding to the collaborative failure trajectories of multiple agents, the correction results are determined based on counterfactual reasoning, and reflections are generated for the responsible agent; 2) Learning solution based on stateful reflective memory: Combining contextual memory with reinforcement learning, a state-aware reflective decision-making process is proposed, and reflective memory is transformed into a cybernetic object through two core operations: writing and reading; 3) Large language model agent reinforcement learning solution based on reflection and memory mechanisms: Constructing a three-layer model of execution-evaluation-reflection, and transforming failure experiences into readable, storable, and searchable text reflections to achieve continuous learning without the need for model parameter fine-tuning.

[0004] However, AI agent training in traditional virtual simulation environments generally suffers from the following common problems in complex, long-term, high-security-constraint, and multi-step task scenarios: 1) Insufficient utilization of failure experience: Failure experience and deviation trajectories are simply stored and randomly sampled, lacking structured attribution and root cause localization. The agent cannot learn from failures, is prone to repeating the same mistakes, and has low training convergence efficiency; 2) Lack of autonomous review and correction capabilities: Policy updates rely on fixed reward functions and manual parameter tuning. During training, it cannot autonomously trigger reviews based on real-time performance, autonomously generate correction strategies, and inject them into the training process. The degree of automation is low, and the cost of manual intervention is high; 3) Passive and unretrievalable experience replay mechanism: The experience replay mechanism is passive and random reuse, and does not support retrieval and targeted calling based on error type and optimization direction. Historical optimization experience cannot be effectively inherited, making it difficult to form an accumulative and reusable optimization knowledge system; 4) Single optimization dimension: Training optimization methods usually only adjust model parameters or sampling strategies, lacking a systematic solution for multi-level collaborative optimization of training data, reward mechanism, model structure, and training process, resulting in limited optimization effects. The aforementioned problems collectively result in high training costs, poor stability, and weak generalization ability for intelligent agents in virtual space, making it particularly difficult to meet the requirements for high reliability and high efficiency in security-sensitive scenarios. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, apparatus, electronic device and storage medium for intelligent agent self-training optimization.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, the present invention provides a self-training optimization method for intelligent agents, employing the following technical solution: A self-training optimization method for intelligent agents, comprising: Construct a virtual simulation space that matches the target training task, initialize the basic model of the agent, and configure the basic parameters for reflection; The intelligent agent is driven to perform online training in the virtual simulation space, and when any preset reflection condition is detected, the corresponding reflection is triggered; In response to the triggering of reflection, the training data and relevant historical memory data within the reflection window are extracted, and the training data and relevant historical memory data are input into the pre-trained large language model to generate a structured reflection report containing root cause analysis. The structured reflection report is parsed to generate reflection memory units and stored in the reflection memory bank; the reflection memory bank supports retrieval by error type and optimization direction; Semantic parsing and multi-source data fusion attribution verification are performed on the reflexive memory unit to identify the core root cause and generate correction strategies corresponding to each of the multiple training optimization dimensions. The aforementioned correction strategies are injected into the training architecture of the agent according to their priority and then retraining is performed. The reflective memory bank is updated based on the effectiveness verification data from retraining.

[0007] The beneficial effects of this invention are as follows: By constructing a virtual simulation space and configuring basic reflection parameters, the agent can autonomously trigger a reflection mechanism during training, perform structured attribution analysis on failure experiences, and generate a reflection report containing root cause analysis. By parsing the reflection results into searchable reflection memory units and locking in the core root causes based on multi-source data fusion attribution verification, a multi-dimensional correction strategy is generated, realizing autonomous learning and continuous optimization from failures. By injecting the correction strategy into the training architecture according to priority and executing retraining, a fully autonomous closed loop from training, reflection, optimization to retraining is formed, completing self-training optimization without human intervention. Compared with traditional training methods, this method can effectively reduce the training cost of the agent in the virtual space, improve training stability and generalization ability, and is especially suitable for high-reliability and high-efficiency training needs in security-sensitive scenarios.

[0008] Based on the above technical solution, the present invention can be further improved as follows.

[0009] Furthermore, the configuration reflection basic parameters include: Configure the relevant parameters for the periodic triggering conditions of global reflection and the event triggering conditions of emergency reflection in the dual-mode reflection triggering rules; wherein, the periodic triggering condition is that the training rounds reach a preset number of rounds or the training duration reaches a preset duration; the event triggering condition is that a preset failure event occurs during the training process; Configure the filling specifications and data format of each field in the reflection report generation template; wherein, each field in the reflection report generation template includes reflection identifier, reflection type, reflection time, reflection window period, training data statistical indicators, deviation or failure analysis indicators, optimization suggestion indicators, and effect verification indicators; The storage structure of the reflexive memory is configured as a hybrid storage structure including a vector database and a relational database; wherein, the vector database is used to store the feature vectors of the reflexive memory units; and the relational database is used to store the structured information of the reflexive memory units. The pre-trained large language model is configured to be fine-tuned using a reflective case dataset from the target task domain; Configure the constraint boundaries of each of the aforementioned correction strategies; wherein, the constraint boundaries include the fine-tuning range of model parameters, the weight adjustment range of the reward function, and the number of scene enhancement samples generated; Configure the convergence criteria for the agent; wherein the convergence criteria include average task completion rate, recurrence rate of similar errors, and training loss fluctuation.

[0010] The beneficial effects of adopting the above-mentioned further scheme are as follows: By configuring dual-mode reflection trigger rules, the agent can perform global reviews periodically and trigger emergency reflections immediately when failure events occur, balancing training efficiency and the timeliness of risk response; by configuring standardized reflection report templates, the structured and parsable nature of reflection results is ensured, providing a standardized data foundation for subsequent attribution analysis and strategy generation; by configuring a reflection memory that uses a hybrid storage of vector and relational databases, efficient storage and multi-dimensional retrieval of reflection experience are achieved, enabling the rapid location and reuse of historical optimization experience; by fine-tuning the pre-trained large language model using domain reflection cases, the accuracy and relevance of the generated reflection reports are improved; by configuring the constraint boundaries of each correction strategy, training instability caused by excessive strategy adjustment is effectively prevented; and by configuring a convergence criterion for joint judgment of multiple indicators, the scientific nature and reliability of the training termination conditions are ensured. These configurations together constitute the complete foundation for the operation of the reflection mechanism, providing reliable support for the agent's autonomous closed-loop optimization.

[0011] Furthermore, in response to triggering reflection, the extraction of training data and relevant historical memory data within the reflection window includes: In response to triggering the global reflection, the full training data of the current training cycle is extracted as the training data within the reflection window. In response to triggering the emergency reflection, training data of a preset number of steps before and after the preset failure event is extracted as training data within the reflection window period; Historical memory data related to the type and error type of the target training task are retrieved from the basic memory bank, the task process knowledge base, and the reflection memory bank.

[0012] The beneficial effects of adopting the above-mentioned further solutions are as follows: By distinguishing between two trigger scenarios—global reflection and emergency reflection—and extracting the full training data and trajectory data of the preset number of steps before and after the event, the reflection analysis can cover the overall performance of the entire training cycle while focusing on the local details of sudden failure events, thus balancing comprehensiveness and specificity. By retrieving historical memory data related to the current task type and error type from the basic memory bank, task process knowledge base, and reflection memory bank, rich reference data is provided for the generation of reflection reports, enabling root cause analysis to draw on historical experience, avoid viewing each failure in isolation, and improve the accuracy and efficiency of attribution.

[0013] Furthermore, the method also includes: The reflective memory unit is vectorized and encoded to generate the feature vector of the reflective memory unit; The feature vector of the reflective memory unit is matched with the feature vector of the historical memory unit in the reflective memory bank based on similarity. If the similarity is not lower than a preset threshold, the historical memory unit is updated based on the feature vector of the reflective memory unit; If the similarity is lower than the preset threshold, the reflective memory unit is stored as a new memory unit in the reflective memory bank.

[0014] The beneficial effects of adopting the above-mentioned further scheme are as follows: By vectorizing and encoding reflective memory units and matching them with historical memories based on similarity, efficient deduplication and orderly management of reflective experiences are achieved. When a newly generated reflective memory unit is highly similar to an existing historical memory unit, only the original historical memory unit is updated and supplemented, avoiding the repeated storage of similar failure experiences and maintaining the conciseness and efficiency of the memory bank. When the new reflective memory unit belongs to a problem that has not been recorded before, it is entered into the database as an independent memory unit, continuously expanding the knowledge system. This mechanism ensures that the reflective memory bank always maintains a state of high information density and low redundancy, which not only improves retrieval efficiency but also provides high-quality knowledge support for subsequent attribution analysis and strategy generation.

[0015] Furthermore, the semantic parsing and multi-source data fusion attribution verification of the reflective memory unit are performed to identify the core root cause and generate correction strategies corresponding to multiple training optimization dimensions, including: Semantic parsing is performed on the reflection memory unit to extract multiple reflection points and assign weights to each reflection point; Collect the multi-source data; wherein, the multi-source data includes the agent's basic model data, virtual environment data, and task constraint data; For each of the aforementioned reflection points, the correlation coefficient between the reflection point and the agent's basic model data, virtual environment data, and task constraint data is calculated, and the correlation coefficients are weighted to obtain a weighted fusion correlation coefficient. If the weighted fusion correlation coefficient is greater than or equal to the preset correlation coefficient, then the reflection point is determined to be the core root cause; If the weighted fusion correlation coefficient is less than the preset correlation coefficient, then the reflection point is determined to be a misattribution caused by environmental random noise. Based on each of the core root causes, corresponding correction strategies are generated for the training data dimension, reward function dimension, model dimension, and training process dimension, and the priority of each correction strategy is determined based on the weight of each of the core root causes.

[0016] The beneficial effects of adopting the above-mentioned further approach are as follows: By semantically parsing the reflective memory unit and extracting weighted reflective points, and combining multi-source information such as agent basic model data, virtual environment data, and task constraint data for correlation calculation and weighted fusion, accurate attribution verification of failure causes is achieved. This method can effectively distinguish between the true core root cause and misattribution caused by random environmental noise, avoiding incorrect optimization decisions due to accidental interference. Based on this, correction strategies are generated according to the type of core root cause, encompassing four dimensions: training data, reward function, model structure, and training process. The execution priority of each correction strategy is determined based on the weight of the core root cause, ensuring that the most important optimization directions are prioritized. This achieves full automation and precision from problem localization to strategy generation.

[0017] Furthermore, the correction strategies corresponding to the training data dimension, reward function dimension, model dimension, and training process dimension generated based on each of the core root causes include: If the core root cause is related to the training data, a first correction strategy corresponding to the dimension of the training data is generated; wherein, the first correction strategy includes generating enhanced scene samples for high-frequency deviation scenarios and expanding them into the experience replay pool. If the core root cause is related to the reward function, a second correction strategy corresponding to the dimension of the reward function is generated; wherein, the second correction strategy includes adjusting the weight of the reward function, supplementing dense rewards, or amplifying the constraint penalty intensity; If the core root cause is related to the model, a third correction strategy corresponding to the model dimension is generated; wherein, the third correction strategy includes adjusting the network structure or fine-tuning the model parameters; If the core root cause is related to the training process, a fourth correction strategy corresponding to the training process dimension is generated; wherein, the fourth correction strategy includes optimizing the task decomposition scheme or adjusting the training rhythm.

[0018] The beneficial effects of adopting the above-mentioned further approach are as follows: It accurately matches the corresponding correction strategy based on the type of the core root cause, achieving a targeted transformation from problem identification to solution. When the core root cause is related to the training data, the experience replay pool is expanded by generating enhanced scenario samples, compensating for insufficient data in specific scenarios. When the core root cause is related to the reward function, the guiding role of the reward signal is optimized by adjusting weights, supplementing dense rewards, or amplifying the penalty intensity. When the core root cause is related to the model, the model's decision-making ability is directly improved by adjusting the network structure or fine-tuning parameters. When the core root cause is related to the training process, the scientific nature and efficiency of the training process are improved by optimizing the task decomposition scheme or adjusting the training rhythm. This targeted approach based on root cause type avoids blind adjustments, ensuring that each optimization is purposeful and significantly improving the pertinence and effectiveness of strategy correction.

[0019] Furthermore, updating the reflective memory bank based on the retraining performance validation data includes: If the retraining effect verification data shows that the correction strategy is effective or partially effective, then the correction strategy is added to the corresponding reflective memory unit and marked as a reusable strategy. If the retraining effect verification data indicates that the correction strategy is ineffective, the reasons for the ineffectiveness are analyzed and a second reflection is triggered to generate a new correction strategy.

[0020] The beneficial effects of adopting the above-mentioned further scheme are as follows: For effective or partially effective correction strategies, they are added to the corresponding reflective memory units and marked as reusable strategies, allowing successful optimization experiences to be deposited and accumulated. These experiences can then be directly invoked when encountering similar problems in the future, avoiding repeated analysis and trial and error. For ineffective correction strategies, the reasons for failure are analyzed and secondary reflection is triggered to regenerate more accurate optimization schemes, avoiding the continued impact of ineffective strategies. This mechanism enables the reflective memory bank to continuously improve itself during use, continuously enhancing the quality of experience and providing a reliable knowledge guarantee for the long-term stable training of the agent.

[0021] Secondly, the present invention provides an intelligent agent self-training optimization device, which adopts the following technical solution: A self-training optimization device for intelligent agents, comprising: The training initialization module is used to construct a virtual simulation space that matches the target training task, initialize the basic model of the agent, and configure the basic parameters for reflection. An agent training module is used to drive the agent to perform online training in the virtual simulation space; The reflection condition trigger module is used to trigger the corresponding reflection when any preset reflection condition is detected. The reflection report generation module is used to respond to the triggering of reflection, extract training data and relevant historical memory data within the reflection window, and input the training data and relevant historical memory data into the pre-trained large language model to generate a structured reflection report containing root cause analysis. The reflection memory management module is used to parse the structured reflection report, generate reflection memory units, and store them in the reflection memory library; the reflection memory library supports retrieval by error type and optimization direction; The correction strategy generation module is used to perform semantic parsing and multi-source data fusion attribution verification on the reflective memory unit, lock in the core root cause, and generate correction strategies corresponding to multiple training optimization dimensions. A closed-loop training optimization module is used to inject the aforementioned correction strategies into the training architecture of the agent according to priority and perform retraining. The reflective memory update module is used to update the reflective memory bank based on the effect verification data of retraining.

[0022] Thirdly, the present invention provides an electronic device that adopts the following technical solution: An electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the computer program, implements the agent self-training optimization method as described in any of the first aspects.

[0023] Fourthly, the present invention provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the agent self-training optimization method as described in any of the first aspects.

[0024] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0025] Figure 1 A flowchart illustrating the intelligent agent self-training optimization method provided by this invention; Figure 2 A schematic diagram of the intelligent agent self-training optimization device provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0027] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the intelligent agent self-training optimization method provided by the present invention. Figure 1 As shown, the method may include the following steps S101-S107.

[0028] S101. Construct a virtual simulation space that matches the target training task, initialize the basic model of the agent, and configure the basic parameters for reflection.

[0029] Specifically, first, a virtual simulation space matching the target training task is constructed through the following process.

[0030] 1) Scene Physical Rule Configuration: Based on the real-world characteristics of the target task, configure the physical rule parameters of the virtual environment, including collision detection thresholds, mechanical feedback coefficients, and environmental interference parameters (such as wind resistance in high-altitude operations and tissue elasticity in surgical simulations), ensuring that the consistency error between the physical characteristics of the virtual environment and the real scene does not exceed 5%. Through high-precision physical rule configuration, the training experience of the agent in the virtual space can be effectively transferred to the real scene, avoiding policy failure due to environmental differences.

[0031] 2) Training Task Decomposition: The target training task is broken down into 3 to 8 core sub-task nodes, clearly defining the input / output criteria, execution order, and connection conditions for each sub-task. For example, a surgical simulation task can be decomposed into sub-tasks such as puncture, separation, and resection, with clear completion markers set for each sub-task. Through reasonable task decomposition, the complexity of individual decision-making steps is reduced, enabling the agent to master various skills in stages, improving training efficiency and the interpretability of the strategy.

[0032] 3) Definition of Hard Constraints: Define hard constraints during agent training, including safety regulations and operational boundaries, and set corresponding violation thresholds. Examples include safety distance thresholds for high-risk operations and tissue damage assessment criteria for surgical simulations. Violating hard constraints will trigger emergency reflection and task termination. By pre-defining hard constraints, it ensures that the agent always operates within safe boundaries during training, avoiding irreversible errors, which is particularly suitable for safety-sensitive scenarios.

[0033] 4) Definition of State Observation Space: Determine the data type, dimensions, and acquisition frequency of the state observation space. For example, the acquisition frequency for image data is 10 to 30 frames / second, and the acquisition frequency for numerical data (such as instrument pose and environmental parameters) is 100Hz. The state observation data is then standardized and preprocessed to the [0,1] interval. This standardized state observation space definition ensures that the agent can comprehensively perceive environmental information, providing a complete data foundation for subsequent decision-making and reflection.

[0034] 5) Action Execution Space Definition: This defines the action types, ranges, and execution precision within the action execution space. Examples include the six-degree-of-freedom range and step size precision of surgical instruments, the decision-making action options for game-theoretic agents, and the explicit environmental feedback logic for each action. Through refined action space definition, each decision-making action of the agent receives accurate environmental feedback, providing a reliable causal link for policy optimization.

[0035] 6) Basic Reward Function Configuration: A composite reward mechanism combining final reward, node reward, and constraint penalty is adopted. The final reward is set based on task completion rate: 100% completion corresponds to a positive reward of 100-200, while incomplete completion corresponds to a negative reward of 50-100. Node rewards are set based on the completion status of each subtask, with a positive reward of 10-50 for completing a subtask. Constraint penalties are set for violations of hard constraints, with a negative reward of 50-100 for each violation. Through this composite reward mechanism, the agent can focus on achieving the final task goal while also perceiving the execution quality of intermediate steps, effectively alleviating the reward sparsity problem in long-sequential tasks.

[0036] 7) Task success and failure criteria: Clearly define the task success threshold, such as a task completion rate of no less than 95% and no hard constraint violations; clearly define the task failure criteria, such as three or more hard constraint violations, a task completion rate of less than 30%, or a single major error. These clear criteria provide a clear evaluation basis for subsequent reflection triggers and training convergence.

[0037] After the virtual simulation space is constructed, the basic model of the intelligent agent is initialized through the following process.

[0038] 1) Selection and Construction of the Basic Decision-Making Model: Based on the characteristics of the target training task, a suitable deep reinforcement learning algorithm is selected. For complex continuous action scenarios (such as surgery or high-risk operations), the SAC or TD3 algorithm is used; for discrete action scenarios (such as game theory or simple logical reasoning), the PPO algorithm is used. A deep neural network is constructed, comprising an input layer, hidden layers, and an output layer. The hidden layers are set to 3 to 5 layers, with 64 to 256 neurons per layer. The input layer receives standardized observation data, and the output layer outputs action commands or action probability distributions. Through targeted algorithm selection and network structure design, the basic decision-making model can adapt to the characteristics of different task scenarios, providing a reliable initial strategy for subsequent reflective optimization.

[0039] 2) Functional Module Construction: A state perception module, an action decision-making module, and a reward calculation module are constructed. The state perception module uses CNN to process image-based observation data and MLP to process numerical observation data, extracting core features from the observation data with an accuracy of no less than 90% and a processing latency of no more than 50ms. The action decision-making module works in conjunction with the basic decision model, receiving the feature output from the state perception module and combining it with historical decision data to output action commands that conform to the action space constraints, with a decision latency of no more than 100ms. The reward calculation module receives state data from environmental feedback in real time, calculates the current reward value according to the basic reward function, and simultaneously inputs the reward data into the experience replay pool, with a calculation error of no more than 1%. Through the coordinated operation of these functional modules, the agent can efficiently complete the entire decision-making chain from environmental perception to action output, laying the foundation for online training.

[0040] 3) Basic Memory Initialization: Initialize the experience replay pool and task flow knowledge base. The initial capacity of the experience replay pool is set to 10,000 to 100,000 records, used to store observation states, action instructions, reward signals, and state transition data during agent training. It is updated using a first-in, first-out (FIFO) mechanism to ensure data timeliness. The task flow knowledge base stores the standard procedures, operating specifications, and common error cases for the target task, using a structured format and supporting fast retrieval by task node and error type, with a retrieval latency of no more than 10ms. The initialization of the basic memory provides efficient storage and retrieval support for the agent's experience reuse and historical knowledge retrieval.

[0041] After the basic model of the intelligent agent is initialized, configure the following basic reflection parameters.

[0042] 1) Dual-mode reflection trigger rules: Configure the triggering parameters for global reflection and emergency reflection. The periodic triggering condition is when the training rounds reach a preset number of rounds or the training duration reaches a preset duration. The configurable range for the preset number of rounds is 10 to 100 training rounds, with 50 to 100 for complex tasks and 10 to 30 for simple tasks. The recommended preset duration is 30 to 120 minutes, adjusted according to task complexity. The event triggering conditions are preset failure events that occur during training, specifically including the following three categories: First, major failure triggering, where the agent exhibits behavior that triggers the task termination condition and the absolute value of the negative reward exceeds a preset threshold, which is set to 60% to 80% of the final negative reward, such as tissue damage in surgical simulation or electric shock in high-risk operations; Second, continuous deviation triggering, where the agent's decision deviation for a preset number of consecutive steps exceeds a preset threshold, with the number of consecutive steps configurable from 5 to 20 steps, and the deviation threshold set according to the action accuracy requirements, such as an instrument pose deviation not exceeding 0.5mm; Third, task completion triggering, where the task completion rate in a single training round is lower than a preset lower limit, which is set to 30% to 50% and adjusted according to the task difficulty. Through dual-mode reflection triggering rules, the agent can perform global reviews periodically and respond instantly when failure events occur, balancing training efficiency with timely risk control.

[0043] 2) Standardized Reflection Report Generation Template: Configure the filling specifications and data format for each field in the reflection report generation template. The template fields include reflection identifier, reflection type, reflection time, reflection window period, training data statistical indicators, bias or failure analysis indicators, optimization suggestion indicators, and effect verification indicators. The standardized reflection report template ensures that each reflection output has a consistent format and complete information, facilitating subsequent structured analysis and attribution analysis.

[0044] 3) Reflection Memory Storage Structure: The reflection memory is configured with a hybrid storage structure combining a vector database and a relational database. The vector database stores the feature vectors of the reflection memory units, while the relational database stores their structured information. Storage capacity can be dynamically expanded based on training needs. This hybrid architecture supports both efficient similarity retrieval based on feature vectors and precise queries based on structured information, providing a technical guarantee for the efficient reuse of historical experience.

[0045] 4) Reflective Analysis of NLP Model Parameters: Configure a pre-trained large language model and fine-tune it using a reflective case dataset from the target task domain. The pre-trained large language model can be a base model such as BERT or RoBERTa. Target task domains include medical, industrial, etc. The fine-tuning dataset contains 1000 to 5000 reflective cases from the target task domain. The model's semantic parsing accuracy should be no less than 85%, and the processing latency should not exceed 200ms. Through fine-tuning with the reflective case dataset from the target task domain, the large language model can more accurately understand the professional terminology and decision-making logic in the target task scenario, improving the accuracy and relevance of the generated reflective report.

[0046] 5) Policy Adjustment Constraints: Configure the constraint boundaries for each adjustment policy, including the fine-tuning range of model parameters, the adjustment range of reward function weights, and the number of scene enhancement samples generated. Specifically, the fine-tuning range of model parameters, such as setting the learning rate to 0.001 to 0.01, the adjustment range of reward function weights to ±50%, and the number of scene enhancement samples generated to 50 to 200 sets, avoids over-adjustment leading to training instability. By setting reasonable constraint boundaries, it ensures that each policy adjustment is performed within a controllable range, preventing drastic fluctuations in model performance due to excessive adjustment.

[0047] 6) Training Convergence Criteria: Configure convergence criteria for the agent, including average task completion rate, recurrence rate of similar errors, and training loss fluctuation. For example, if the average task completion rate is not less than 95% over 100 consecutive training rounds, the recurrence rate of similar errors does not exceed 5%, and the training loss fluctuation does not exceed 10%, training is considered converged when all three criteria are met simultaneously. This multi-criteria convergence standard avoids misjudgments that may arise from a single criterion, ensuring the scientific validity and reliability of the training termination conditions.

[0048] S102. Drive the intelligent agent to perform online training in the virtual simulation space, and trigger the corresponding reflection when any preset reflection condition is met.

[0049] Specifically, the agent is driven to perform online training in an initialized virtual simulation space through the following process.

[0050] 1) State Acquisition: The agent uses the state perception module to collect multi-dimensional observations of the virtual environment in real time at a preset acquisition frequency, including image and numerical data. After acquisition, the data is immediately standardized and preprocessed, and a mean filtering algorithm is used to remove noise data, with a noise removal rate of no less than 90%. Through efficient state acquisition and preprocessing, the accuracy and reliability of the observation data input to the decision-making model are ensured, providing a high-quality data foundation for subsequent decision output and reflective analysis.

[0051] 2) Decision Output: The preprocessed observation data is input into the basic decision model. The model combines the current state and historical decision experience to output action instructions that conform to the action execution space constraints. After the action instructions are output, their validity is verified to avoid exceeding the action range. Through a strict decision output and verification mechanism, it is ensured that every action of the agent is executed within the safe boundary, reducing the risk of training failure caused by out-of-bounds actions.

[0052] 3) Environmental Interaction: The agent executes action commands in a virtual environment. Based on physical rules and task constraints, the environment provides corresponding reward signals and state transition data. The reward signals are calculated in real-time by the reward calculation module, and the state transition data includes changes in environmental state and updates to task progress. Through real-time environmental interaction and feedback, the agent can perceive the direct impact of each action, providing immediate causal information for strategy optimization.

[0053] 4) Data storage: The observation state, action commands, reward signals, and state transition data of the current training step are stored in the experience replay pool according to a unified format to ensure data integrity and traceability. This standardized data storage accumulates comprehensive training trajectory data for subsequent model iterations and reflective analysis.

[0054] 5) Basic Iteration: Following the selected deep reinforcement learning algorithm, the parameters of the basic decision model are updated using data from the experience replay pool. The learning rate for each iteration is controlled between 0.001 and 0.01, and the number of iterations is dynamically adjusted according to the training progress to ensure that the model gradually converges. Through continuous basic iteration, the agent's decision-making ability is continuously optimized during training, providing a continuously improving baseline strategy for subsequent reflective optimization.

[0055] During online training, the entire training process is monitored in real time, and it is simultaneously determined whether preset reflection conditions are met. If periodic trigger conditions are met, global reflection is triggered. If event-triggered reflection conditions are met, emergency reflection is triggered. If neither periodic nor event-triggered reflection conditions are met, online training continues, the training status is continuously monitored, and data collection, decision output, environment interaction, and basic model iteration are performed synchronously. After each training round, key indicators such as task completion rate and number of errors are recorded for subsequent reflection and analysis.

[0056] S103. In response to the triggered reflection, extract the training data and relevant historical memory data within the reflection window, and input the training data and relevant historical memory data into the pre-trained large language model to generate a structured reflection report containing root cause analysis.

[0057] Specifically, in response to triggered reflection, training data and relevant historical memory data within the reflection window are extracted through the following process.

[0058] 1) Defining the Reflection Window: The scope of the reflection window is defined based on the type of reflection. The window for global reflection is the entire duration of the current training cycle (N training rounds); the window for emergency reflection is the period from a preset number of steps before the triggering event to a preset number of steps after the event (recommended 5-10 steps before and after the event), ensuring coverage of the complete decision-making trajectory before and after the event. By distinguishing the window scopes for global and emergency reflection, subsequent data analysis can both comprehensively review cycle performance and accurately focus on sudden failures, avoiding irrelevant data interfering with root cause judgment.

[0059] 2) Current Training Data Extraction: In response to a global reflection, extract all training data from the current training cycle as training data within the reflection window; in response to an emergency reflection, extract training data for a preset number of steps before and after a preset failure event as training data within the reflection window. Specifically, the training data extracted within the window includes: ①Environmental observation sequence: All observation data arranged in chronological order, including multi-dimensional data such as image data and numerical data, with the timestamp corresponding to each observation data. ② Decision-making action sequence: All action instructions output by the agent within the window period, labeled with the execution time, action type, and action parameters of each action; ③ Reward signal sequence: All reward data from environmental feedback, labeled with the action and state corresponding to each reward, distinguishing between positive rewards, negative rewards and reward reasons; ④ State transition trajectory: The complete trajectory of the change in environmental state as the agent acts, with the triggering conditions and magnitude of change for each state transition marked; ⑤ Task completion data: Task completion rate and subtask completion status for each training round within the global reflection annotation period; Task completion rate for the training round corresponding to the emergency reflection annotation trigger event; ⑥ Accurately locate failure nodes: Mark the occurrence time, triggering action, failure type, and corresponding reward signal of all failure events within the window period.

[0060] 3) Retrieval of Historical Memory Data: Historical memory data related to the target training task and error type is retrieved from the experience replay pool, task process knowledge base, and reflection memory base of the basic memory bank. This includes: ① decision-making trajectories and reflection memories of similar historical failures; ② decision-making experiences and optimization strategies of successful historical cases; ③ corresponding operational specifications and constraints in the task process knowledge base. After retrieval, the historical memory data is initially compared with the current data, and relevant historical memory data with a similarity of no less than 60% is selected for root cause analysis. By incorporating similar historical successes and failures into this reflection, we avoid viewing a single failure in isolation, and use historical knowledge to help verify the accuracy of the root causes and improve the targeting of corrective suggestions.

[0061] After data extraction is complete, the following process is used to build a large model for generating a reflection report, which in turn generates a structured reflection report.

[0062] 1) Large-scale model construction and fine-tuning: Based on pre-trained large language models (such as LLaMA and ChatGLM), fine-tuning is performed using a reflective case dataset from the target task domain (containing 1000-5000 reflective cases within the target task domain). This optimizes the model's ability to understand training data, error cases, and technical terms, ensuring that the report generation accuracy is no less than 85%. Through domain-specific reflective case fine-tuning, the large language model can understand the decision-making logic and failure patterns in target domains such as healthcare and industry, improving the accuracy and professionalism of the generated reflective reports.

[0063] 2) Multimodal Input Processing: The training trajectory encoded data (processed using a Transformer encoder with 768 encoding dimensions), reward data, and historical memory data within the reflection window are fused in a multimodal manner to generate model input features, with a fusion error not exceeding 3%. Through multimodal fusion processing, the large language model can simultaneously perceive the training trajectory, reward feedback, and historical experience, providing more comprehensive input information for root cause analysis and optimization suggestion generation.

[0064] 3) Generate reports based on templates: Generate corresponding structured reflection reports according to preset standardized templates based on the reflection type (global / emergency).

[0065] The structured reflection report for global reflection covers the entire training data within the period, and the specific content of each field is as follows: ① Reflection basic information: including reflection identifier, reflection type (global), reflection time, training cycle (N training rounds), and training duration; ② Overall review of the training status within the cycle: average task completion rate, total number of successful training rounds and success rate, total number of failed training rounds and failure rate, average reward value, and trend of training loss within the cycle; ③ Statistics and classification of high-frequency decision-making biases: Decision-making biases that occur at least 5 times within the statistical period are classified according to the type of bias (perception bias, decision-making bias, action bias), and the frequency and scope of influence of each type of bias are marked. ④ Success / failure rate analysis of each task node: For each sub-task node, the success rate and failure rate are statistically analyzed, and the main reasons for failure at each node are analyzed (e.g., failure of the puncture sub-task is mostly due to instrument position deviation). ⑤ Common root cause analysis of failure cases: Screen similar failure cases within the period (similarity not less than 70%), analyze common root causes, and distinguish between subjective reasons (unreasonable model parameters, flawed decision-making logic) and objective reasons (environmental interference, insufficient sample). ⑥ Overall optimization direction and correction suggestions: For common root causes, propose 3-5 specific optimization directions, and each direction corresponds to clear correction suggestions (such as fine-tuning of model parameters, scene enhancement). ⑦ Performance validation metrics: Set the core validation metrics for the next round of training, including targets for improving task completion, reducing error rate, and improving training convergence speed.

[0066] The structured reflection report for emergency response covers the single or consecutive failure trajectory that triggered the event. The specific content of each field is as follows: ① Reflection basic information: including reflection identifier, reflection type (emergency), reflection time, triggering event type, and triggering time; ② Precise location of failure events: Identify the occurrence point, triggering action, corresponding observation status, and reward signal of the failure event, and mark the specific manifestations of the failure (such as tissue damage or insufficient safety distance). ③ Analysis of direct and root causes of failure: First, analyze the direct causes of failure (such as excessive force in the movement), then delve into the root causes (such as insufficient extraction of tissue tension by the sensory module), and verify the accuracy of the root causes by combining similar historical cases. ④ Correlation analysis between this failure and similar historical failures: Compare similar historical failure cases, analyze the commonalities and differences between this failure and historical failures, and determine whether they are caused by the same root causes; ⑤ Targeted Correction Directions and Emergency Optimization Suggestions: For the root cause, propose 1-3 immediately actionable correction suggestions to prioritize resolving the current failure issue; ⑥ Performance verification metrics: Set specific verification metrics for the next round of training to ensure that the revised strategy is verifiable (such as the absence of similar errors and a failure rate of 0).

[0067] By selecting different template fields according to the type of reflection and forcibly filling in standardized content, the structured reflection report is ensured to have a unified structure and complete information coverage, which facilitates subsequent structured parsing, machine reading, and multi-source data fusion attribution verification.

[0068] All reflection reports are generated using a unified natural language structured format, with the specific specifications as follows.

[0069] ① Field Specifications: Strictly follow the template field order when writing, with each field in a separate paragraph and labeled with its name, to avoid missing fields or disordered order.

[0070] ② Semantic Standardization: Use professional and rigorous technical language to avoid vague expressions (e.g., "significant deviation" should be clearly defined as "deviation exceeds the preset threshold of 0.5mm"), and ensure clear data labeling (e.g., "success rate 78%" should include the sample size). By eliminating ambiguous expressions, improve the accuracy of the large language model and subsequent semantic parsing units in understanding the reflection content (no less than 85%).

[0071] ③ Formatting Standards: The report uses a consistent font and line spacing throughout. Key data (such as success rate and root causes) are highlighted in bold. Failure cases and corrective suggestions are listed in sequence to ensure accurate extraction of core points during subsequent natural language processing. By standardizing the format, the reflection report achieves both human readability and machine parsing.

[0072] S104. Parse the structured reflection report, generate reflection memory units and store them in the reflection memory bank; the reflection memory bank supports retrieval by error type and optimization direction.

[0073] Specifically, the generated structured reflection report is parsed through the following process to generate standardized reflection memory units.

[0074] 1) Report parsing: Natural language processing technology is used to perform semantic parsing on each field of the structured reflection report, extract core information, and remove redundant expressions (such as repeated background descriptions). The parsing accuracy is no less than 90%.

[0075] 2) Memory Unit Construction: Each reflective memory unit contains 9 core fields, the content of which strictly corresponds to the parsed report information, as follows: ① Unique Identifier ID: Generated using the format "Reflection Type-Timestamp-Random Code" to ensure that each memory unit is unique, such as "Global-20260407-123456"; ②Task Scenario Tags: Label the type of the current training task (e.g., laparoscopic surgery, high-altitude live-line work) to facilitate subsequent retrieval by scenario; ③ Reflection Type Label: Label the reflection type (global reflection or emergency reflection) to distinguish the memory of different reflection scenarios; ④ Deviation or failure feature encoding: Vector encoding is used to encode the types and manifestations of deviations or failures, with the encoding dimensions consistent with the reflective memory bank (768 dimensions). ⑤ Attribution results: Extract the root cause analysis content from the reflection report, identify the core root causes (1 to 2), and avoid vague descriptions of multiple root causes; ⑥ Optimize the direction vector: Transform the optimized direction into a fixed-dimensional vector (768 dimensions) to facilitate subsequent similarity matching; ⑦ Correction suggestion text: Extract all correction suggestions from the reflection report, and indicate the specific operation and parameter range for each suggestion; ⑧ Associated Historical Memory ID: Label the ID of historical memory unit with a similarity of no less than 60% to the current memory unit to facilitate reuse across rounds; ⑨ Effect verification label: Initially labeled as "unverified", it will be updated to "effective", "partially effective" or "ineffective" after subsequent training is completed.

[0076] After the memory units are constructed, a vector database is built as a reflective memory bank through the following process to achieve efficient storage and retrieval of reflective memories.

[0077] 1) Vector Database Selection and Setup: Milvus or Pinecone vector databases will be selected, and a storage architecture adapted to reflexive memory units will be built. Storage capacity can be dynamically expanded, supporting efficient storage of millions of memory units. By selecting a high-performance vector database, the reflexive memory will maintain stable retrieval performance even in large-scale data scenarios.

[0078] 2) Vectorization of Memory Units: A text embedding model (such as BGE-large or Text-Embedding-3-large) is used to vectorize the structured information of each reflective memory unit, generating a 768-dimensional fixed-dimensional feature vector with a vectorization error of no more than 2%. Through high-precision vectorization, the semantic information of the reflective memory unit is transformed into a mathematical expression that can be used for similarity calculation.

[0079] 3) Vector Storage: The generated feature vectors are stored in a vector database, while the structured information (9 core fields) of the reflexive memory units are stored in an associated relational database, achieving a one-to-one correspondence between vectors and structured information, with a storage latency of no more than 50ms. This hybrid storage architecture of vector and relational databases supports both efficient similarity retrieval based on feature vectors and precise queries based on structured fields.

[0080] 4) Multi-dimensional Index Construction: Three core indexes are established to ensure efficient memory retrieval. The task scenario index is built according to task scenario tags, supporting fast filtering of memory units by scenario with a retrieval speed of no more than 10ms; the error type index is built according to deviation or failure feature encoding, supporting retrieval of similar memory units by error type; and the optimization direction index is built according to optimization direction vectors, supporting retrieval of memory units with similar optimization directions. Through the construction of multi-dimensional indexes, the reflection memory bank can quickly locate target memory units from multiple dimensions such as task scenario, error type, and optimization direction.

[0081] 5) Retrieval Performance Guarantee: Optimize the index structure to ensure that the retrieval latency for a single memory unit does not exceed 10ms, and the latency for batch retrieval (within 100 entries) does not exceed 50ms, achieving millisecond-level retrieval, matching, and reuse of reflective memories. This high-performance retrieval guarantee enables the agent to quickly retrieve relevant historical experiences during the reflection process, improving reflection efficiency.

[0082] In one embodiment, the reflective memory unit is vectorized and encoded to generate a feature vector of the reflective memory unit; the feature vector of the reflective memory unit is matched with the feature vector of the historical memory unit in the reflective memory bank for similarity; if the similarity is not lower than a preset threshold, the historical memory unit is updated based on the feature vector of the reflective memory unit; if the similarity is lower than the preset threshold, the reflective memory unit is stored as a new memory unit in the reflective memory bank.

[0083] Specifically, the newly generated reflective memory units are matched with existing historical memory units in the reflective memory bank using cosine similarity to avoid memory redundancy. The specific process is as follows.

[0084] 1) Similarity Calculation: The cosine similarity algorithm is used to calculate the similarity between the feature vector of the newly generated memory unit and the feature vectors of all historical memory units in the reflection memory bank. The similarity calculation accuracy is no less than 95%. High-precision similarity calculation ensures the reliability of the matching results.

[0085] 2) Matching and Processing: If the similarity is not lower than the preset threshold (recommended value 0.85, which can be adjusted according to the task scenario; for complex scenarios, it can be reduced to 0.8), it is determined to be a memory unit of the same type. The historical memory unit is then updated, specifically by supplementing new case data (current failed or deviation cases) and optimization effect data (if any), updating the associated historical memory ID, and retaining the original core information to avoid memory redundancy. If the similarity is lower than the preset threshold, it is determined to be a completely new memory unit. Its feature vector is stored in the vector database, and its structured information is stored in the relational database. At the same time, the index structure is updated to achieve continuous expansion of the memory system. Through similarity matching and differentiation processing, the reflection memory bank always maintains a state of high information density and low redundancy, which avoids the repeated storage of similar experiences and continuously absorbs new knowledge.

[0086] 3) Matching Records: The results of each similarity match (matched historical memory ID, similarity value, and processing method) are recorded and stored in the log of the reflection memory bank for easy tracking and optimization. Complete matching records provide data support for the operation monitoring and parameter tuning of the reflection memory bank.

[0087] S105. Perform semantic analysis and multi-source data fusion attribution verification on the reflective memory unit to identify the core root cause and generate correction strategies corresponding to each of the multiple training optimization dimensions.

[0088] In one embodiment, step S105 may include the following sub-steps.

[0089] 1) Perform semantic parsing on the reflective memory units to extract multiple reflective points and assign weights to each reflective point.

[0090] Specifically, a finely tuned domain-specific natural language understanding model (such as BERT, RoBERTa, or a large vertical domain model) is used to perform deep semantic parsing and extract key reflection points from the reflective memory units. First, the structured information of the reflective memory units is used as input, focusing on attribution results, correction suggestion text, and bias or failure feature encodings. Related historical memory unit information is also input to assist in parsing. Then, deep semantic analysis is performed on the input information to identify core information, including high-frequency bias types, root cause classifications, key optimization directions, constraint violations, and action adjustment suggestions. Redundant information is filtered during the parsing process, achieving an accuracy rate of no less than 85%. Next, the extracted reflective points are weighted based on the three core dimensions of bias, with a total weight of 1 to ensure reasonable priority in subsequent strategy generation. The evaluation criteria are as follows: **Impact Scope:** This dimension measures the number of task nodes affected by the deviation or failure. A larger impact scope results in a higher weight, with a value ranging from 0.3 to 0.5. **Occurrence Frequency:** This dimension measures the number of times the deviation or failure occurs during training. Higher frequency results in a higher weight, with a value ranging from 0.2 to 0.4. **Severity:** This dimension measures whether the deviation or failure violates hard constraints or leads to major failures. Higher severity results in a higher weight, with a value ranging from 0.3 to 0.5. Finally, the extracted reflection points and their corresponding weights are standardized for subsequent attribution verification. A log of the analysis process is also recorded for future traceability. Through deep semantic parsing and weight assignment, the structured information in the reflection memory unit is transformed into weighted reflection points.

[0091] 2) Collect multi-source data; among which, multi-source data includes basic model data of the intelligent agent, virtual environment data, and task constraint data.

[0092] Specifically, three core data types are collected: basic agent model data, virtual environment data, and task constraint data, to ensure the comprehensiveness of attribution verification. ① Basic agent model data includes model training gradient data, parameter configuration data, and loss change data, used to verify whether the root cause is related to model parameters; ② Virtual environment data includes environmental state transition data, environmental disturbance parameters, and physical rule configuration data, used to verify whether the root cause is related to environmental factors; ③ Task constraint data includes task hard constraint rules, operational specifications, and success or failure criteria, used to verify whether the root cause is related to constraint violation. Through the comprehensive collection of these three types of multi-source data, objective evidence from the three dimensions of model, environment, and constraints is provided for the attribution verification of key reflection points, ensuring the reliability of the verification results.

[0093] 3) For each reflection point, calculate the correlation coefficient between the reflection point and the agent's basic model data, virtual environment data and task constraint data, and perform weighted calculation on each correlation coefficient to obtain a weighted fusion correlation coefficient.

[0094] Specifically, the correlation coefficients between each reflection point and the agent's basic model data, virtual environment data, and task constraint data are calculated separately. Then, fusion weights are set according to the data correlations, with model data having a weight of 0.4, environment data having a weight of 0.3, and constraint data having a weight of 0.3. The three correlation coefficients are then weighted to obtain a weighted fusion correlation coefficient, with a fusion error not exceeding 3%. Through weighted fusion calculation, the degree of support of the three types of data for the reflection points is comprehensively considered, avoiding the one-sided judgment that may be caused by a single data source.

[0095] 4) If the weighted fusion correlation coefficient is greater than or equal to the preset correlation coefficient, the key point of reflection is determined to be the core root cause; if the weighted fusion correlation coefficient is less than the preset correlation coefficient, the key point of reflection is determined to be a misattribution caused by random environmental noise.

[0096] Specifically, if the correlation between the root cause and the multi-source data is not less than 70%, the attribution is considered accurate, and the reflection point is identified as the core root cause. If the correlation between the root cause and the multi-source data is less than 70%, the attribution is considered inaccurate, possibly due to random environmental noise, and the reflection point is identified as a misattribution. After excluding misattributions caused by random environmental noise, one or two core optimizable root causes are identified to ensure that the root causes can be resolved through subsequent correction strategies, and an attribution verification report is generated. By determining the correlation threshold, the true core root causes are effectively distinguished from accidental environmental noise interference, avoiding ineffective strategy adjustments due to misattributions.

[0097] 5) Generate correction strategies for each of the training data dimension, reward function dimension, model dimension, and training process dimension based on each core root cause, and determine the priority of each correction strategy based on the weight of each core root cause.

[0098] Specifically, based on the core reflection points (i.e., root causes) and their weights after fusion verification, a comprehensive correction strategy is generated, covering the entire process of agent training to ensure the relevance and executability of the correction strategy. The principle for generating the correction strategy is as follows: the priority of the correction strategy for each training optimization dimension is determined according to the weight of the core reflection points; the higher the weight of the core reflection point, the higher the priority of the corresponding correction strategy; the correction strategy must clearly define the operation content, parameter range, and execution steps to ensure that it can be directly injected into the training process.

[0099] In one embodiment, correction strategies are generated based on each core root cause, corresponding to the training data dimension, reward function dimension, model dimension, and training process dimension, specifically including the following four cases.

[0100] 1) If the core root cause is related to the training data, then generate a first correction strategy corresponding to the dimensions of the training data. The first correction strategy includes generating enhanced scene samples for high-frequency bias scenarios and expanding them into the experience replay pool.

[0101] Specifically, for high-frequency deviation or failure scenarios, a targeted virtual scene enhancement solution is generated according to the following process to address the issues of uneven sample distribution and insufficient targeting: ①Scenario Enhancement Needs Analysis: Based on the key points of reflection, clarify the types of scenarios that need enhancement (such as scenarios with different tissue tension during gallbladder separation) and the number of enhancements (50 to 200 sets, adjusted according to the deviation frequency). ② Enhanced scene generation: Based on the physical rules of the virtual simulation space, an enhanced scene that meets the requirements is generated, ensuring that the consistency error between the scene and the real scene does not exceed 5%, and the scene parameters cover all situations in which deviation occurs; ③ Sample expansion: Store 30% to 50% of the generated enhanced scene samples into the experience replay pool, and label the scene type and the corresponding deviation type to facilitate targeted learning by the model; ④ Sample update: Regularly clean up redundant samples in the experience replay pool (such as samples with a similarity of not less than 90%) to ensure the diversity of the sample pool and increase the sample utilization rate to more than 60%.

[0102] By generating enhanced samples for high-frequency deviation scenarios and expanding them into the experience replay pool, the lack of data in specific scenarios is compensated for, enabling the model to learn the correct decision-making patterns in deviation scenarios more fully.

[0103] 2) If the core root cause is related to the reward function, then a second correction strategy corresponding to the reward function dimension is generated. This second correction strategy includes adjusting the reward function weights, supplementing dense rewards, or amplifying the constraint penalty.

[0104] Specifically, to address the issues of sparse reward signals, insufficient constraint penalties, and biased target weights, the weight allocation of the reward function is automatically adjusted according to the following process: ① Reward function analysis: Based on the key points of reflection, analyze the shortcomings of the current reward function, such as the lack of rewards at key nodes and insufficient punishment. ② Weighting adjustment: Adjust the weights of rewards or penalties for each part according to the severity and frequency of the deviation, with the adjustment range controlled within ±50% to avoid over-adjustment; ③Intensive Reward Supplement: Intensive rewards are supplemented at key task nodes (such as puncture and separation nodes in surgery). For each key node completed without deviation, a positive reward of 10 to 50 is given, which solves the pain point of delayed rewards for long-cycle tasks. ④ Increased penalty intensity: Increased penalty intensity for behaviors that violate hard constraints or lead to major failures, with the penalty weight for a single violation increased by 50% to 100%, strengthening the model's learning of constraints; ⑤ Verify the adjustment effect: After adjustment, verify the rationality of the reward function through small-batch training (10 to 20 training rounds) to ensure that the reward signal can effectively guide the model optimization.

[0105] By adjusting the weight of the reward function, supplementing dense rewards, and amplifying the intensity of constraint penalties, the guiding role of the reward signal is optimized, enabling the agent to more clearly perceive the boundary between correct and incorrect behavior.

[0106] 3) If the core root cause is related to the model, a third correction strategy corresponding to the model dimension is generated. This third correction strategy includes adjusting the network structure or fine-tuning the model parameters.

[0107] Specifically, to address issues such as perceptual bias, flawed decision-making logic, and insufficient generalization ability, the following process is used to generate model fine-tuning or structural optimization solutions: ① Model problem identification: Combining key points of reflection with model training data, identify the core problems of the model, such as insufficient feature extraction in the perception module and logical deviation in the decision-making module; ② Model structure optimization (if needed): Adjust the network structure to address structural defects, such as adding an attention layer to the perception module, adjusting the number of neurons in the hidden layer, and supplementing the constraint loss function to ensure that the structure adapts to the task requirements; ③ Model parameter fine-tuning: The gradient descent method is used to fine-tune the model parameters, specifically including: freezing high-level parameters and only fine-tuning the parameters of the perception module, with the learning rate set to 0.001 to 0.005; adjusting the attention mechanism weights and increasing the attention ratio of core features (such as tissue tension and safety distance); optimizing the network dropout parameters (0.1 to 0.3) to improve the model's generalization ability; and supplementing the constraint loss function to strengthen the model's learning of hard constraints. ④ Fine-tuning and validation: After fine-tuning, validate the model performance through small-batch training (20 to 30 training rounds) to ensure that the model loss decreases and the bias is reduced.

[0108] By adjusting the network structure or fine-tuning the model parameters, the model's decision-making ability and generalization performance are directly improved, thereby reducing decision bias caused by model defects at its root.

[0109] 4) If the core root cause is related to the training process, a fourth correction strategy corresponding to the training process dimension is generated. This fourth correction strategy includes optimizing the task breakdown scheme or adjusting the training pace.

[0110] Specifically, to address issues such as unreasonable training pace and unscientific task breakdown, the course learning plan will be automatically adjusted according to the following process: ① Training process analysis: Based on the key points of reflection, analyze the shortcomings of the current training process, such as unreasonable sub-task training order and insufficient training of low success rate sub-tasks; ②Task decomposition and optimization: Break down complex tasks into multi-level sub-tasks, clarify the training order and training rounds of each sub-task, and ensure that training is gradual; ③Specialized training setup: For sub-tasks with low success rates (success rate below 60%), increase the number of specialized training rounds (50 to 200 training rounds) to prioritize improving the completion rate of sub-tasks; ④ Training rhythm adjustment: Adjust the number and duration of training rounds. Increase the number of training rounds for complex sub-tasks and decrease the number of training rounds for simple sub-tasks to avoid ineffective training; ⑤ Training plan update: Generate a new training plan, clarify the training objectives, training content and verification indicators for each stage, and ensure that the training process proceeds in an orderly manner.

[0111] By optimizing the task breakdown scheme and adjusting the training pace, the scientific nature and efficiency of the training process were improved, enabling the agent to gradually master various skills at a reasonable pace.

[0112] S106. Inject each correction strategy into the agent's training architecture according to priority and perform retraining.

[0113] Specifically, the following process is used to construct an autonomous iterative closed loop from training, reflection, optimization to retraining, thereby enabling the precise injection of correction strategies and the initiation of retraining.

[0114] 1) Closed-loop architecture construction: Clearly define the connection logic of each link in the closed loop, namely "online training → reflection triggering → reflection report generation → reflection memory storage → reflection analysis → policy generation → policy injection → retraining → memory update." Data is exchanged in real time between each link without human intervention. Through a complete closed-loop architecture design, the entire process from training to reflection to optimization is automated, enabling continuous self-optimization of the agent without human intervention.

[0115] 2) Strategy Priority Setting: Based on the weight of the key points of reflection and the degree of influence of root causes, set the execution priority for multi-dimensional correction strategies. For example, the priority from high to low is: model dimension, reward function dimension, training data dimension, and training process dimension, ensuring that core issues are addressed first. By clearly prioritizing, the most important optimization directions are handled first, avoiding the impact of improper strategy execution order on optimization results.

[0116] 3) Policy Injection Preparation: The generated multi-dimensional correction policies are organized into standardized execution instructions according to priority. The execution time, execution steps, and parameter settings of each correction policy are clearly defined to ensure that the correction policies can be directly injected into the agent's training architecture. Through standardized instruction organization, the correction policies can be accurately identified and executed by the agent's training architecture, reducing information loss or deviation during the correction policy injection process.

[0117] 4) Retraining Initiation: After injecting the multi-dimensional correction strategy into the agent's training architecture, the next round of online training is initiated. Simultaneously, training monitoring nodes are set up to track the policy execution effect in real time. Timely retraining initiation and effect monitoring ensure that the correction strategy can quickly take effect and provide data support for subsequent policy effectiveness evaluation.

[0118] The precise execution of the correction strategy injection is carried out in sequence according to the priority of each dimension of the correction strategy. The specific execution process is as follows.

[0119] It should be noted that the priority of each correction strategy is determined based on the weight of each core root cause. The higher the weight of the core root cause, the higher the priority of the correction strategy for its corresponding dimension. The following is only an example of priority execution order. In actual execution, the priority is dynamically adjusted according to the weight of the core root cause and is not limited to the fixed priority order described below.

[0120] 1) Execution of the training data dimension correction strategy: First, the scene generation code in the virtual simulation space is automatically modified to add logic for generating enhanced scenes, ensuring that targeted enhanced scenes can be automatically generated during subsequent training. Second, the generated enhanced scene samples are stored in the experience replay pool according to a preset ratio, and the sample index is updated to facilitate rapid model retrieval and learning. Finally, the accuracy of scene generation and the integrity of sample storage are checked to ensure that the strategy is executed correctly. Through the precise execution of the data layer correction strategy, the lack of data in specific scenarios is compensated for, enabling the model to more fully learn the correct decision-making patterns in biased scenarios.

[0121] 2) Execution of the reward function dimension correction strategy: First, the code of the agent's reward calculation module is modified in real time, updating the weight allocation, dense reward settings, and penalty intensity parameters of the reward function. After modification, the adjusted reward rules are immediately adopted in the next training round without restarting the training process. Finally, the accuracy of the reward calculation is verified through 1 to 2 training rounds to ensure that the reward signal meets the adjustment requirements. Through the precise execution of the reward function layer correction strategy, the guiding role of the reward signal is optimized, enabling the agent to more clearly perceive the boundary between correct and incorrect behaviors.

[0122] 3) Execution of Model-Dimensional Correction Strategies: First, modify the model network structure (if necessary) according to the correction strategy requirements, fine-tune the model parameters, and generate an optimized model file. Second, replace the original base decision model with the optimized model file as the initial model for the next round of training. Finally, check the correctness of the model structure and the accuracy of parameter fine-tuning to ensure the model can operate normally. Through the precise execution of model-dimensional correction strategies, the model's decision-making ability and generalization performance are directly improved, fundamentally reducing decision bias caused by model defects.

[0123] 4) Execution of training process-level correction strategies: First, modify the task breakdown scheme, training round arrangement, and sub-task weight settings for course learning to generate a new training plan; second, synchronize the new training plan to the online training module to ensure that subsequent training is executed according to the new plan; finally, check the rationality of the training plan to ensure that the sub-task training order and specialized training rounds meet the strategy requirements. Through the precise execution of training process-level correction strategies, the scientific nature and efficiency of the training process are improved, enabling the agent to gradually master various skills at a reasonable pace.

[0124] Based on the training architecture after the injection of the correction policy, the agent performs a new round of training and verifies the effectiveness of the correction policy according to the following process.

[0125] 1) Retraining execution: The agent performs a new round of online training under the updated virtual space, model architecture and training plan. The training process is consistent with the online training process to ensure the continuity of the training process.

[0126] 2) Real-time monitoring of results: Set up training monitoring nodes and collect core training indicators every 10 training rounds, including task completion, error rate, reward value and training loss, and track the trend of indicator changes in real time.

[0127] 3) Validation: Compare the monitored training metrics with the preset validation metrics in the reflection report to verify the effectiveness of the modified strategy. If the metrics meet the validation objectives (e.g., task completion rate increases by at least 10%, error rate decreases by at least 50%), the strategy is considered effective; if the metrics do not meet the validation objectives but show significant improvement (e.g., task completion rate increases by 5% to 10%), the strategy is considered partially effective; if the metrics show no improvement or even worsen, the strategy is considered ineffective.

[0128] 4) Verification result recording: Record the verification results (valid, partially valid or invalid) and the corresponding indicator data into the corresponding reflective memory unit to provide a basis for subsequent memory updates.

[0129] S107. Update the reflective memory bank based on the effect verification data of retraining.

[0130] Specifically, after a new round of training is completed, the effectiveness of the revised strategy is evaluated and the reflective memory units are updated through the following process.

[0131] 1) Data Extraction for Effectiveness Validation: Extract all core metrics data from the new round of training, including task completion rate, error rate, reward value, training loss, recurrence rate of similar errors, and execution data of strategies at each level. This comprehensive data extraction provides a complete quantitative basis for evaluating strategy effectiveness.

[0132] 2) Strategy Effectiveness Evaluation: Based on the extracted data and the verification indicators in the reflection report, a comprehensive evaluation of the revised strategies generated in this reflection will be conducted to determine whether the evaluation result is effective, partially effective, or ineffective, with an evaluation accuracy rate of no less than 90%. Through objective quantitative evaluation, it is ensured that the actual effect of each revised strategy is accurately recorded.

[0133] 3) Memory Unit Updates: Evaluation results are added as effect tags to the corresponding reflective memory units, along with the evaluation time and core validation metrics data, ensuring the completeness of the memory unit information. Timely memory unit updates ensure that each memory in the reflective memory bank has a clear effect tag, facilitating subsequent retrieval and reuse.

[0134] For different effect verification data, update the reflection memory bank through the following process to ensure its practicality and accuracy.

[0135] 1) Handling of Effective or Partially Effective Strategies: If the validation data from retraining indicates that the modified strategy is effective or partially effective, the modified strategy is added to the corresponding reflective memory unit and marked as a reusable strategy. Specifically, firstly, the optimization performance data from the new round of training (such as the improvement in task completion rate and the decrease in error rate) and new training trajectory data are added to the corresponding reflective memory unit; secondly, the optimization direction vector and bias or failure feature encoding of the memory unit are updated to improve the accuracy of subsequent memory retrieval; finally, the effective or partially effective modified strategies are marked as reusable strategies, facilitating rapid recall for similar tasks and similar biases in the future. By marking and reusing effective strategies, successful optimization experience can be accumulated and retained, allowing for direct recall when encountering similar problems in the future, avoiding repeated analysis and trial and error.

[0136] 2) Handling Ineffective Policies: If the validation data from retraining indicates that the revised policy is ineffective, the reasons for ineffectiveness are analyzed and a second reflection is triggered to generate a new revised policy. Specifically, firstly, combining the new round of training data, model data, and environment data, the core reasons for the policy's ineffectiveness are analyzed, such as incorrect root cause judgment, unreasonable policy parameters, or excessive environmental interference. Secondly, based on the analysis of the reasons for ineffectiveness, new reflection points are generated to clarify new optimization directions. Finally, the second reflection process is automatically triggered, and the revised policy is regenerated based on the new reflection points and injected into the next round of training to prevent the ineffective policy from continuously affecting the training effect. Through the analysis of the causes of ineffective policies and the second reflection, it is ensured that every failure can be transformed into a new learning opportunity, driving the continuous evolution of the reflection mechanism.

[0137] Based on the updated reflection memory bank, the reflection mechanism is self-optimized and the training loop is continuously iterated. The specific process is as follows.

[0138] 1) Self-optimization of the reflection mechanism: On the one hand, using updated reflection memory bank data, the reflection report generation model, reflection memory analysis model, and strategy generation model are fine-tuned to optimize the semantic parsing accuracy and strategy generation relevance, with each fine-tuning resulting in a performance improvement of at least 5%. On the other hand, parameters such as the reflection trigger threshold, similarity matching threshold, and strategy correction constraint boundary are dynamically adjusted based on training results to ensure the reflection mechanism adapts to the training progress. Through the self-optimization of the reflection mechanism, the reflection capability of the entire system continuously improves with the accumulation of training experience.

[0139] 2) Closed-Loop Iteration Training: Real-time monitoring of the agent's task performance metrics, compared with preset convergence criteria, including average task completion rate not lower than a preset threshold over a preset number of consecutive training rounds, recurrence rate of the same type of error not higher than a preset threshold, and training loss fluctuation not exceeding a preset range. If the convergence criteria are met, training is terminated, the optimized agent model is output, and the final reflection memory, training data, and correction strategies are compiled and archived for future reuse. If the convergence criteria are not met, a closed-loop iteration from training, reflection, optimization to retraining continues, repeating the corresponding steps until the convergence criteria are met. Through explicit convergence criteria and iteration decisions, the training process is ensured to terminate in a timely manner after reaching the expected performance, avoiding the waste of resources from ineffective training.

[0140] 3) Iteration Log: The process, effects, and parameter adjustments of each closed-loop iteration are recorded in detail and stored in the system log for easy tracking, optimization, and review. This comprehensive iteration log provides complete data support for continuous system improvement and troubleshooting.

[0141] The self-training optimization method for intelligent agents provided by this invention constructs a virtual simulation space and configures basic reflection parameters, enabling the agent to autonomously trigger a reflection mechanism during training. This allows for structured attribution analysis of failure experiences, generating a reflection report containing root cause analysis. By parsing the reflection results into searchable reflection memory units and identifying core root causes based on multi-source data fusion attribution verification, a multi-dimensional correction strategy is generated, achieving autonomous learning and continuous optimization from failures. By injecting the correction strategy into the training architecture according to priority and executing retraining, a fully autonomous closed loop from training, reflection, optimization to retraining is formed, completing self-training optimization without human intervention. Compared with traditional training methods, this method effectively reduces the training cost of intelligent agents in virtual space, improves training stability and generalization ability, and is particularly suitable for high-reliability and high-efficiency training requirements in security-sensitive scenarios.

[0142] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of the intelligent agent self-training optimization device provided by the present invention. Figure 2 As shown, the device may include: The training initialization module 201 is used to construct a virtual simulation space that matches the target training task, initialize the basic model of the agent, and configure the basic parameters for reflection. The agent training module 202 is used to drive the agent to perform online training in the virtual simulation space; The reflection condition triggering module 203 is used to trigger the corresponding reflection when any preset reflection condition is detected. The reflection report generation module 204 is used to respond to the triggering of reflection, extract the training data and relevant historical memory data within the reflection window, and input the training data and relevant historical memory data into the pre-trained large language model to generate a structured reflection report containing root cause analysis. The Reflection Memory Management Module 205 is used to parse structured reflection reports, generate reflection memory units, and store them in the reflection memory bank; the reflection memory bank supports retrieval by error type and optimization direction; The correction strategy generation module 206 is used to perform semantic parsing and multi-source data fusion attribution verification on the reflective memory unit, lock in the core root cause, and generate correction strategies corresponding to multiple training optimization dimensions. The closed-loop training optimization module 207 is used to inject each correction strategy into the training architecture of the agent according to priority and perform retraining. The Reflection Memory Update Module 208 is used to update the reflection memory library based on the effect verification data of retraining.

[0143] In some embodiments, the intelligent agent self-training optimization device of the present invention can be implemented in a combination of hardware and software. As an example, the intelligent agent self-training optimization device of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the intelligent agent self-training optimization method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0144] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0145] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned agent self-training optimization methods. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the agent self-training optimization method shown in any embodiment of the present invention by calling the computer program.

[0146] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0147] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0148] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0149] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0150] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0151] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0152] It should be noted that, Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0153] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described intelligent agent self-training optimization methods.

[0154] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0155] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned agent self-training optimization method.

[0156] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0157] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0158] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0159] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0160] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0161] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0162] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0163] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A self-training optimization method for an intelligent agent, characterized in that, include: Construct a virtual simulation space that matches the target training task, initialize the basic model of the agent, and configure the basic parameters for reflection; The intelligent agent is driven to perform online training in the virtual simulation space, and when any preset reflection condition is detected, the corresponding reflection is triggered; In response to the triggering of reflection, the training data and relevant historical memory data within the reflection window are extracted, and the training data and relevant historical memory data are input into the pre-trained large language model to generate a structured reflection report containing root cause analysis. The structured reflection report is parsed to generate reflection memory units and stored in the reflection memory bank; the reflection memory bank supports retrieval by error type and optimization direction; Semantic parsing and multi-source data fusion attribution verification are performed on the reflexive memory unit to identify the core root cause and generate correction strategies corresponding to each of the multiple training optimization dimensions. The aforementioned correction strategies are injected into the training architecture of the agent according to their priority and then retraining is performed. The reflective memory bank is updated based on the effectiveness verification data from retraining.

2. The agent self-training optimization method according to claim 1, characterized in that, The configuration reflects the basic parameters, including: Configure the relevant parameters for the periodic triggering conditions of global reflection and the event triggering conditions of emergency reflection in the dual-mode reflection triggering rules; wherein, the periodic triggering condition is that the training rounds reach a preset number of rounds or the training duration reaches a preset duration; the event triggering condition is that a preset failure event occurs during the training process; Configure the filling specifications and data format of each field in the reflection report generation template; wherein, each field in the reflection report generation template includes reflection identifier, reflection type, reflection time, reflection window period, training data statistical indicators, deviation or failure analysis indicators, optimization suggestion indicators, and effect verification indicators; The storage structure of the reflexive memory is configured as a hybrid storage structure including a vector database and a relational database; wherein, the vector database is used to store the feature vectors of the reflexive memory units; and the relational database is used to store the structured information of the reflexive memory units. The pre-trained large language model is configured to be fine-tuned using a reflective case dataset from the target task domain; Configure the constraint boundaries of each of the aforementioned correction strategies; wherein, the constraint boundaries include the fine-tuning range of model parameters, the weight adjustment range of the reward function, and the number of scene enhancement samples generated; Configure the convergence criteria for the agent; wherein the convergence criteria include average task completion rate, recurrence rate of similar errors, and training loss fluctuation.

3. The agent self-training optimization method according to claim 2, characterized in that, The response to triggering reflection involves extracting training data and relevant historical memory data within the reflection window, including: In response to triggering the global reflection, the full training data of the current training cycle is extracted as the training data within the reflection window. In response to triggering the emergency reflection, training data of a preset number of steps before and after the preset failure event is extracted as training data within the reflection window period; Historical memory data related to the type and error type of the target training task are retrieved from the basic memory bank, the task process knowledge base, and the reflection memory bank.

4. The agent self-training optimization method according to claim 2, characterized in that, The method further includes: The reflective memory unit is vectorized and encoded to generate the feature vector of the reflective memory unit; The feature vector of the reflective memory unit is matched with the feature vector of the historical memory unit in the reflective memory bank based on similarity. If the similarity is not lower than a preset threshold, the historical memory unit is updated based on the feature vector of the reflective memory unit; If the similarity is lower than the preset threshold, the reflective memory unit is stored as a new memory unit in the reflective memory bank.

5. The agent self-training optimization method according to claim 1, characterized in that, The process involves semantic parsing and multi-source data fusion attribution verification of the reflective memory unit to identify the core root cause and generate correction strategies corresponding to multiple training optimization dimensions, including: Semantic parsing is performed on the reflection memory unit to extract multiple reflection points and assign weights to each reflection point; Collect the multi-source data; wherein, the multi-source data includes the agent's basic model data, virtual environment data, and task constraint data; For each of the aforementioned reflection points, the correlation coefficient between the reflection point and the agent's basic model data, virtual environment data, and task constraint data is calculated, and the correlation coefficients are weighted to obtain a weighted fusion correlation coefficient. If the weighted fusion correlation coefficient is greater than or equal to the preset correlation coefficient, then the reflection point is determined to be the core root cause; If the weighted fusion correlation coefficient is less than the preset correlation coefficient, then the reflection point is determined to be a misattribution caused by environmental random noise. Based on each of the core root causes, corresponding correction strategies are generated for the training data dimension, reward function dimension, model dimension, and training process dimension, and the priority of each correction strategy is determined based on the weight of each of the core root causes.

6. The agent self-training optimization method according to claim 5, characterized in that, The correction strategies generated based on the core root causes for the training data dimension, reward function dimension, model dimension, and training process dimension respectively include: If the core root cause is related to the training data, a first correction strategy corresponding to the dimension of the training data is generated; wherein, the first correction strategy includes generating enhanced scene samples for high-frequency deviation scenarios and expanding them into the experience replay pool. If the core root cause is related to the reward function, a second correction strategy corresponding to the dimension of the reward function is generated; wherein, the second correction strategy includes adjusting the weight of the reward function, supplementing dense rewards, or amplifying the constraint penalty intensity; If the core root cause is related to the model, a third correction strategy corresponding to the model dimension is generated; wherein, the third correction strategy includes adjusting the network structure or fine-tuning the model parameters; If the core root cause is related to the training process, a fourth correction strategy corresponding to the training process dimension is generated; wherein, the fourth correction strategy includes optimizing the task decomposition scheme or adjusting the training rhythm.

7. The agent self-training optimization method according to claim 1, characterized in that, The update of the reflective memory bank based on the performance validation data from retraining includes: If the retraining effect verification data shows that the correction strategy is effective or partially effective, then the correction strategy is added to the corresponding reflective memory unit and marked as a reusable strategy. If the retraining effect verification data indicates that the correction strategy is ineffective, the reasons for the ineffectiveness are analyzed and a second reflection is triggered to generate a new correction strategy.

8. A self-training optimization device for intelligent agents, characterized in that, include: The training initialization module is used to construct a virtual simulation space that matches the target training task, initialize the basic model of the agent, and configure the basic parameters for reflection. An agent training module is used to drive the agent to perform online training in the virtual simulation space; The reflection condition trigger module is used to trigger the corresponding reflection when any preset reflection condition is detected. The reflection report generation module is used to respond to the triggering of reflection, extract training data and relevant historical memory data within the reflection window, and input the training data and relevant historical memory data into the pre-trained large language model to generate a structured reflection report containing root cause analysis. The reflection memory management module is used to parse the structured reflection report, generate reflection memory units, and store them in the reflection memory library; the reflection memory library supports retrieval by error type and optimization direction; The correction strategy generation module is used to perform semantic parsing and multi-source data fusion attribution verification on the reflective memory unit, lock in the core root cause, and generate correction strategies corresponding to multiple training optimization dimensions. A closed-loop training optimization module is used to inject the aforementioned correction strategies into the training architecture of the agent according to priority and perform retraining. The reflective memory update module is used to update the reflective memory bank based on the effect verification data of retraining.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the agent self-training optimization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the agent self-training optimization method as described in any one of claims 1 to 7.