A reinforcement learning partition pre-interaction method and system
By constructing a bias metric function to divide the trust interval in deep reinforcement learning and adopting a differentiated interaction strategy, the problem of deviation between virtual pre-training and the real environment is solved, thereby improving the reliability and training efficiency of the agent.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGNAN UNIV
- Filing Date
- 2026-04-10
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, the virtual pre-training of deep reinforcement learning agents and the model deviation from the real environment lead to a decrease in the reliability of the agent in solving the problem. Furthermore, virtual pre-training may increase the burden and risk of subsequent online fine-tuning.
We employ a reinforcement learning partitioning pre-interaction method, which divides the trust interval of the real environment by constructing a bias metric function, and uses differentiated environment interaction strategies and empirical data sampling mechanisms to adjust parameters in combination with virtual and real environments, including coarse and fine parameter tuning stages.
By effectively combining virtual and real environments, the robustness of intelligent agents can be improved, the effectiveness of virtual interaction can be maximized, the dependence on real interaction can be reduced, and safe, efficient and reliable parameter adjustment can be achieved.
Smart Images
Figure CN122154741A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of reinforcement learning technology, and in particular to a reinforcement learning partitioning pre-interaction method and system. Background Technology
[0002] Deep Reinforcement Learning (DRL) has shown great potential in solving complex sequential decision-making problems, such as robot control, autonomous driving, and industrial process optimization. However, training DRL agents typically relies on extensive trial-and-error interactions with the environment, which is costly, risky, or simply impractical in many real-world applications.
[0003] To reduce reliance on real-world interactions, pre-training based on virtual models has become a mainstream approach. This method involves large-scale, low-cost pre-training of the agent in a computer-simulated virtual environment. Once the agent has mastered the basic strategies, it is then transferred to a real environment for online fine-tuning. However, the effectiveness of this method hinges on a crucial prerequisite: the virtual model must exhibit a high degree of consistency with the real environment. In practical engineering, due to factors such as model simplification, unmodeled dynamics, parameter mismatch, and time-varying environments, the virtual environment inevitably deviates from the real environment—a phenomenon known as model bias.
[0004] When model bias is significant, purely virtual pre-training can lead to serious problems: the optimal policy learned by the agent in the virtual environment may be based on a mismatched environmental dynamics model, resulting in poor performance or even complete failure in the real environment. This negative transfer phenomenon not only renders the initial pre-training ineffective and wastes computational resources, but may also generate misleading and dangerous policies, increasing the burden, time, and security risks of the subsequent online fine-tuning phase. The agent needs to forget erroneous knowledge and readjust to the real environment, leading to reduced overall training efficiency and decreased convergence reliability.
[0005] Therefore, there is an urgent need in this field for a method to adjust the parameters of a reinforcement learning agent that can sense and adapt to model biases, so as to improve the reliability of the agent in solving the final problem. Summary of the Invention
[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the parameter adjustment of the agent in the prior art of deep reinforcement learning cannot effectively combine the virtual environment and the real environment, which leads to a decrease in the reliability of the agent in solving the problem.
[0007] To address the aforementioned technical problems, this invention provides a reinforcement learning partitioning pre-interaction method, comprising:
[0008] Step S1: Construct a deviation measurement function to evaluate the model deviation of the virtual environment in the pre-interaction framework relative to the real environment; divide the working space of the real environment into multiple trust intervals based on the model deviation, wherein the trust intervals include an absolute trust interval, a relatively trust interval, and a non-trust interval.
[0009] Step S2: Based on the trust interval, a differentiated environment interaction strategy is adopted to coarsely tune the parameters of the reinforcement learning agent, so as to differentiate the frequency of the agent's interaction with the real environment within the corresponding trust interval.
[0010] Step S3: In the parameter coarse adjustment stage, the experience data generated by the interaction between the agent and the virtual environment is stored in the virtual experience pool, and the experience data generated by the interaction between the agent and the real environment is stored in the real experience pool. Data is sampled from the virtual experience pool and the real experience pool based on preset priority weights to update the network parameters of the agent.
[0011] Step S4: Monitor the performance indicators of the agent during the parameter coarse adjustment stage. When the performance indicators meet the preset interaction mode switching conditions, control the agent to end the parameter coarse adjustment and enter the parameter fine adjustment stage. Otherwise, continue in the parameter coarse adjustment stage until the performance indicators meet the preset interaction mode switching conditions.
[0012] Step S5: In the parameter fine-tuning stage, the agent fully interacts with the real environment and adjusts the agent parameters until convergence. The agent with converged parameters is used to solve complex sequence decision problems.
[0013] In one embodiment of the present invention, the deviation measurement function of step S1 Used to reflect operating condition variables The relative deviation between the steady-state gain of the virtual environment and the real environment is expressed as:
[0014] ;
[0015] in, For operating condition variables, and These respectively represent the operating conditions Below, the steady-state gain of the virtual environment versus the real environment; It is an absolute value.
[0016] In one embodiment of the present invention, the method for dividing the working space of the real environment into multiple trust intervals based on the model deviation in step S1 includes:
[0017] Based on the aforementioned deviation measurement function The calculation results are compared with the preset deviation threshold to reflect the actual working conditions in the real environment. The space is divided into three trust zones, namely:
[0018] Absolute Trust Range :
[0019] ;
[0020] distrust interval :
[0021] ;
[0022] More Confidence Range :
[0023] ;
[0024] in, and These are the high and low thresholds for dividing the trust interval, respectively.
[0025] In one embodiment of the present invention, the differentiated environmental interaction strategy in step S2 includes:
[0026] Within an absolute trust range with minimal deviation In this case, the agent interacts only with the virtual environment to maximize efficiency;
[0027] Within a moderately confident range In this scenario, the intelligent agent primarily interacts with the virtual environment, supplemented by interactions with the real environment. The interaction ratio between the virtual and real environments is 9:1, with a small amount of interaction with the real environment used to achieve online correction of the intelligent agent.
[0028] In the distrust interval with large deviation In this way, the intelligent agent can fully interact with the real environment, avoiding the misleading effects of model bias.
[0029] In one embodiment of the present invention, in step S3, when sampling data from the virtual experience pool and the real experience pool, the experience data from the real experience pool is assigned a higher sampling weight than the experience data from the virtual experience pool, so as to efficiently utilize the scarce data of the agent's interaction with the real environment. The sample data... Probability of being sampled Designed as follows:
[0030] ;
[0031] in, Weights are sampled based on real experience. Sample weights for virtual experience / =2, This represents the number of samples in the real experience pool. This represents the number of samples in the virtual experience pool.
[0032] In one embodiment of the present invention, in step S4,
[0033] The performance metric of the agent in the parameter coarse-tuning phase is the performance evaluation of the algorithmic decision-making that controls the agent's parameter adjustment in a virtual environment, and the performance evaluation function is designed as follows:
[0034] ;
[0035] in, For intelligent agent algorithm decision making, For reward decay coefficient, As the reward for the current round, The number of rounds for testing the trajectory;
[0036] After each training round, the agent's algorithmic decisions are run in a virtual environment. The performance average of the trajectory is constructed based on the performance evaluation function after a complete set trajectory. with standard deviation :
[0037] ;
[0038] ;
[0039] The average performance of the trajectory with standard deviation Design the conditions for switching interaction modes.
[0040] In one embodiment of the present invention, step S4, when the performance index meets the preset interaction mode switching conditions, includes:
[0041] When the average performance of the trajectory Meet the minimum standards To ensure that the agent's algorithmic decision-making possesses basic control capabilities and standard deviation. Less than the threshold To ensure that the agent's algorithmic decision-making has stabilized, it indicates that the performance indicators meet the preset interaction mode switching conditions, expressed as:
[0042] and .
[0043] To address the aforementioned technical problems, this invention provides a reinforcement learning partitioning pre-interaction system, comprising:
[0044] The construction module is used to construct a deviation measurement function to evaluate the model deviation of the virtual environment in the pre-interaction framework relative to the real environment; based on the model deviation, the working space of the real environment is divided into multiple trust intervals, wherein the trust intervals include an absolute trust interval, a relatively trust interval, and a non-trust interval.
[0045] The segmentation module is used to coarsely tune the parameters of the reinforcement learning agent based on the trust interval and using differentiated environmental interaction strategies, so as to differentiate the frequency of the agent's interaction with the real environment within the corresponding trust interval.
[0046] Parameter coarse tuning module: During the parameter coarse tuning stage, it stores the experience data generated by the interaction between the agent and the virtual environment into the virtual experience pool, stores the experience data generated by the interaction between the agent and the real environment into the real experience pool, and samples data from the virtual experience pool and the real experience pool based on preset priority weights to update the network parameters of the agent.
[0047] Judgment module: Used to monitor the performance indicators of the agent during the parameter coarse adjustment stage. When the performance indicators meet the preset interaction mode switching conditions, the agent is controlled to end the parameter coarse adjustment and enter the parameter fine adjustment stage. Otherwise, it continues to be in the parameter coarse adjustment stage until the performance indicators meet the preset interaction mode switching conditions.
[0048] Parameter fine-tuning module: During the parameter fine-tuning stage, the agent fully interacts with the real environment to adjust the agent's parameters until convergence. The agent with converged parameters is used to solve complex sequence decision problems.
[0049] To address the aforementioned technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the reinforcement learning partition pre-interaction method described above.
[0050] To address the aforementioned technical problems, the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the reinforcement learning partition pre-interaction method described above.
[0051] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:
[0052] The reinforcement learning partitioning pre-interaction method described in this invention can effectively combine virtual and real environments, ensuring the reliability of the agent's final problem-solving after parameter adjustment.
[0053] This invention can effectively improve the robustness of reinforcement learning agents, maximize the utility of virtual interaction, minimize dependence on expensive real interaction, and achieve safe, efficient, and reliable parameter adjustment. Attached Figure Description
[0054] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.
[0055] Figure 1 This is a flowchart of the method of the present invention;
[0056] Figure 2 This is a schematic diagram illustrating the division of the real-world working environment into multiple trust intervals in an embodiment of the present invention.
[0057] Figure 3 This is a structural diagram of the liquid level control system for a single-tank water tank in an embodiment of the present invention;
[0058] Figure 4 This is a flowchart of the TD3 algorithm used in the embodiments of the present invention;
[0059] Figure 5 This is a policy network architecture diagram in an embodiment of the present invention;
[0060] Figure 6 This is a comparison chart of the reward curves for coarse parameter adjustment and conventional parameter adjustment in this invention. Detailed Implementation
[0061] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.
[0062] Example 1
[0063] Reference Figure 1 As shown, this invention relates to a reinforcement learning partitioning pre-interaction method, comprising:
[0064] Step S1: Construct a deviation measurement function to evaluate the model deviation of the virtual environment in the pre-interaction framework relative to the real environment; divide the working space of the real environment into multiple trust intervals based on the model deviation, wherein the trust intervals include an absolute trust interval, a relatively trust interval, and a non-trust interval.
[0065] Step S2: Based on the trust interval, a differentiated environment interaction strategy is adopted to coarsely tune the parameters of the reinforcement learning agent, so as to differentiate the frequency of the agent's interaction with the real environment within the corresponding trust interval.
[0066] Step S3: In the parameter coarse adjustment stage, the experience data generated by the interaction between the agent and the virtual environment is stored in the virtual experience pool, and the experience data generated by the interaction between the agent and the real environment is stored in the real experience pool. Data is sampled from the virtual experience pool and the real experience pool based on preset priority weights to update the network parameters of the agent.
[0067] Step S4: Monitor the performance indicators of the agent during the parameter coarse adjustment stage. When the performance indicators meet the preset interaction mode switching conditions, control the agent to end the parameter coarse adjustment and enter the parameter fine adjustment stage. Otherwise, continue in the parameter coarse adjustment stage until the performance indicators meet the preset interaction mode switching conditions.
[0068] Step S5: In the parameter fine-tuning stage, the agent fully interacts with the real environment and adjusts the agent parameters until convergence. The agent with converged parameters is used to solve complex sequence decision problems.
[0069] The following is a detailed description of this embodiment:
[0070] This embodiment uses the liquid level of a single water tank as the controlled object and applies a PID control system. The control system structure is as follows: Figure 3 As shown. The reinforcement learning algorithm chosen is the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, and the algorithm flow is as follows. Figure 4 As shown. The PID parameters of the intelligent agent output controller are the proportional gain coefficients. Integral gain coefficient and differential gain coefficient The optimization objective of reinforcement learning is the system's control effect. The environment returns the control error. Error integral and error differential Policy network architecture such as Figure 5 As shown, a complete coarse-tuning process for reinforcement learning parameters based on trust partitioning is constructed and compared with traditional reinforcement learning parameter tuning methods.
[0071] Real-world environment of a single-tank water level system for:
[0072]
[0073]
[0074] Description of the liquid level system for a single-tank water tank: The inlet water flow rate is the control input of the control system. The outlet water flow rate is not directly controlled by the system. The opening degree of the outlet valve is preset and fixed. Therefore, the outlet water flow rate is entirely determined by the current water tank level. Decide.
[0075] A simplified model of the liquid level system of a single water tank, i.e., a virtual environment. for:
[0076]
[0077] in, Cross-sectional area The nominal value, The flow coefficient of the outlet valve The nominal value.
[0078] Sources of model bias:
[0079] 1. Linearization error: The virtual environment assumes that the outflow velocity and liquid level are related. It is directly proportional to the water flow rate in the real environment simulation environment, while the water flow rate in the simulation environment is directly proportional to the water flow rate in the real environment. Related, and in The situation becomes more complex under these influences. This is the main source of model bias.
[0080] 2. Parameter mismatch: The cross-sectional area used by the virtual environment and Flow coefficient of the outlet valve It is the nominal value, and the actual value. , There are differences.
[0081] Reference Figure 1 As shown, this invention relates to a method for coarse tuning of reinforcement learning parameters based on trust partitioning. The method specifically includes the following steps:
[0082] S1. This step aims to quantify the virtual environment. Simulated environment with real environment The differences between them are identified, and different trust work areas are divided according to the degree of difference.
[0083] First, determine the state variables used to characterize the system's operating conditions. In this embodiment, the liquid level of the controlled object is used as the operating condition variable. In other embodiments, other observable states such as pressure, flow rate, and temperature can be selected based on the system characteristics.
[0084] Secondly, define the deviation measurement function. Used to calculate at a specific liquid level Below, virtual environment Simulated environment with real environment The relative deviation in the steady-state response. The specific functional form is as follows:
[0085]
[0086] in, and These represent the liquid levels. Below, virtual environment With real environment steady-state gain, The absolute value. Deviation measurement function. It intuitively reflects the specific liquid level. The accuracy of the model.
[0087] The model parameters are set to: Maximum safe water level in the water tank Minimum working liquid level in the water tank , , , , .
[0088] Deviation threshold set to and According to the deviation measurement function Based on the calculation results, the entire operating space is divided into the following trust intervals:
[0089] Absolute Trust Range :
[0090]
[0091] distrust interval :
[0092]
[0093] More Confidence Range :
[0094]
[0095] S2. Based on the trust interval to which the currently observed operating condition belongs, the agent interacts and learns in different environments. The algorithm flow is as follows: Figure 2 As shown.
[0096] like Then the intelligent agent only interacts with the virtual environment. Interaction. This strategy aims to maximize the use of low-cost, high-efficiency virtual interactions.
[0097] like Then the intelligent agent only interacts with the real environment. Interaction. This strategy aims to completely avoid being misled by model bias and learn the correct dynamics of the region through real-world interaction data.
[0098] like In this case, the intelligent agent adopts a hybrid interaction strategy, with the interaction ratio between the virtual environment and the real environment being 9:1. This strategy aims to use virtual interaction as the main method, supplemented by a small amount of real interaction, to correct model biases and achieve online fine-tuning of the model.
[0099] S3. Intelligent Agents and Experience tuples generated by interaction Stored in the virtual experience pool Intelligent agents and The experiences generated through interaction are stored in the real experience pool. .
[0100] Due to the high cost and scarcity of real-world interactions, they are given higher priority during sampling to prevent them from being overwhelmed by massive amounts of virtual experiences. Specifically, in this embodiment, each sample... Probability of being sampled The design is as follows:
[0101]
[0102] in, Weights are sampled based on real experience. As virtual experience sampling weights, this example sets , . This represents the number of samples in the real experience pool. This represents the number of samples in the virtual experience pool.
[0103] The agent uses the sampled batch data to update the agent's network parameters (specifically...) Figure 4 (Parameters of the policy network and value network in the text).
[0104] S4. In this embodiment, the decision on whether to enter the parameter fine-tuning stage is made by evaluating the performance of the agent in the virtual environment (specifically, the performance evaluation of the algorithm decision-making for controlling the adjustment of agent parameters in the virtual environment).
[0105] After every 10 rounds of coarse parameter tuning, the current policy is tested in the virtual environment. Run the test trajectory independently for 5 complete runs and calculate its performance. :
[0106]
[0107] in, For current intelligent agent algorithm decision-making, The reward for the current round. The number of rounds for testing the trajectory.
[0108] Then calculate the average performance and standard deviation of these 5 evaluations:
[0109] ,
[0110] When the average performance of the trajectory Meet the minimum standards To ensure that the agent's algorithmic decision-making possesses basic control capabilities and standard deviation. Less than the threshold To ensure that the agent's algorithmic decision-making has stabilized, it indicates that the performance metrics meet the preset interaction mode switching conditions:
[0111] and
[0112] This example sets , .
[0113] If the interaction mode switching conditions are met, the coarse parameter adjustment phase ends. The agent will then be completely detached from the virtual environment, and all subsequent parameter adjustments and interactions will be performed in the real environment. The process then proceeds to the parameter fine-tuning stage, further optimizing the strategy to adapt to the subtle characteristics of the real environment.
[0114] S5. During the parameter fine-tuning phase, the agent is completely synchronized with the real environment. Interaction allows the agent's parameters to be adjusted until the policy converges, and the policy can be further optimized to adapt to the subtle characteristics of the real environment.
[0115] Figure 6 This diagram compares the reward curves of coarse-tuning the trust partition parameters and the conventional parameter adjustment method. Both methods use the same interaction mode switching condition, where solid dots indicate the end of coarse-tuning and entry into the fine-tuning phase in the real environment. The diagram shows that the reward change is smaller during the phase transition in coarse-tuning with trust partitioning, indicating that trust partitioning makes the interaction environment of the policy closer to the real environment during the coarse-tuning phase.
[0116] Example 2
[0117] This embodiment provides a reinforcement learning partition pre-interaction system, including...
[0118] The construction module is used to construct a deviation measurement function to evaluate the model deviation of the virtual environment in the pre-interaction framework relative to the real environment; based on the model deviation, the working space of the real environment is divided into multiple trust intervals, wherein the trust intervals include an absolute trust interval, a relatively trust interval, and a non-trust interval.
[0119] The segmentation module is used to coarsely tune the parameters of the reinforcement learning agent based on the trust interval and using differentiated environmental interaction strategies, so as to differentiate the frequency of the agent's interaction with the real environment within the corresponding trust interval.
[0120] Parameter coarse tuning module: During the parameter coarse tuning stage, it stores the experience data generated by the interaction between the agent and the virtual environment into the virtual experience pool, stores the experience data generated by the interaction between the agent and the real environment into the real experience pool, and samples data from the virtual experience pool and the real experience pool based on preset priority weights to update the network parameters of the agent.
[0121] Judgment module: Used to monitor the performance indicators of the agent during the parameter coarse adjustment stage. When the performance indicators meet the preset interaction mode switching conditions, the agent is controlled to end the parameter coarse adjustment and enter the parameter fine adjustment stage. Otherwise, it continues to be in the parameter coarse adjustment stage until the performance indicators meet the preset interaction mode switching conditions.
[0122] Parameter fine-tuning module: During the parameter fine-tuning stage, the agent fully interacts with the real environment to adjust the agent's parameters until convergence. The agent with converged parameters is used to solve complex sequence decision problems.
[0123] Example 3
[0124] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the reinforcement learning partition pre-interaction method described in Embodiment 1.
[0125] Example 4
[0126] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the reinforcement learning partition pre-interaction method described in Embodiment 1.
[0127] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.
[0128] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0131] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A reinforcement learning partitioning pre-interaction method, characterized in that, include: Step S1: Construct a deviation metric function to evaluate the model deviation of the virtual environment in the pre-interaction framework relative to the real environment; Based on the model bias, the working space of the real environment is divided into multiple trust intervals, wherein the trust intervals include an absolute trust interval, a relatively trust interval, and a non-trust interval. Step S2: Based on the trust interval, a differentiated environment interaction strategy is adopted to coarsely tune the parameters of the reinforcement learning agent, so as to differentiate the frequency of the agent's interaction with the real environment within the corresponding trust interval. Step S3: In the parameter coarse adjustment stage, the experience data generated by the interaction between the agent and the virtual environment is stored in the virtual experience pool, and the experience data generated by the interaction between the agent and the real environment is stored in the real experience pool. Data is sampled from the virtual experience pool and the real experience pool based on preset priority weights to update the network parameters of the agent. Step S4: Monitor the performance indicators of the agent during the parameter coarse adjustment stage. When the performance indicators meet the preset interaction mode switching conditions, control the agent to end the parameter coarse adjustment and enter the parameter fine adjustment stage. Otherwise, continue in the parameter coarse adjustment stage until the performance indicators meet the preset interaction mode switching conditions. Step S5: In the parameter fine-tuning stage, the agent fully interacts with the real environment and adjusts the agent parameters until convergence. The agent with converged parameters is used to solve complex sequence decision problems.
2. The reinforcement learning partitioning pre-interaction method according to claim 1, characterized in that: The deviation measurement function in step S1 Used to reflect operating condition variables The relative deviation between the steady-state gain of the virtual environment and the real environment is expressed as: ； in, For operating condition variables, and These respectively represent the operating conditions Below, the steady-state gain of the virtual environment versus the real environment; It is an absolute value.
3. The reinforcement learning partitioning pre-interaction method according to claim 1, characterized in that: The method for dividing the working space of the real environment into multiple trust intervals based on the model bias in step S1 includes: Based on the aforementioned deviation measurement function The calculation results are compared with the preset deviation threshold to reflect the actual working conditions in the real environment. The space is divided into three trust zones, namely: Absolute Trust Range : ； distrust interval : ； More Confidence Range : ； in, and These are the high and low thresholds for dividing the trust interval, respectively.
4. The reinforcement learning partitioning pre-interaction method according to claim 1, characterized in that: The differentiated environmental interaction strategies in step S2 include: Within an absolute trust range with minimal deviation In this case, the agent interacts only with the virtual environment to maximize efficiency; Within a moderately confident range In this scenario, the intelligent agent primarily interacts with the virtual environment, supplemented by interactions with the real environment. The interaction ratio between the virtual and real environments is 9:1, with a small amount of interaction with the real environment used to achieve online correction of the intelligent agent. In the distrust interval with large deviation In this way, the intelligent agent can fully interact with the real environment, avoiding the misleading effects of model bias.
5. The reinforcement learning partitioning pre-interaction method according to claim 1, characterized in that: In step S3, when sampling data from the virtual experience pool and the real experience pool, the experience data from the real experience pool is assigned a higher sampling weight than the experience data from the virtual experience pool, so as to efficiently utilize the scarce data on the interaction between the agent and the real environment. The sample data... Probability of being sampled Designed as follows: ； in, Weights are sampled based on real experience. For virtual experience sampling weights, / =2, This represents the number of samples in the real experience pool. This represents the number of samples in the virtual experience pool.
6. The reinforcement learning partitioning pre-interaction method according to claim 1, characterized in that: In step S4 The performance index of the agent in the parameter coarse-tuning stage is the performance evaluation of the algorithm decision-making of controlling the agent's parameter adjustment in the virtual environment, and the performance evaluation function is designed as follows: ； in, For current intelligent agent algorithm decision-making, For reward decay coefficient, The reward for the current round. The number of rounds for testing the trajectory; After each training round, the agent's algorithmic decisions are run in a virtual environment. The performance average of the trajectory is constructed based on the performance evaluation function after a complete set trajectory. with standard deviation : ； ； The average performance of the trajectory with standard deviation Design the conditions for switching interaction modes.
7. The reinforcement learning partitioning pre-interaction method according to claim 6, characterized in that: The method for step S4 when the performance index meets the preset interaction mode switching conditions includes: When the average performance of the trajectory Meet the minimum standards To ensure that the agent's algorithmic decision-making possesses basic control capabilities and standard deviation. Less than the threshold To ensure that the agent's algorithmic decision-making has stabilized, it indicates that the performance indicators meet the preset interaction mode switching conditions, expressed as: and .
8. A reinforcement learning partitioning pre-interaction system, characterized in that, include: Module: Used to build a bias metric function to evaluate the model bias of the virtual environment in the pre-interaction framework relative to the real environment; Based on the model bias, the working space of the real environment is divided into multiple trust intervals, wherein the trust intervals include an absolute trust interval, a relatively trust interval, and a non-trust interval. The segmentation module is used to coarsely tune the parameters of the reinforcement learning agent based on the trust interval and using differentiated environmental interaction strategies, so as to differentiate the frequency of the agent's interaction with the real environment within the corresponding trust interval. Parameter coarse tuning module: During the parameter coarse tuning stage, it stores the experience data generated by the interaction between the agent and the virtual environment into the virtual experience pool, stores the experience data generated by the interaction between the agent and the real environment into the real experience pool, and samples data from the virtual experience pool and the real experience pool based on preset priority weights to update the network parameters of the agent. Judgment module: Used to monitor the performance indicators of the agent during the parameter coarse adjustment stage. When the performance indicators meet the preset interaction mode switching conditions, the agent is controlled to end the parameter coarse adjustment and enter the parameter fine adjustment stage. Otherwise, it continues to be in the parameter coarse adjustment stage until the performance indicators meet the preset interaction mode switching conditions. Parameter fine-tuning module: During the parameter fine-tuning stage, the agent fully interacts with the real environment to adjust the agent's parameters until convergence. The agent with converged parameters is used to solve complex sequence decision problems.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the reinforcement learning partition pre-interaction method as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the reinforcement learning partition pre-interaction method as described in any one of claims 1 to 7.