Reinforcement learning training method for simulation agent behavior strategy control

Abstract

The invention provides a reinforcement learning training method for simulation agent behavior strategy control. The reinforcement learning training method for simulation agent behavior strategy control comprises the steps of obtaining a sample according to interaction of an agent in an environment; comparing the sample with a rule of an expert rule set, and calculating a fitting degree; a target expert rule set is obtained according to the fitting value and a standard threshold value, and the target expert rule set is an expert rule set meeting the standard threshold value; calculating the similarity between the current execution action and the expert rule set; rewarding according to the similarity; and circulating the steps to realize reinforcement learning training. Through sample interaction, expert rule set external intervention and a reward incentive mode, the learning training method is finally enhanced, and the technical problem that an intelligent agent reinforcement learning method is lacked at present can be solved.

Description

Technical field:

[0001] The present invention relates to the field of reinforcement learning training, and in particular to a reinforcement learning training method for simulation intelligent body behavior strategy control. Background technology:

[0002] In recent years, with the rapid development of artificial intelligence technology, more and more industries have begun to use intelligent models to replace humans to complete operations, identification, decision-making and other tasks. Compared with humans, intelligent models can execute instructions tirelessly, and their performance after training is relatively stable. Therefore, they are favored by many practitioners in the industry. In task scenarios with complex environmental factors and high task flexibility, reinforcement learning is one of the most effective means of training intelligent agents. Under the constraints of unknown environments and long action sequence task goals, reinforcement learning can interact with the environment in real time and can consider long-term rewards. Compared with other methods, it has special advantages and can better cope with various different working environments.

[0003] However, reinforcement learning still has its flaws. Unlike traditional supervised learning, reinforcement learning does not have any prior knowledge to guide it. The agent needs to explore the environment step by step from scratch, and use this to gain tiny experiences to optimize its own action plan, thereby gradually approaching the goal. Such a training process consumes a lot of time. When the dimension and complexity of the working environment are high, the agent cannot process such a large amount of data, which will cause the time to expand to an unacceptable level, and it is difficult to guarantee the final training effect. In addition, due to the black-box decision-making algorithm of reinforcement learning, the behavior and decision-making of the agent usually do not have any explainable theoretical basis, and it is difficult for humans to understand and optimize.

[0004] Expert rules are the embodiment of human knowledge and experience, representing the code of conduct for humans in a specific environment, at a specific time, or facing a specific situation. The training of traditional reinforcement learning relies entirely on the reward feedback obtained by the agent when exploring the environment, so it is blind. Incorporating some knowledge of expert rules into the reward feedback can make the learning goal of the agent clearer and more purposeful, and can also make the decision of the agent more in line with human expectations. The present invention first uses the agent to explore in the simulation environment by itself, and then compares whether the current simulation environment is similar to the existing expert rules, evaluates the behavior of the agent using similar rules, and integrates this part of the evaluation into the training goal of the agent. In the above manner, the agent can be influenced by the experiential knowledge in the expert rules while exploring by itself, thereby reducing the time waste caused by blind exploration and making the behavior of the agent more in line with human expectations.

[0005] There is an urgent need for a reinforcement learning training method for simulated intelligent agent behavior strategy control, which will help solve the existing technical problem of lack of a reinforcement learning method for intelligent agents. Summary of the invention:

[0006] In one embodiment, the present invention provides a reinforcement learning training method for simulating the behavior strategy control of an intelligent agent, which uses interactive samples, external intervention through an expert rule set, and a reward incentive method to ultimately enable a reinforcement learning training method, which helps to solve the existing technical problem of the lack of a reinforcement learning method for an intelligent agent.

[0007] The reinforcement learning training method for simulating intelligent agent behavior strategy control includes:

[0008] Get a sample based on the agent’s interaction in the environment;

[0009] Compare the sample with the rules of the expert rule set and calculate the degree of fit;

[0010] Obtaining a target expert rule set according to the fitting value and a standard threshold, wherein the target expert rule set is an expert rule set that meets the standard threshold;

[0011] Calculate the similarity between the currently executed action and the expert rule set;

[0012] Awarding according to said similarity;

[0013] The above steps are repeated to implement reinforcement learning training.

[0014] In one embodiment, the rewarding according to the similarity includes:

[0015] Calculate the estimated future return of this round of training based on the sample;

[0016] The estimated return is combined with the exploration reward and the expert reward to obtain the single-step comprehensive reward.

[0017] In one embodiment, the future estimated reward of the current round of training is calculated based on the sample by using the state value matrix and the current exploration to establish, and the discount factor to calculate the future estimated reward after the current action is executed.

[0018] In one embodiment, after obtaining the single-step comprehensive reward step according to the estimated reward combined with the exploration reward and the expert reward, the method further includes:

[0019] Update the policy parameters and state value matrix according to the policy gradient;

[0020] Updates the cumulative discount using the cumulative discount factor.

[0021] In one embodiment, before the above steps are repeated to implement the reinforcement learning training step, the method further includes:

[0022] Update the environment status.

[0023] In one embodiment, before obtaining a sample step according to the interaction of the agent in the environment, the method further includes:

[0024] Determine the initial state, including information about the state space, action space, and expert rule set, to provide input for the algorithm;

[0025] Initialize the strategy optimizer parameters and cumulative discount factor.

[0026] In one embodiment, for the state space S={s 1 ,s 2 ,...,s 270}, action space A = {a 1 ,a 2 ,...,a 25}, expert rule set E = {e 1 ,e 2 ,…,e 85000}, s∈S,a∈A,k∈[1,85000].

[0027] In one embodiment, the initialization data is the learning rate α for the state value matrix w , strategy parameter learning rate α θ , expert rule learning rate a e , the discount factor γ is assigned a set initial value, the strategy parameter θ and the state value matrix w are assigned random values, and the cumulative discount I is assigned to 1.

[0028] In one embodiment, the implicit state value matrix learning rate α in the optimizer is w , strategy parameter learning rate α θ , expert rule learning rate α e They are set to 0.001, 0.001, and 0.01 respectively, the discount factor γ is set to 0.05, random initial values ​​are assigned to the strategy parameter θ and the state value matrix w, and the cumulative discount I is assigned to 1. Description of the drawings:

[0029] Figure 1 The figure is a flowchart of a reinforcement learning training method for simulating intelligent agent behavior strategy control in one embodiment of the present invention. Figure 2 FIG. 4 is a schematic diagram of Mahalanobis distance in another embodiment of the present invention. Specific embodiment:

[0030] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0032] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0033] It should also be understood that, although the present application has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present application that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0034] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0035] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to determine the true intent based on the user's historical operations and to avoid unnecessary or redundant details that make the present application unclear. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but are merely used as the basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0036] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0037] In order to speed up the training and improve the working level of the intelligent agent, a combination of human driving expert rules and agent autonomous exploration learning is selected for the behavioral strategy training of the simulation control agent in a certain intelligent driving virtual verification scenario. First, human driving industry practitioners drive in a virtual environment (the virtual environment contains a total of 270 environmental parameters and 25 action parameters), and collect human action data during this period. After data cleaning, key event selection, rule extraction and other links, a total of 85,000 human driving expert rules are finally obtained, and training work is carried out on this basis.

[0038] Figure 1 A schematic diagram of a reinforcement learning training method for simulating intelligent agent behavior strategy control in one embodiment of the present invention Figure 2FIG. 1 is a schematic diagram of Mahalanobis distance in another embodiment of the present invention. Figure 1 and Figure 2 As shown, in one embodiment, the present invention provides a reinforcement learning training method for simulating intelligent agent behavior strategy control, and the reinforcement learning training method for simulating intelligent agent behavior strategy control includes:

[0039] S101, obtain a sample based on the interaction of the agent in the environment.

[0040] In this step, specific steps of a reinforcement learning training method for simulating intelligent agent behavior strategy control are provided.

[0041] Use the initial strategy π(θ) to interact in the environment. The current state of the interactive environment is When π(θ) takes action A t , the exploration reward is Explore and get a sample:

[0042]

[0043] S102, comparing the sample with the rules of the expert rule set, and calculating the degree of fit.

[0044] For each rule e in the expert rule set E i , respectively calculate its relationship with the current environment state S t The fit between

[0045] The rule fit is calculated as the product of the fit between the environment state and each specific parameter in the expert rule. This is because the fit of different parameters is not a simple linear additive relationship with the overall fit of the rule. When the fit of multiple parameters is high, the correlation between the rule and the current environment will increase rapidly, and the guiding significance for the intelligent agent will be greater. Therefore, the fit of different parameters is a multiplicative relationship, which will help to amplify the influence of rules that are more closely related to the current environment.

[0046] The calculation of the degree of fit between individual parameters is based on statistical principles. Since the state of the simulation environment is a constantly changing fuzzy variable, and the expert rule is a single sampling of the environment, mechanically measuring the similarity between the environment and the rule parameters will inevitably cause statistical errors. Therefore, the present invention regards each parameter in the expert rule as a Gaussian distribution, whose mean μ is the parameter value of the current rule, and the variance σ 2 is the overall variance of the parameter in all expert rules, and its distribution function is F(·), which can be expressed as:

[0047]

[0048] When measuring the fit between the environment and the rule parameters, we first calculate the Gaussian distribution in the current environment parameters. The distribution function under According to the principle of symmetry, the expert rule parameters As the symmetry axis, calculate the distribution function of the Gaussian distribution at the symmetric position According to statistical principles, when a sample falls between two values, it represents Environmental parameters and The expert rule parameters deviate from each other, and vice versa, which means that the two are well matched. Therefore, the function area between the two values ​​of Gaussian distribution can be used to represent the degree of deviation between the two values, and the degree of fit is It can be expressed as:

[0049]

[0050] S103, obtaining a target expert rule set according to the fitting value according to a standard threshold, wherein the target expert rule set is an expert rule set that meets the standard threshold.

[0051] Set a fitness threshold φ 0 , and filter out all the ones whose fit with the current environment state is greater than φ 0 Expert rules form a set of relevant expert rules

[0052] By setting the fitting threshold, users can actively control the minimum correlation between expert rules and the environment, so as to flexibly adjust to different simulation environments and rule sets, avoid using wrong rules to guide the intelligent agent and interfere with normal training.

[0053] S104, calculating the similarity between the currently executed action and the expert rule set. The similarity between the current action and the expert action is calculated based on all the rules related to the current environment, which can give full play to the role of the expert rules, so that they can provide more knowledge and experience information in a single step, and at the same time can reduce the potential errors that may be caused by a single rule sample.

[0054] For the relevant expert rule set E t Each rule in Calculate its action and the currently executed action a respectively t Similarity The weighted average of all rules in terms of fit is calculated to obtain the similarity between the current execution action and the expert rule, which is the expert reward r e .

[0055] The similarity between the current action and the expert rule action is calculated using the Mahalanobis distance. Mahalanobis distance is a statistical metric that is often used in discrete probability distribution data processing tasks. Similar to metrics such as Euclidean distance and Manhattan distance, it is also used as a similarity metric to evaluate data, but it can better deal with problems such as inconsistent scales of various dimensions of high-dimensional linear data and non-independent and identically distributed. Its idea can be seen as an evolution of the Euclidean distance, that is, rotating each variable around the principal component to make each dimension independent of each other; then standardizing the data to make each dimension identically distributed, so that the Mahalanobis distance can be independent of the measurement scale. For a specific multi-parameter vector x, its Mahalanobis distance can be expressed as:

[0056]

[0057] Where μ is the sample mean, Σ -1 is the covariance matrix. In particular, when data points x and y satisfy independent and identical distribution, their Mahalanobis distance is:

[0058]

[0059] The action similarities of multiple rules in the relevant expert rule set are finally integrated into the final action similarity (i.e., expert reward) by weighted averaging, which enables rules that are more closely related to the current environment to have a greater influence on the training of the intelligent agent, making the final reward more instructive.

[0060] According to the similarity, rewards are given, and the weighted average of all rules is calculated on the fit degree to obtain the similarity between the current execution action and the expert rule, which is the expert reward r e ;

[0061] S105, loop the above steps to implement reinforcement learning training.

[0062] The present invention provides a specific implementation of a reinforcement learning training method for simulating intelligent agent behavior strategy control.

[0063] In one embodiment, the rewarding according to the similarity includes:

[0064] Calculate the estimated future return of this round of training based on the sample;

[0065] The calculation of the estimated future return can be expressed as:

[0066]

[0067] where q(·) is the state value function.

[0068] The estimated return is combined with the exploration reward and the expert reward to obtain the single-step comprehensive reward.

[0069] In this embodiment, a specific implementation method for calculating the future estimated return is provided, and then the single-step comprehensive reward is obtained, through the state value matrix w, the current exploration reward and discount factor γ to calculate the estimated future return rlt after the current action is executed.

[0070] The exploration reward, expert reward and estimated return are linearly combined to obtain the single-step comprehensive reward r t .

[0071] In one embodiment, the future estimated reward of the current round of training is calculated based on the sample by using the state value matrix and the current exploration to establish, and the discount factor to calculate the future estimated reward after the current action is executed.

[0072] In one embodiment, after obtaining the single-step comprehensive reward step according to the estimated reward combined with the exploration reward and the expert reward, the method further includes:

[0073] Update the policy parameters and state value matrix according to the policy gradient; Update the policy parameters θ and state value matrix w according to the policy gradient;

[0074] Update the cumulative discount I using the cumulative discount factor.

[0075] The policy parameters θ and the state value matrix w are updated using the baseline policy gradient algorithm based on the self-benefit idea. Specifically, for the policy parameters θ, the update goal is to reduce -Iq(S t ,a t ,w)lnπ(a t |S;θ), so its update method can be expressed as:

[0076]

[0077] For the state value matrix w, its update goal is to reduce [r l t -q(S t ,a t ;w)] 2 , so its update method can be expressed as:

[0078]

[0079] The update method of the cumulative discount I can be expressed as:

[0080] I←γI

[0081] In one embodiment, before the above steps are repeated to implement the reinforcement learning training step, the method further includes:

[0082] Update the environment status, S t =S t+1 .

[0083] In one embodiment, before obtaining a sample step according to the interaction of the agent in the environment, the method further includes:

[0084] Determine the initial state, including information about the state space, action space, and expert rule set, to provide input for the algorithm;

[0085] Initialize the strategy optimizer parameters and cumulative discount factor.

[0086] In one embodiment, for the state space S={s 1 ,s 2 ,…,s 270}, action space A = {a 1 ,a 2 ,…,a 25}, expert rule set E = {e 1 ,e 2 ,…,e 85000}, s∈S,a∈A,k∈[1,85000].

[0087] In one embodiment, the initialization data is the learning rate α for the state value matrix w , strategy parameter learning rate α θ , expert rule learning rate a e , the discount factor γ is assigned a set initial value, the strategy parameter θ and the state value matrix w are assigned random values, and the cumulative discount I is assigned to 1.

[0088] In one embodiment, the implicit state value matrix learning rate α in the optimizer is w , strategy parameter learning rate α θ , expert rule learning rate α e They are set to 0.001, 0.001, and 0.01 respectively, the discount factor γ is set to 0.05, random initial values ​​are assigned to the strategy parameter θ and the state value matrix w, and the cumulative discount I is assigned to 1.

[0089] Below we qualitatively analyze the overall steps, such as an agent. An agent is an agent that can perceive the environment and take actions to achieve specific goals. It can be software, hardware or a system with autonomy, adaptability and interaction. The agent perceives changes in the environment (such as through sensors or data input), makes judgments and decisions based on the knowledge and algorithms it has learned, and then performs actions to affect the environment or achieve predetermined goals. Agents are widely used in the field of artificial intelligence, and are commonly seen in automated systems, robots, virtual assistants, and game characters. The core is the ability to learn autonomously and continuously evolve to better complete tasks and adapt to complex environments.

[0090] Taking the example of an agent learning to climb a ladder, we explain it in combination with the following steps.

[0091] (1) Determine the initial state, including state space, action space, and expert rule set related information to provide input for the algorithm. The present invention involves two most important elements, one is the space element, the other is the behavior element, and an external element, namely the expert rule element. In the example of climbing a ladder, the state space is the space element. For example, one or more vertical planes are arranged in a space. When the intelligent agent moves to the front of the plane, it performs the climbing action, namely the behavior element, and the surrounding behavior space is the action space.

[0092] (2) Initialize the strategy optimizer parameters and cumulative discount factor.

[0093] (3) Use the initial strategy to interact in the environment, perform actions and obtain exploration rewards. In this step, an interaction is performed. The interaction in the environment here refers to the agent making an action, such as moving forward 1 meter, and obtaining some information from the environment after the action, such as the terrain within 10 meters around the agent. This process can be defined as an interaction.

[0094] (4) Compare the fit between the current environment state and the conditional state in the expert rules, select the expert rules whose fit reaches the threshold, and obtain the relevant expert rule set.

[0095] We first compare the scenarios faced during climbing and whether they are compatible with the scenarios required for the expert rules to take effect. For example, not all expert rules are applicable to guiding climbing. We digitize and parameterize the above behaviors to determine whether the data matches the rules in the expert rules, that is, determine which rules are expert rules that interact with this behavior, select these expert rules as the target expert rules, and form a set.

[0096] (5) Compare the similarity between the currently executed action and the actions in the relevant expert rule set, and make an expert evaluation of the action based on the degree of similarity to obtain expert rewards.

[0097] If we find that this behavior has a high degree of similarity with the rules in the target expert rules, then we believe that he is learning in the direction we expect, and we give a corresponding reward, namely the expert reward.

[0098] (6) Calculate the estimated return in this round of training after the action is executed.

[0099] In addition, the estimated reward must also be evaluated. For example, in the climbing behavior, the agent standing up is the first step in climbing. When the agent completes this step, it can be rewarded. In addition to the reward of this step, the subsequent climbing steps are expected to give a future estimated reward to motivate the agent to continue with subsequent actions.

[0100] (7) Combine the exploration reward, expert reward and estimated return to get the comprehensive reward.

[0101] In addition to the estimated rewards mentioned above, it also involves the agent's exploration incentives, that is, the rewards for its exploration of the current step, and expert rules, that is, the rewards after executing the rules after given certain rule constraints. After integrating the several, a comprehensive reward is obtained to complete a kind of guidance.

[0102] (8) Update the policy parameters and state value matrix according to the policy gradient.

[0103] In addition, the weight of the incentive should also be set. For example, after the agent completes the first step of climbing, the incentives for the second and subsequent steps need to be weighted. It is best if the weight incentives for the later steps are smaller.

[0104] (9) Update cumulative discounts.

[0105] (10) Update the environment state and repeat steps (3)-(10) until the training is completed. When the environment changes, the above climbing learning reinforcement process is repeated continuously, and the purpose of reinforcement learning is finally achieved.

[0106] The following is an explanation of how to complete the above method through mathematical algorithms.

[0107] Based on the above initial configuration, in order to obtain the relevant expert rule set, we first need to measure the current environment state S t With each rule e in the expert rule set i The fit between the conditional states contained in Then set a standard fitting threshold φ 0 , filter out all expert rules whose fit with the current environment state is greater than the threshold to form the relevant expert rule set E t ,Right now:

[0108]

[0109] For the environmental status and specific expert rules i , its fitting degree The degree of fit between each specific parameter of the two can be expressed The product of, that is:

[0110]

[0111] in Environmental status parameter With rule condition status parameter The similarity between them is related to the statistical deviation between them and can be expressed as:

[0112]

[0113] Where F(·) is the distribution function of Gaussian distribution, and its mean is variance is the statistical variance of the j-th conditional state parameter in each rule in the expert rule set E, that is:

[0114]

[0115] The similarity between the current execution action and the actions in the relevant expert rule set can be expressed as the weighted average of the similarity with each relevant rule action in the rule fit. and related rule actions The action similarity can be expressed as:

[0116]

[0117] Where D M (·) is the Mahalanobis distance, which can be calculated as:

[0118]

[0119] where Σ -1 is the covariance matrix. According to the action similarity of a single rule, the expert reward It is expressed as:

[0120]

[0121] Comprehensive Rewards t The construction method is to explore the reward Expert Rewards and the estimated return r l t The linear combination of the three parts can be expressed as:

[0122]

[0123] Finally, the overall data is used to explain in detail:

[0124] Use the initial strategy π(θ) to interact in the environment. The current state of the interactive environment is When , the action taken by π(θ) is The exploration reward is This exploration yielded a sample:

[0125]

[0126] For the current environment state S t , and calculate the degree of fit between it and the conditional state of each rule in the expert rule set E respectively.

[0127] For every rule e in E i , S t With e i The degree of fit between is related to the degree of fit between each specific parameter of the two, and the calculation is based on statistical principles. and Its specific parameters and The fit between It can be expressed as:

[0128]

[0129] Where F(·) is the distribution function of a specific Gaussian distribution, and the mean of the distribution is variance is the statistical variance of the j-th conditional state parameter in each rule in the expert rule set E, that is:

[0130]

[0131] S t With e i The degree of fit It can be expressed as the product of all parameter fitness, that is:

[0132]

[0133] Set the fitting threshold φ 0 is 0.6, select all t The degree of fit is not less than φ 0 The rules of composition S t Related expert rule sets E t ,Right now:

[0134]

[0135] For the currently executing action A t , respectively calculate its correlation with the relevant expert rule set E t The similarity between each related rule action in .

[0136] For E t Each relevant rule in A t and The similarity of actions between Through the current action Related rule actions Mahalanobis distance To calculate, that is:

[0137]

[0138] By reverse normalizing the Mahalanobis distance, we can get the action similarity between the two.

[0139]

[0140] Calculate the relevant expert rule set E t After calculating the similarity between each rule and the current action, the fitting degree of the relevant rules is used as the weight, and the weighted average of all action similarities is performed to obtain the overall expert similarity of the current action, that is, the expert reward r e :

[0141]

[0142] According to the state value matrix w, the current exploration reward and discount factor γ to calculate the estimated future return r after the current action is executed l t :

[0143]

[0144] where q(·) is the state value function.

[0145] Getting exploration rewards Expert Rewards and the estimated return r l t After the three rewards, the comprehensive reward r of this learning is constructed through linear combination t ,Right now:

[0146]

[0147] Use the baseline policy gradient algorithm based on self-interest to update the policy parameters θ and the state value matrix w:

[0148]

[0149] Update the cumulative discount I using the cumulative discount factor:

[0150] I←γI

[0151] Update environment status: S t =S t+1 .

[0152] Repeat the above steps until the training is completed.

[0153] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A reinforcement learning training method for simulating intelligent agent behavior strategy control, characterized in that: The reinforcement learning training method for simulating intelligent agent behavior strategy control includes: Get a sample based on the agent’s interaction in the environment; Compare the sample with the rules of the expert rule set and calculate the degree of fit; Obtaining a target expert rule set according to the fitting value and a standard threshold, wherein the target expert rule set is an expert rule set that meets the standard threshold; Calculate the similarity between the currently executed action and the expert rule set; Awarding according to said similarity; The above steps are repeated to implement reinforcement learning training.

2. The reinforcement learning training method for simulating intelligent agent behavior strategy control according to claim 1, characterized in that: The rewarding according to the similarity comprises: Calculate the estimated future return of this round of training based on the sample; The estimated return is combined with the exploration reward and the expert reward to obtain the single-step comprehensive reward.

3. The reinforcement learning training method for simulating intelligent agent behavior strategy control according to claim 2, characterized in that: The future estimated return of this round of training is calculated based on the sample as established by the state value matrix and the current exploration, and the discount factor is used to calculate the future estimated return after the current action is executed.

4. The reinforcement learning training method for simulating intelligent agent behavior strategy control according to claim 3, characterized in that: After obtaining the single-step comprehensive reward step according to the estimated reward combined with the exploration reward and the expert reward, the method further includes: Update the policy parameters and state value matrix according to the policy gradient; Updates the cumulative discount using the cumulative discount factor.

5. The reinforcement learning training method for simulating intelligent agent behavior strategy control according to claim 4, characterized in that: Before the above steps are cycled to implement the reinforcement learning training step, the method further includes: Update the environment status.

6. The reinforcement learning training method for simulating intelligent agent behavior strategy control according to claim 5, characterized in that: Before obtaining a sample step according to the interaction of the agent in the environment, the method further comprises: Determine the initial state, including information about the state space, action space, and expert rule set, to provide input for the algorithm; Initialize the strategy optimizer parameters and cumulative discount factor.

7. The reinforcement learning training method for simulating intelligent agent behavior strategy control according to claim 6, characterized in that: For the state space S = {s1,s2,...,s 270 }, action space A = {a1, a2, ..., a 25 }, expert rule set E = {e1, e2, …, e 85000 }, 8. The reinforcement learning training method for simulating intelligent agent behavior strategy control according to claim 7, characterized in that: Initialize the data to the state value matrix learning rate α w , strategy parameter learning rate α θ , expert rule learning rate a e , the discount factor γ is assigned a set initial value, the strategy parameter θ and the state value matrix w are assigned random values, and the cumulative discount I is assigned to 1.

9. The reinforcement learning training method for simulating intelligent agent behavior strategy control according to claim 8, characterized in that: The implicit state value matrix learning rate α in the optimizer w , strategy parameter learning rate α θ , expert rule learning rate α e They are set to 0.001, 0.001, and 0.01 respectively, the discount factor γ is set to 0.05, random initial values ​​are assigned to the strategy parameter θ and the state value matrix w, and the cumulative discount I is assigned to 1.

Images