Deep reinforcement learning decision optimization method and device, computer equipment and medium

Abstract

The invention relates to the technical field of deep reinforcement learning, and discloses a deep reinforcement learning decision optimization method and device, computer equipment and a medium, and the method comprises the steps: detecting whether a candidate network cluster is in a strategy evaluation state or not; if in the strategy evaluation state, obtaining an evaluation result corresponding to each training node output by the strategy evaluation node; training nodes in the candidate network cluster are optimized according to the evaluation result until the candidate network cluster meets a preset termination condition, and a target action decision network is obtained; and determining a target action node in the target action decision network, and taking an action strategy corresponding to the target action node as a target action strategy. According to the method, the problems that a deep reinforcement learning decision optimization scheme is low in training efficiency, local optimum and experience forgetting are likely to occur, the intelligent agent correlation utilization of a multi-node scheme is insufficient, and a single-node scheme is not suitable for multi-node training are solved.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning, and in particular to an optimization method, device, computer equipment and medium for deep reinforcement learning decision making. Background Art

[0002] With the development of artificial intelligence, deep reinforcement learning integrates the capabilities of deep learning and reinforcement learning, and has achieved remarkable results in many fields, such as robot control, Go, and arcade games. Reinforcement learning is used to solve the problem of action strategy learning in the interaction between intelligent agents and the environment. The action node-comment node framework is the mainstream, and the proximal strategy optimization method is one of them. The intelligent agent selects actions based on the state of the environment. The environment has state transfer and reward feedback. The action strategy is mostly a neural network mapping. The reward and cumulative reward are important for decision optimization. The experience pool stores interaction data for training.

[0003] However, deep reinforcement learning decision optimization schemes have many problems in practical applications: when reinforcement learning methods solve action decision problems, the number of sampling times and calculation time are large, the training efficiency is low, and it is difficult to meet the needs of actual scenarios. It is also easy to fall into local optimality and experience forgetting problems. Insufficient stability will lead to strategy collapse. In addition, the actual task scenarios are complex, and the reward function and hyperparameter adjustment work increase the requirements for stability and training efficiency. Although the multi-node distributed decision optimization scheme can increase the diversity of experience, improve the ability to jump out of the local optimality, and increase the training speed, it does not fully utilize the correlation between multiple networks between agents. The optimization scheme of the single-node reinforcement learning method mainly focuses on improving the exploration and utilization capabilities and correcting the estimation bias. It is a single-agent training mode and cannot be directly extended to multi-node training. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide an optimization method, device, computer equipment and medium for deep reinforcement learning decision-making, so as to solve the problems of low training efficiency, easy occurrence of local optimality and experience forgetting in deep reinforcement learning decision-making optimization schemes, insufficient utilization of the correlation of intelligent agents in multi-node schemes, and unsuitability of single-node schemes for multi-node training.

[0005] In a first aspect, an embodiment of the present invention provides a method for optimizing deep reinforcement learning decisions, the method comprising:

[0006] Detecting whether the candidate network cluster is in a policy evaluation state, wherein the candidate network cluster includes multiple groups of training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a public experience pool, and each group of training nodes includes n training nodes, where n≥1;

[0007] If it is in the strategy evaluation state, obtaining the evaluation results corresponding to each of the training nodes output by the strategy evaluation node;

[0008] Optimizing the training nodes in the candidate network cluster according to the evaluation result until the candidate network cluster meets a preset termination condition to obtain a target action decision network;

[0009] A target action node is determined in the target action decision network, and an action strategy corresponding to the target action node is used as a target action strategy.

[0010] Furthermore, before detecting whether the candidate network cluster is in a policy evaluation state, the method further includes:

[0011] Initializing multiple node components of training nodes in the original network cluster, wherein the node components include original action nodes, original comment nodes, a preset environment, and a private experience pool;

[0012] Using the original action node of each training node to interact with the corresponding preset environment to obtain round experience, and storing the round experience in the corresponding private experience pool and public experience pool;

[0013] The original network cluster is optimized based on the private experience pool and the public experience pool to obtain a candidate network cluster.

[0014] Furthermore, the optimizing the original network cluster based on the private experience pool and the public experience pool to obtain a candidate network cluster includes:

[0015] Collecting a first experience sample in a private experience pool of the training node;

[0016] Using the first experience sample to train the original comment node until a training condition is met to obtain a target comment node;

[0017] Collecting a second experience sample from the public experience pool, and merging the first experience sample with the second experience sample to obtain a merged experience sample;

[0018] Using the combined experience sample to train the original action node to obtain an action node;

[0019] The trained inter-group comment nodes are obtained, and the original network cluster is adjusted based on the action nodes, the target comment nodes, and the inter-group comment nodes to obtain a candidate network cluster.

[0020] Furthermore, obtaining the trained inter-group comment nodes includes:

[0021] Identify the grouping of training nodes in the original network cluster;

[0022] Determine a plurality of node groups according to the grouping situation, and determine an original inter-group comment node that matches each of the node groups according to a preset association relationship;

[0023] Collecting a third experience sample in a private experience pool within the node group;

[0024] The third experience sample is used to train the original inter-group comment nodes that match the node grouping to obtain trained inter-group comment nodes.

[0025] Furthermore, the detecting whether the candidate network cluster is in a policy evaluation state includes:

[0026] Obtaining the current training round number of the candidate network cluster;

[0027] Determine whether the current number of training rounds reaches a preset number of single-batch training rounds;

[0028] If the number of single-batch training rounds is reached, the candidate network cluster is controlled to enter a strategy evaluation state; if the number of single-batch training rounds is not reached, the training operation is continued until the number of single-batch training rounds is reached.

[0029] Furthermore, before obtaining the evaluation results corresponding to the training nodes output by the strategy evaluation node, the method further includes:

[0030] Utilize the action nodes in the candidate network cluster to interact with the corresponding preset environment to obtain environment feedback data;

[0031] Analyze the action strategy output by the action node in the training node according to the environmental feedback data to obtain an action score;

[0032] The action score is transmitted to a policy evaluation node in the candidate network cluster.

[0033] Further, obtaining the evaluation results corresponding to each of the training nodes output by the strategy evaluation node includes:

[0034] receiving an action score from the training node;

[0035] Utilizing the inter-group comment node associated with the training node, analyzing the action strategy output by the action node in the training node to obtain an inter-group action score;

[0036] The training node is evaluated based on the action score and the inter-group action score to obtain a corresponding evaluation result.

[0037] Further, the optimizing the training nodes in the candidate network cluster according to the evaluation result until the candidate network cluster meets a preset termination condition to obtain a target action decision network includes:

[0038] Determine the dominant nodes and the inferior nodes in the candidate network cluster according to the evaluation result;

[0039] Based on the first network parameter of the dominant node, performing a replacement operation on the second network parameter of the disadvantaged node;

[0040] Monitoring the training status of each batch of each training node in the candidate network cluster;

[0041] It is determined whether the candidate network cluster meets a preset termination condition according to the training situation, and when the preset termination condition is met, the candidate network cluster is used as the target action decision network.

[0042] In a second aspect, an embodiment of the present invention provides an optimization device for deep reinforcement learning decision making, the device comprising:

[0043] A detection module, used to detect whether the candidate network cluster is in a policy evaluation state, wherein the candidate network cluster includes multiple groups of training nodes, multiple inter-group comment nodes, policy evaluation nodes and a public experience pool, each group of training nodes includes n training nodes, n≥1;

[0044] An acquisition module, used for acquiring the evaluation results corresponding to each of the training nodes output by the strategy evaluation node if in the strategy evaluation state;

[0045] An optimization module, used to optimize the training nodes in the candidate network cluster according to the evaluation result, until the candidate network cluster meets a preset termination condition, thereby obtaining a target action decision network;

[0046] The determination module is used to determine the target action node in the target action decision network and use the action strategy corresponding to the target action node as the target action strategy.

[0047] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0049] The method provided in the embodiment of the present application has the following beneficial effects:

[0050] The method provided in the embodiment of the present application grasps the training progress by detecting the candidate network cluster strategy evaluation status, provides a decision-making basis for the optimization process, ensures that the training is carried out reasonably, and avoids waste of resources and erroneous operations. Obtaining the evaluation results can understand the performance of the training nodes, provide guidance for the optimization operation, and deal with different performance nodes in a targeted manner to improve the network cluster performance, training efficiency and quality. Optimizing the training nodes based on the evaluation results can propagate the characteristics of the advantageous nodes and improve the overall performance. Continuously optimizing until the preset termination conditions are met can ensure the quality and stability of the target action decision network, reduce local optimal solutions, and improve the accuracy and reliability of deep reinforcement learning decisions.

[0051] The method provided in the embodiment of the present application starts the training of the network cluster from a stable and reasonable initial state, providing a basis for interaction and optimization. Initializing the node components can avoid training problems caused by the initial settings. The action node interacts with the environment to obtain and store experience, increase the source and diversity of experience, and the private and public experience pool storage methods are the basis for training at different levels, which is conducive to making full use of experience data, improving training effects and reducing acquisition costs. The dual experience pool optimization method uses experience from different sources to promote the optimization of the original network cluster. Combining the two experiences can mine more information, improve the quality of training, and ensure the acquisition of high-quality candidate network clusters.

[0052] The method provided in the embodiment of the present application samples from a private experience pool and can utilize the personalized experience of training nodes to improve the pertinence of single-node training and tap into potential advantages. Using specific experience samples to train comment nodes can improve the accuracy of their action value assessments, thereby guiding action nodes to select better strategies and improving the decision-making capabilities of network clusters. Collecting and merging public and private experience samples can comprehensively consider cluster and individual experience, enrich training data, make action node training more comprehensive, and improve adaptability and network cluster performance. Using a variety of trained nodes to adjust the network cluster can give full play to the advantages of each node, make the structure and parameters more reasonable, and improve synergy and performance in deep reinforcement learning decisions.

[0053] The method provided in the embodiment of the present application can clarify the characteristics of the relationship between groups by identifying the groups, provide a basis for training the comment nodes between groups, and ensure pertinence and effectiveness. According to the grouping, the matching comment nodes between groups can be closely associated with the corresponding group training nodes, so as to better evaluate the action value and improve the synergy of inter-group training. Sampling the comment nodes between the training groups from the private experience pool within the group can focus on the common characteristics within the group, improve the accuracy of the evaluation, promote the collaborative training of the nodes within the group, and improve the performance of the network cluster.

[0054] The method provided in the embodiment of the present application obtains return data through the interaction between the action node and the environment, provides a basis for analysis and evaluation, and helps to adjust the training direction. The scoring of action strategies can quantify the advantages and disadvantages, facilitate comparative evaluation, intuitively reflect the performance of action nodes, and provide reference indicators for comprehensive evaluation and optimization. Transmitting action scores to the strategy evaluation node can enable it to obtain comprehensive information, ensure that the evaluation results are accurate and comprehensive, and provide a basis for optimizing training nodes.

[0055] The method provided in the embodiment of the present application ensures that the strategy evaluation node has key information of each node by receiving action scores, which affects the accuracy of performance judgment. By obtaining inter-group action scores through inter-group comment nodes, action nodes can be evaluated from different angles, making the evaluation more comprehensive and objective. The comprehensive action score and inter-group action score evaluation training node can comprehensively consider the performance at the individual and cluster levels, provide targeted guidance for optimization operations, and improve the optimization effect.

[0056] The method provided in the embodiment of the present application provides a clear goal for network cluster optimization by identifying advantageous and disadvantageous nodes. The parameter replacement operation can transfer the characteristics of advantageous nodes to improve the performance of disadvantageous nodes, avoid falling into local optimal solutions, and improve overall performance. Monitoring the training situation can timely discover abnormal problems, ensure stability and reliability, and adjust the optimization strategy accordingly to ensure that the training develops as expected. By determining the target action decision network by judging the termination conditions, the quality of network performance can be guaranteed, and over- or under-training can be avoided, so that it can effectively perform decision-making tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 is a flowchart of a method for optimizing deep reinforcement learning decision making according to an embodiment of the present invention;

[0059] Figure 2 is a schematic diagram of a network architecture of a candidate network cluster according to an embodiment of the present invention;

[0060] Figure 3 is a schematic diagram of a network architecture of a training node in a candidate network cluster according to an embodiment of the present invention;

[0061] Figure 4 is a schematic diagram of the working principle of action nodes in a multi-joint robot control scenario based on a physical simulation engine according to an embodiment of the present invention;

[0062] Figure 5is a schematic diagram of the working principle of a policy evaluation node according to an embodiment of the present invention;

[0063] Figure 6 is a schematic diagram of the working principle of an inter-group comment node according to an embodiment of the present invention;

[0064] Figure 7 is a schematic diagram of collecting experience in a public experience pool according to an embodiment of the present invention;

[0065] Figure 8 is a flow chart of another method for optimizing deep reinforcement learning decision making according to an embodiment of the present invention;

[0066] Fig. 9 is a structural block diagram of a device for optimizing deep reinforcement learning decision making according to an embodiment of the present invention;

[0067] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0069] According to an embodiment of the present invention, an optimization method, apparatus, computer device and medium for deep reinforcement learning decision-making are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0070] In this embodiment, a method for optimizing deep reinforcement learning decision making is provided. Figure 1 is a flowchart of a method for optimizing deep reinforcement learning decision making according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0071] Step S11, detecting whether the candidate network cluster is in a policy evaluation state, wherein the candidate network cluster includes multiple groups of training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a public experience pool, and each group of training nodes includes n training nodes, n≥1.

[0072] It should be noted that if Figure 2As shown in Figure 1, the network architecture of the candidate network cluster includes: strategy evaluation nodes, public experience pool, training nodes (divided into m node groups, each with n training nodes) and corresponding m inter-group comment nodes. The strategy evaluation state refers to the stage where the network cluster enters the evaluation of the action strategy of each training node. At this stage, the quality of the strategy can be judged based on the performance of the previous training node in the environment, and then decide whether to copy or remove the node to optimize the performance of the entire network cluster.

[0073] In the embodiment of the present application, before obtaining the evaluation results corresponding to each training node output by the strategy evaluation node, the method further includes the following steps A1-A3:

[0074] Step A1, using the action nodes in the candidate network cluster to interact with the corresponding preset environment to obtain environment feedback data.

[0075] It should be noted that if Figure 3 As shown in Figure 1, the network architecture of each training node in the candidate network cluster includes an action node, a preset environment, a private experience pool, and a target comment node. The main function of the action node is to output action selection through its built-in action network according to the current environment state. For example, Figure 4 As shown in the figure, in the multi-joint robot control scenario based on the physical simulation engine MuJoCo, the action node will determine the robot's next action based on the robot's current position, angle, speed and other environmental status information, such as the rotation angle of the joints, the direction of movement, etc. The preset environment is the environment set for the decision optimization task. For a multi-joint robot, it includes various terrains, obstacles, etc., and the state will change with the action performed by the action node.

[0076] Specifically, when an action node performs an action in a preset environment, the environment will give feedback based on the execution of the action. These feedback data are the environment reward data. For example, when the robot is walking, if it successfully avoids an obstacle, the environment will give a positive reward value (positive experience); if it hits an obstacle, it will give a negative reward value (negative experience). These reward data reflect the effect of the action node action strategy in the current environment.

[0077] Step A2: Analyze the action strategy output by the action node in the training node according to the environmental feedback data to obtain the action score.

[0078] Specifically, environmental feedback data is an important basis for analyzing action strategies. By analyzing these data, we can understand the performance of the action strategies adopted by the action nodes in the preset environment. The action score is a quantitative evaluation of the action strategies output by the action nodes. For example, if the environmental feedback data shows that the robot can complete the task efficiently within a period of time (such as reaching the target position quickly and avoiding all obstacles), the corresponding action strategy will get a higher score; conversely, if the robot frequently makes mistakes (such as colliding with obstacles many times or failing to reach the target for a long time), the action strategy will get a lower score. This score can intuitively reflect the pros and cons of the action strategy.

[0079] Step A3: Transmit the action score to the policy evaluation node in the candidate network cluster.

[0080] Specifically, Figure 5 As shown in the figure, the strategy evaluation node is a unit that evaluates the action strategies of all actors. In each training round, it obtains evaluation indicators such as the cumulative reward of the round, the cumulative number of steps of the round or the difference between strategies, and the action score corresponding to each training node, so as to decide the copy or removal of the training node based on the strategy copy and removal mechanism. The action score is transmitted to the strategy evaluation node so that it can comprehensively evaluate the action strategy of the candidate network cluster by integrating the performance of each action node, so as to make a more accurate decision by combining other performance information.

[0081] In the embodiment of the present application, step S11 includes the following steps B1-B3:

[0082] Step B1, obtaining the current training round number of the candidate network cluster.

[0083] Specifically, in the process of reinforcement learning training, training is carried out in rounds. In each round, the actor interacts with the environment and updates its own parameters and experience pool according to the results. The number of training rounds is a counting method for the training process, which is used to determine the stage and progress. Specifically, the number of completed training rounds can be recorded by a counter. When a round of training is completed (including the interaction between the actor and the environment, the critic training, the actor training based on new experience, etc.), the counter is increased by 1. The current number of training rounds can be obtained by querying the counter value.

[0084] Step B2, determining whether the current number of training rounds reaches the preset number of single batch training rounds.

[0085] Specifically, when establishing a candidate network cluster, the number of single-batch training rounds will be preset, and its value can be determined according to factors such as training tasks and model complexity. For example, a simple multi-joint robot action strategy training task can be set to 50, and a task in a complex environment can be set to 200. It is the threshold for determining the timing of strategy evaluation. Judging whether the current number of training rounds reaches the preset value is to determine whether to enter the strategy evaluation stage. If the threshold is not reached, it means that the training is accumulating experience and optimizing actors and critics. If the threshold is reached, it is necessary to evaluate the training situation to consider whether to adjust the training nodes.

[0086] Step B3: If the number of single-batch training rounds is reached, the candidate network cluster is controlled to enter the strategy evaluation state; if the number of single-batch training rounds is not reached, the training operation is continued until the number of single-batch training rounds is reached.

[0087] Specifically, when the current number of training rounds reaches the preset number of single-batch training rounds, the candidate network cluster enters the policy evaluation state. At this time, the policy evaluation node will comprehensively evaluate the action strategy of each training node and integrate various information (such as actor action performance, environmental feedback data, etc.) to prepare for subsequent node optimization. If the preset value is not reached, the training continues, the actor interacts with the environment to sample and obtain new round experience and store it, the critic is trained based on the new experience data, and the actor uses the new experience and the critic's value estimation to continuously self-train to optimize the action strategy until the single-batch training round number is reached before entering the policy evaluation state.

[0088] Step S12: If it is in the strategy evaluation state, the evaluation results corresponding to each training node output by the strategy evaluation node are obtained.

[0089] In the embodiment of the present application, step S12 includes the following steps C1-C3:

[0090] Step C1, receiving action scores from training nodes.

[0091] Specifically, the strategy evaluation node can receive the action scores transmitted from each training node, ensuring that each training node can accurately send its own action score to the strategy evaluation node for subsequent comprehensive evaluation.

[0092] Step C2, using the inter-group comment node associated with the training node, analyzes the action strategy output by the action node in the training node to obtain the inter-group action score.

[0093] Specifically, the inter-group comment node is an evaluation unit trained based on all the experience in the training node group. Unlike the comment node in a single training node, it can evaluate the action strategy from the macro perspective of the experience of all nodes in the group. In the strategy evaluation state, for each training node, the action strategy output by the action node is further analyzed using its associated inter-group comment node, and the inter-group action score is obtained by comprehensively evaluating the relevant experience of other nodes in the group and the performance of the action node itself, which can supplement the shortcomings of the action score obtained from the experience of a single node and provide more comprehensive and multi-dimensional evaluation information.

[0094] As an example, Figure 6 As shown in the figure, multiple private experience pools (private experience pool 1, private experience pool 2...private experience pool n) sample the corresponding comment nodes (critic 1, critic 2...critic n), and these comment nodes sample the inter-group comment nodes (inter-group critic 1). Inter-group critic 1 will comprehensively consider the information obtained from each critic and analyze the action strategy output by the action node in each training node. It can use the information sampled from multiple private experience pools to comprehensively evaluate the action strategy from a macro perspective, and then derive the inter-group action value (inter-group action score). This scoring method based on the experience of multiple nodes within the group can avoid the limitations of relying solely on the experience of a single node for evaluation, thereby providing a richer and more accurate reference for strategy evaluation.

[0095] Step C3, evaluating the training nodes based on the action scores and the inter-group action scores to obtain corresponding evaluation results.

[0096] Specifically, after obtaining the training node's own action score and the inter-group action score, the strategy evaluation node comprehensively considers the two, such as using weighted average (the weight is determined according to the actual situation) to combine the two scores into a comprehensive evaluation index according to the rules. This result can accurately reflect the performance of the training node in the candidate network cluster, and is an important basis for the training node optimization operation (such as determining the good and bad nodes), which helps to improve the performance of the network cluster and obtain a better action decision network.

[0097] Step S13, optimizing the training nodes in the candidate network cluster according to the evaluation results until the candidate network cluster meets the preset termination condition, thereby obtaining the target action decision network.

[0098] In the embodiment of the present application, step S13 includes the following steps D1-D4:

[0099] Step D1, determining the dominant nodes and the inferior nodes in the candidate network cluster according to the evaluation results.

[0100] Specifically, after a comprehensive evaluation of each training node, the corresponding evaluation results are obtained, which can comprehensively reflect the performance of each node in the current training stage and are an important basis for determining the superior and inferior nodes. The determination method is usually to set a threshold or use a sorting method. For example, a node with an evaluation result higher than a certain threshold can be determined as an advantageous node, which performs well in terms of interacting with the environment and executing action strategies, and has advantages such as high action efficiency and good strategy adaptability; a node with an evaluation result lower than the threshold is an inferior node, which performs relatively poorly in all aspects and needs further optimization and adjustment.

[0101] Step D2: Based on the first network parameters of the dominant node, a replacement operation is performed on the second network parameters of the inferior node.

[0102] Specifically, in the training nodes, each node has network parameters that determine its behavior and performance. For example, the action strategy network parameters of the action node (actor) affect the action selection, and the action value estimation network parameters of the comment node (critic) affect the accuracy of value assessment. The first network parameter is for the dominant node, and the second network parameter is for the inferior node. The replacement operation is to introduce the good network parameters of the dominant node into the inferior node to improve its performance and optimize the cluster performance. The replacement operation is to copy the first network parameters of the dominant node, add a small amount of noise (to avoid overfitting), and then replace the second network parameters of the inferior node, so that the inferior node inherits the characteristics of the dominant node, while having room for change to adapt to different training situations.

[0103] Step D3, monitoring the training status of each batch of each training node in the candidate network cluster.

[0104] After optimizing the training nodes, it is necessary to continuously monitor the training status of each subsequent batch of nodes. The monitoring content includes changes in the experience data of interaction with the environment (such as reward value, state transfer, etc.), network parameter updates after training based on new experience data, changes in action strategy performance (such as accuracy, efficiency), etc. Through monitoring, we can understand the effect of the optimization operation. If it is found that the inferior node after parameter replacement performs poorly, it is necessary to further analyze the reasons for targeted improvement.

[0105] Step D4, judging whether the candidate network cluster meets the preset termination condition according to the training situation, and when the preset termination condition is met, taking the candidate network cluster as the target action decision network.

[0106] Specifically, the termination conditions will be preset before training, which are related to factors such as training node performance, number of training rounds, and number of batches. For example, it can be set that the performance differences of all training node action networks are small and the changes in each batch of training are small, or when a certain number of training rounds and batches are reached, the conditions are met. After monitoring the training status of each batch of each training node, it is compared with the preset termination conditions. If it is satisfied, it means that the candidate network cluster has been fully trained and optimized, and it can be used as the target action decision network. The network contains optimized nodes, the overall performance is ideal, and the output action strategy is relatively optimal.

[0107] Step S14, determining a target action node in the target action decision network, and using the action strategy corresponding to the target action node as the target action strategy.

[0108] In the embodiment of the present application, in the optimized target action decision network, the method of determining the target action node includes but is not limited to: selecting according to the performance of each action node in the previous training, such as high reward value obtained by interacting with the environment, high efficiency of action strategy execution, good action accuracy and other outstanding performances; determining according to specific task requirements, selecting the fastest response for tasks that require rapid response, and selecting the action node with the best execution accuracy for tasks that focus on accuracy. After determining the target action node, its corresponding action strategy is identified as the target action strategy. After the strategy is optimized and screened throughout the process, it is expected to achieve the best decision-making effect under the current task scenario and network architecture, and help complete related decision optimization tasks, such as allowing multi-joint robots to complete walking, grasping and other action tasks more efficiently and accurately.

[0109] In the embodiment of the present application, before step S11, the following steps S21-S23 are also included:

[0110] Step S21, initializing multiple node components of the training nodes in the original network cluster, wherein the node components include original action nodes, original comment nodes, preset environments, and private experience pools.

[0111] In the embodiment of the present application, the original network cluster before the training process starts is the basic framework for subsequent training and optimization operations, including multiple training nodes and related components (original action nodes, original comment nodes, preset environment, private experience pool), and each component is in the original preset state. The original action node outputs action selection according to the environmental state, such as determining the robot action in the multi-joint robot control scene; the original comment node evaluates the action and environmental state to provide a basis for training optimization; the preset environment is set according to the task and will change due to the action of the action node; the private experience pool is used to store the private experience data accumulated by the training node itself for subsequent component training optimization.

[0112] Specifically, initialization is to give the node component an initial state or parameter value so that it can operate normally in subsequent training. For example, set the initial action strategy network parameters of the original action node, determine the initial action value estimation network parameters of the original comment node, and set the initial state conditions of the preset environment. For example, the parameters of the original action node are given initial weight values ​​based on prior knowledge to preliminarily select actions, and the parameters of the original comment node are given initial values ​​to preliminarily evaluate actions and states.

[0113] Step S22, using the original action node of each training node to interact with the corresponding preset environment to obtain round experience, and store the round experience in the corresponding private experience pool and public experience pool.

[0114] In the embodiment of the present application, the original action node of each training node interacts with the corresponding preset environment based on the initial action strategy network parameters, outputs action selection according to the environment state, and the preset environment changes accordingly and gives a reward value, thereby generating round experience, including information such as environment state, action, and reward value. After obtaining the round experience, it needs to be stored for subsequent training optimization, both in the private experience pool of the corresponding training node for its own training optimization, and in the public experience pool for other nodes to use, so as to improve the efficiency of network cluster training, realize experience sharing and collaborative training, such as other training node action node training can be combined with public and private experience for more comprehensive optimization.

[0115] Step S23, optimizing the original network cluster based on the private experience pool and the public experience pool to obtain a candidate network cluster.

[0116] In the embodiment of the present application, step S23 includes the following steps E1-E5:

[0117] Step E1, collecting a first experience sample in a private experience pool of a training node.

[0118] Specifically, the private experience pool stores the private experience obtained by the interaction between the training node and the environment. It is its unique data and can reflect the action execution of the node in a specific environment. The first experience sample is collected from the pool for the purpose of optimizing the training of related components. The collection method can be based on rules or random selection to obtain representative data as samples, providing a data basis for the training of the original comment node.

[0119] Step E2, using the first experience sample to train the original comment node until the training condition is met to obtain the target comment node.

[0120] Specifically, the original comment node is mainly used to evaluate the action of the action node and the current environmental state, give an estimate of the action value and calculate the action gradient, and provide a basis for judging the quality of the action for training. It is trained with the first experience sample to enable it to more accurately give reasonable value estimates for different states and action combinations, and improve the ability to optimize action strategies. During training, the sample is input into the original comment node, which adjusts its internal parameters based on the sample information combined with its own network structure and algorithm to make the output value estimate closer to the actual situation. This continues until the preset training conditions are met, such as the error between the estimated value and the true value is less than a specific threshold or a certain number of iterations are completed. Once the training is completed, the target comment node is obtained, and its evaluation of the action value will be more accurate.

[0121] Step E3: collect a second experience sample from the public experience pool, and merge the first experience sample with the second experience sample to obtain a merged experience sample.

[0122] Specifically, the public experience pool is a node shared resource pool that stores all training node environment interaction experience data and can provide diverse experience for training nodes to improve training efficiency. Figure 7 As shown, when collecting the second experience sample of the public experience pool, first sample from the pool and input the data into the experience filter of the training node. The filter is filtered according to the importance constraint related to the action selection probability. The filtered experience is used for action node (actor) training. The action node is combined with the action value of the comment node (critic). Based on the filtered experience, the action strategy gradient is optimized to improve the ability, and then it is merged with the first experience sample of the private experience pool into a merged experience sample. The comprehensive use of public and private experience provides a richer data basis for the original action node training to optimize the strategy.

[0123] Step E4, using the combined experience samples to train the original action nodes to obtain action nodes.

[0124] Specifically, the original action node outputs action selection through the built-in network according to the current environment state. The purpose of training is to enable it to output appropriate actions more accurately and efficiently for different environments, and improve the performance of decision-making optimization tasks, such as helping multi-joint robots to complete tasks accurately. During training, the merged experience samples are input, and it combines the environment, action, feedback and other information in the samples with its own structure and algorithm to adjust internal parameters. After learning and adaptation, it can output better action selection to complete the task after completing training.

[0125] Step E5, obtain the trained inter-group comment nodes, adjust the original network cluster based on the action nodes, target comment nodes and inter-group comment nodes, and obtain the candidate network cluster.

[0126] In the embodiments of the present application, the trained action nodes have optimized action selection capabilities, providing a better foundation for the network cluster to perform action tasks; the target comment node evaluation is more accurate, and can provide reliable action value estimation and gradient calculation; the inter-group comment node can evaluate the action value in multiple dimensions. When adjusting the original network cluster, the three characteristics can be combined to adjust the relevant connection weights, parameters, etc. of the action nodes according to their ability performance; the evaluation result feedback of the target comment node is used to adjust the correlation relationship between nodes and the degree of utilization of the evaluation information by other nodes; the inter-group comment node adjusts the global parameters according to its value judgment. By comprehensively using their characteristics, the original network cluster is adjusted in many aspects to achieve a better state to obtain a candidate network cluster, laying the foundation for subsequent training and decision optimization.

[0127] In the embodiment of the present application, obtaining the trained inter-group comment node includes the following steps E51-E54:

[0128] Step E51, identifying the grouping of training nodes in the original network cluster.

[0129] Specifically, the original network cluster contains multiple training nodes. To optimize the inter-group comment nodes, it is necessary to obtain the grouping of the training nodes. The groups can be divided according to specific rules or according to the actual scene requirements. For example, in the multi-joint robot control scenario, the nodes can be grouped according to the joint control requirements or task modules to facilitate the management and optimization of each group of nodes. Accurately identifying the grouping situation can lay the foundation for determining the matching inter-group comment nodes and subsequent targeted training and adjustment operations.

[0130] Step E52, determining multiple node groups according to the grouping situation, and determining the original inter-group comment nodes that match each node group according to the preset association relationship.

[0131] Specifically, after clarifying the grouping of training nodes, the original inter-group comment nodes that match each group can be determined based on the preset association relationship. This association relationship is set at the beginning of the network cluster construction, taking into account the adaptability of different group training nodes and inter-group comment nodes in terms of functions, data characteristics, etc. For example, the group that controls the robot arm joint action will match the original inter-group comment node that is good at evaluating the value of arm joint actions, so that the action value of the nodes in the group can be evaluated and optimized more accurately in the future.

[0132] Step E53: Collect a third experience sample from the private experience pool within the node group.

[0133] Specifically, each determined node group has a corresponding private experience pool, from which third experience samples are collected, with the purpose of using these samples to train the original inter-group comment nodes that match it. The collection can be carried out by rules or random selection, etc., to obtain representative data as samples, providing a targeted data basis for the original inter-group comment node training, so that it can better adapt to the characteristics and needs of the corresponding node grouping.

[0134] Step E54, using the third experience sample to train the original inter-group comment nodes that match the node groupings, to obtain trained inter-group comment nodes.

[0135] Specifically, the third experience sample collected from the private experience pool within the node group is input into the matching original inter-group comment node. The node continuously adjusts its internal parameters based on the state and action information in the sample, combined with its own network structure and algorithm, to output an action value estimate. This process continues until the preset conditions are met, such as the error between the estimated value and the true value calculated based on the sample is less than a specific threshold or a certain number of iterations are completed. When this is met, the original inter-group comment node is considered to have been trained, and a trained inter-group comment node is obtained. At this time, it will evaluate the corresponding action value within the group more accurately, and can better provide evaluation and optimization guidance for the training nodes within the group.

[0136] Figure 8 is a flowchart of another method for optimizing deep reinforcement learning decision making according to an embodiment of the present invention. Figure 8 As shown, the process includes: initialization stage, environment interaction stage, critic training stage, actor training stage, strategy evaluation stage, condition judgment stage, strategy replication stage, and continuous training and convergence judgment stage;

[0137] Initialization phase: Initialize the network parameters of various training-related nodes (training, inter-group comments, strategy evaluation), experience pool (public), actors (i.e. action nodes) and critics (i.e. evaluation nodes). These parameters will be continuously optimized later.

[0138] Environmental interaction stage: The actor takes action based on the environmental state, the environmental feedback generates a new state, and the generated experience (environmental state, action, reward) is stored in the private and public experience pools.

[0139] Critic training phase: For critics within the training node, the network parameters are optimized by sampling from the private experience pool to evaluate the action value. For inter-group critics outside the training node (i.e., inter-group comment nodes), the experience of all nodes in the group is sampled and optimized to evaluate the action value from a macro perspective to avoid local problems.

[0140] Actor training phase: The actor obtains samples from the private experience pool, obtains samples from the public experience pool based on the action selection probability through the experience filter, calculates the action gradient based on the action value evaluation of the critics within the node and between groups, and then adjusts the network parameters to optimize the action strategy.

[0141] Strategy evaluation phase: The actor interacts with the environment to obtain the cumulative returns (such as rewards and steps) of the current strategy network and reports them to the strategy evaluation node as evaluation data.

[0142] Condition judgment stage: Check whether the number of training rounds reaches the preset value of a single batch. If not, return to the environment interaction stage to continue training. If it reaches, enter the strategy replication stage.

[0143] Strategy replication phase: The strategy evaluation node determines the superior and inferior nodes based on the evaluation data, copies the network parameters of the superior nodes, and replaces the inferior node parameters after adding a small amount of noise.

[0144] Continuous training and convergence judgment phase: Repeat the above training process to monitor whether the cluster converges (small differences in node performance and parameters, small changes in batch training) or reaches the preset termination state. If the conditions are met, the optimal action strategy is output.

[0145] In this embodiment, a device for optimizing deep reinforcement learning decisions is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0146] This embodiment provides an optimization device for deep reinforcement learning decision making, such as Fig. 9 As shown, including:

[0147] A detection module 91 is used to detect whether the candidate network cluster is in a policy evaluation state, wherein the candidate network cluster includes multiple groups of training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a public experience pool, and each group of training nodes includes n training nodes, n≥1;

[0148] An acquisition module 92, for acquiring the evaluation results corresponding to each training node output by the strategy evaluation node if in the strategy evaluation state;

[0149] An optimization module 93 is used to optimize the training nodes in the candidate network cluster according to the evaluation results until the candidate network cluster meets the preset termination condition to obtain the target action decision network;

[0150] The determination module 94 is used to determine the target action node in the target action decision network, and use the action strategy corresponding to the target action node as the target action strategy.

[0151] In an optional embodiment of the present application, the device also includes: a first interaction module, used to initialize multiple node components of the training nodes in the original network cluster, wherein the node components include original action nodes, original comment nodes, preset environments, and private experience pools; using the original action nodes of each training node to interact with the corresponding preset environments to obtain round experience, and storing the round experience in the corresponding private experience pool and public experience pool; optimizing the original network cluster based on the private experience pool and the public experience pool to obtain a candidate network cluster.

[0152] In an optional implementation of the present application, a first interaction module is used to collect a first experience sample in a private experience pool of a training node; use the first experience sample to train an original comment node until the training conditions are met to obtain a target comment node; collect a second experience sample in a public experience pool, and merge the first experience sample with the second experience sample to obtain a merged experience sample; use the merged experience sample to train the original action node to obtain an action node; obtain the trained inter-group comment node, and adjust the original network cluster based on the action node, the target comment node and the inter-group comment node to obtain a candidate network cluster.

[0153] In an optional embodiment of the present application, a first interaction module is used to identify the grouping of training nodes in the original network cluster; determine multiple node groups according to the grouping, and determine the original inter-group comment nodes matching each node group according to a preset association relationship; collect a third experience sample from a private experience pool within the node group; and use the third experience sample to train the original inter-group comment nodes matching the node group to obtain trained inter-group comment nodes.

[0154] In an optional embodiment of the present application, the detection module 91 is used to obtain the current number of training rounds of the candidate network cluster; determine whether the current number of training rounds reaches the preset number of single-batch training rounds; if the number of single-batch training rounds is reached, the candidate network cluster is controlled to enter the strategy evaluation state; if the number of single-batch training rounds is not reached, the training operation is continued until the number of single-batch training rounds is reached.

[0155] In an optional embodiment of the present application, the device also includes: a second interaction module, which is used to utilize the action nodes in the candidate network cluster to interact with the corresponding preset environment to obtain environmental feedback data; analyze the action strategies output by the action nodes in the training nodes according to the environmental feedback data to obtain action scores; and transmit the action scores to the strategy evaluation nodes in the candidate network cluster.

[0156] In an optional embodiment of the present application, the second interaction module is used to receive action scores from training nodes; use the inter-group comment nodes associated with the training nodes to analyze the action strategies output by the action nodes in the training nodes to obtain inter-group action scores; and evaluate the training nodes based on the action scores and the inter-group action scores to obtain corresponding evaluation results.

[0157] In an optional embodiment of the present application, the optimization module 93 is used to determine the dominant nodes and the inferior nodes in the candidate network cluster according to the evaluation results; based on the first network parameters of the dominant nodes, perform a replacement operation on the second network parameters of the inferior nodes; monitor the training status of each batch of each training node in the candidate network cluster; determine whether the candidate network cluster meets the preset termination conditions according to the training status, and when the preset termination conditions are met, use the candidate network cluster as the target action decision network.

[0158] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0159] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0160] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0161] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0163] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0164] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0165] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A deep reinforcement learning decision optimization method, characterized in that: The method comprises: Detecting whether the candidate network cluster is in a policy evaluation state, wherein the candidate network cluster includes multiple groups of training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a public experience pool, and each group of training nodes includes n training nodes, where n≥1; If it is in the strategy evaluation state, obtaining the evaluation results corresponding to each of the training nodes output by the strategy evaluation node; Optimizing the training nodes in the candidate network cluster according to the evaluation result until the candidate network cluster meets a preset termination condition to obtain a target action decision network; A target action node is determined in the target action decision network, and an action strategy corresponding to the target action node is used as a target action strategy.

2. The method according to claim 1, characterized in that Before detecting whether the candidate network cluster is in a policy evaluation state, the method further includes: Initializing multiple node components of training nodes in the original network cluster, wherein the node components include original action nodes, original comment nodes, a preset environment, and a private experience pool; Using the original action node of each training node to interact with the corresponding preset environment to obtain round experience, and storing the round experience in the corresponding private experience pool and public experience pool; The original network cluster is optimized based on the private experience pool and the public experience pool to obtain a candidate network cluster.

3. The method according to claim 2, characterized in that The optimizing the original network cluster based on the private experience pool and the public experience pool to obtain a candidate network cluster includes: Collecting a first experience sample in a private experience pool of the training node; Using the first experience sample to train the original comment node until a training condition is met to obtain a target comment node; Collecting a second experience sample from the public experience pool, and merging the first experience sample with the second experience sample to obtain a merged experience sample; Using the combined experience sample to train the original action node to obtain an action node; The trained inter-group comment nodes are obtained, and the original network cluster is adjusted based on the action nodes, the target comment nodes, and the inter-group comment nodes to obtain a candidate network cluster.

4. The method according to claim 3, characterized in that The step of obtaining the trained inter-group comment node includes: Identify the grouping of training nodes in the original network cluster; Determine a plurality of node groups according to the grouping situation, and determine an original inter-group comment node that matches each of the node groups according to a preset association relationship; Collecting a third experience sample in a private experience pool within the node group; The third experience sample is used to train the original inter-group comment nodes that match the node grouping to obtain trained inter-group comment nodes.

5. The method according to claim 1, characterized in that The detecting whether the candidate network cluster is in a policy evaluation state includes: Obtaining the current training round number of the candidate network cluster; Determine whether the current number of training rounds reaches a preset number of single-batch training rounds; If the number of single-batch training rounds is reached, the candidate network cluster is controlled to enter a strategy evaluation state; if the number of single-batch training rounds is not reached, the training operation is continued until the number of single-batch training rounds is reached.

6. The method according to claim 1, characterized in that Before obtaining the evaluation results corresponding to the respective training nodes output by the strategy evaluation node, the method further includes: Utilize the action nodes in the candidate network cluster to interact with the corresponding preset environment to obtain environment feedback data; Analyze the action strategy output by the action node in the training node according to the environmental feedback data to obtain an action score; The action score is transmitted to a policy evaluation node in the candidate network cluster.

7. The method according to claim 6, characterized in that The obtaining of the evaluation results corresponding to the respective training nodes output by the strategy evaluation node comprises: receiving an action score from the training node; Utilizing the inter-group comment node associated with the training node, analyzing the action strategy output by the action node in the training node to obtain an inter-group action score; The training node is evaluated based on the action score and the inter-group action score to obtain a corresponding evaluation result.

8. The method according to claim 1, characterized in that The step of optimizing the training nodes in the candidate network cluster according to the evaluation result until the candidate network cluster meets a preset termination condition to obtain a target action decision network includes: Determine the dominant nodes and the inferior nodes in the candidate network cluster according to the evaluation result; Based on the first network parameter of the dominant node, performing a replacement operation on the second network parameter of the disadvantaged node; Monitoring the training status of each batch of each training node in the candidate network cluster; It is determined whether the candidate network cluster meets a preset termination condition according to the training situation, and when the preset termination condition is met, the candidate network cluster is used as the target action decision network.

9. A deep reinforcement learning decision optimization device, characterized in that: The device comprises: A detection module, used to detect whether the candidate network cluster is in a policy evaluation state, wherein the candidate network cluster includes multiple groups of training nodes, multiple inter-group comment nodes, policy evaluation nodes and a public experience pool, each group of training nodes includes n training nodes, n≥1; An acquisition module, used for acquiring the evaluation results corresponding to each of the training nodes output by the strategy evaluation node if in the strategy evaluation state; An optimization module, used to optimize the training nodes in the candidate network cluster according to the evaluation result, until the candidate network cluster meets a preset termination condition, thereby obtaining a target action decision network; The determination module is used to determine the target action node in the target action decision network and use the action strategy corresponding to the target action node as the target action strategy.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 8 by executing the computer instructions.

Images