Optimization method and apparatus for deep reinforcement learning decision-making, computer device, and medium

Abstract

The present application relates to the technical field of deep reinforcement learning, and discloses an optimization method and apparatus for deep reinforcement learning decision-making, a computer device, and a medium. The method comprises: detecting whether a candidate network cluster is in a policy evaluation state; if the candidate network cluster is in the policy evaluation state, acquiring an evaluation result corresponding to each training node outputted by a policy evaluation node; on the basis of the evaluation result, optimizing each training node in the candidate network cluster until the candidate network cluster satisfies a preset termination condition, so as to obtain a target action decision-making network; and determining a target action node in the target action decision-making network, and using an action policy corresponding to the target action node as a target action policy. The present application solves the problems of low training efficiency and tendencies toward local optimization and experience forgetting of a deep reinforcement learning decision-making optimization solution, insufficient utilization of agent correlation in a multi-node solution, and unsuitability of a single-node solution for multi-node training.

Description

Optimization methods, devices, computer equipment and media for deep reinforcement learning decision making

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411677640.6, filed on November 21, 2024, entitled “Optimization Method, Apparatus, Computer Equipment and Medium for Deep Reinforcement Learning Decision Making”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of deep reinforcement learning, specifically to optimization methods, apparatus, computer devices, and media for deep reinforcement learning decision-making. Background Technology

[0004] With the development of artificial intelligence, deep reinforcement learning, which integrates the capabilities of deep learning and reinforcement learning, has achieved remarkable results in many fields, such as robot control, Go, and arcade games. Reinforcement learning is used to solve the problem of action policy learning in the interaction between intelligent agents and their environment. The action-review node framework is the mainstream approach, and proximal policy optimization is one such method. The intelligent agent selects actions based on the state of the environment, which involves state transitions and reward feedback. Action policies are mostly neural network mappings, and rewards and cumulative rewards are important for decision optimization. An experience pool stores interaction data for training.

[0005] However, deep reinforcement learning decision optimization schemes have many problems in practical applications: when solving action decision problems, reinforcement learning methods require a large number of samples and computation time, resulting in low training efficiency, making it difficult to meet the needs of real-world scenarios. They are also prone to getting trapped in local optima, experiencing experience loss, and lacking stability, which can lead to policy collapse. Furthermore, the complexity of real-world task scenarios increases the demands on stability and training efficiency due to the need for reward function and hyperparameter tuning. While multi-node distributed decision optimization schemes can increase experience diversity, improve the ability to escape local optima, and increase training speed, they do not fully utilize the correlations between various networks among agents. Single-node reinforcement learning optimization schemes mainly focus on improving exploration and utilization capabilities and correcting estimation biases; they are single-agent training modes and cannot be directly extended to multi-node training. Summary of the Invention

[0006] In view of this, embodiments of this application provide optimization methods, apparatus, computer devices and media for deep reinforcement learning decision-making, in order to solve the problems of low training efficiency, easy occurrence of local optima and experience forgetting in deep reinforcement learning decision-making optimization schemes, insufficient utilization of agent correlation in multi-node schemes and unsuitability of single-node schemes for multi-node training.

[0007] In a first aspect, embodiments of this application provide an optimization method for deep reinforcement learning decision-making, the method comprising:

[0008] Detect whether the candidate network cluster is in the policy evaluation state, wherein the candidate network cluster includes multiple sets of training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a common experience pool, and each set of training nodes includes n training nodes, n≥1;

[0009] If the policy evaluation state is in progress, the evaluation results corresponding to each training node output by the policy evaluation node are obtained.

[0010] The training nodes in the candidate network cluster are optimized based on the evaluation results until the candidate network cluster meets the preset termination conditions, thereby obtaining the target action decision network.

[0011] In the target action decision network, a target action node is determined, and the action strategy corresponding to the target action node is taken as the target action strategy.

[0012] Furthermore, before detecting whether the candidate network cluster is in the policy evaluation state, the method further includes:

[0013] Initialize multiple node components of the training nodes in the original network cluster, wherein the node components include the original action node, the original comment node, the preset environment, and the private experience pool;

[0014] Each training node interacts with its original action node and the corresponding preset environment to obtain round experience, and stores the round experience in the corresponding private experience pool and public experience pool.

[0015] The original network cluster is optimized based on the private experience pool and the public experience pool to obtain a candidate network cluster.

[0016] Furthermore, the optimization of the original network cluster based on the private experience pool and the public experience pool to obtain candidate network clusters includes:

[0017] Collect the first experience sample from the private experience pool of the training node;

[0018] The original comment node is trained using the first experience sample until the training conditions are met, and the target comment node is obtained.

[0019] A second experience sample is collected from the public experience pool, and the first experience sample is merged with the second experience sample to obtain a merged experience sample;

[0020] The original action nodes are trained using the merged experience samples to obtain action nodes;

[0021] Obtain trained inter-group comment nodes, and adjust the original network cluster based on the action nodes, the target comment nodes, and the inter-group comment nodes to obtain a candidate network cluster.

[0022] Furthermore, obtaining the trained inter-group comment nodes includes:

[0023] Identify the grouping of training nodes in the original network cluster;

[0024] Based on the grouping situation, multiple node groups are determined, and the original inter-group comment nodes that match each node group are determined according to the preset association relationship;

[0025] A third experience sample is collected from the private experience pool within the node group;

[0026] The original inter-group comment nodes that match the node grouping are used to train the trained inter-group comment nodes.

[0027] Furthermore, detecting whether the candidate network cluster is in a policy evaluation state includes:

[0028] Obtain the current training round number of the candidate network cluster;

[0029] Determine whether the current training round number has reached the preset single batch training round number;

[0030] If the number of training rounds per batch is reached, the candidate network cluster is controlled to enter the policy evaluation state. If the number of training rounds per batch is not reached, the training operation continues until the number of training rounds per batch is reached.

[0031] Furthermore, before obtaining the evaluation results corresponding to each of the training nodes output by the policy evaluation node, the method further includes:

[0032] By utilizing the action nodes in the candidate network cluster to interact with the corresponding preset environment, environmental feedback data is obtained;

[0033] The action strategies output by the action nodes in the training nodes are analyzed based on the environmental reward data to obtain action scores;

[0034] The action score is transmitted to the policy evaluation node in the candidate network cluster.

[0035] Furthermore, obtaining the evaluation results corresponding to each training node output by the policy evaluation node includes:

[0036] Receive action scores from the training nodes;

[0037] By utilizing the inter-group comment nodes associated with the training nodes, the action strategies output by the action nodes in the training nodes are analyzed to obtain the inter-group action scores;

[0038] The training node is evaluated based on the action score and the inter-group action score to obtain the corresponding evaluation result.

[0039] Furthermore, the step of optimizing the training nodes in the candidate network cluster based on the evaluation results until the candidate network cluster meets a preset termination condition to obtain the target action decision network includes:

[0040] Based on the evaluation results, the dominant and disadvantageous nodes in the candidate network cluster are determined;

[0041] Based on the first network parameters of the advantageous node, a replacement operation is performed on the second network parameters of the disadvantaged node;

[0042] Monitor the training status of each training node in the candidate network cluster for each batch;

[0043] Based on the training results, it is determined whether the candidate network cluster meets the preset termination condition. If the preset termination condition is met, the candidate network cluster is used as the target action decision network.

[0044] Secondly, embodiments of this application provide an optimization apparatus for deep reinforcement learning decision-making, the apparatus comprising:

[0045] The detection module is used to detect whether the candidate network cluster is in the policy evaluation state. The candidate network cluster includes multiple sets of training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a common experience pool. Each set of training nodes includes n training nodes, where n≥1.

[0046] The acquisition module is used to acquire the evaluation results corresponding to each training node output by the policy evaluation node if the policy evaluation state is in progress.

[0047] The optimization module is used to optimize the training nodes in the candidate network cluster according to the evaluation results until the candidate network cluster meets the preset termination conditions to obtain the target action decision network.

[0048] The determination module is used to determine the target action node in the target action decision network and take the action strategy corresponding to the target action node as the target action strategy.

[0049] Thirdly, embodiments of this application provide a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the method described in the first aspect or any corresponding embodiment.

[0050] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment.

[0051] The method provided in this application has the following beneficial effects:

[0052] The method provided in this application grasps the training process by detecting the evaluation state of candidate network cluster strategies, providing a decision-making basis for the optimization process, ensuring reasonable training, and avoiding resource waste and erroneous operations. Obtaining evaluation results allows understanding the performance of training nodes, guiding the direction of optimization operations, and addressing nodes with different performance characteristics in a targeted manner to improve network cluster performance, training efficiency, and quality. Optimizing training nodes based on evaluation results can propagate the characteristics of advantageous nodes and improve overall performance. Continuous optimization until a preset termination condition is met can ensure the quality and stability of the target action decision network, reduce local optima, and improve the accuracy and reliability of deep reinforcement learning decisions.

[0053] The method provided in this application trains the network cluster from a stable and reasonable initial state, providing a foundation for interaction and optimization. Initializing node components avoids training problems caused by initial settings. Action nodes interact with the environment to acquire and store experience, increasing the sources and diversity of experience. Private and public experience pool storage methods provide different levels of training foundations, facilitating the full utilization of experience data, improving training effectiveness, and reducing acquisition costs. The dual-experience-pool optimization method utilizes experience from different sources to drive the optimization of the original network cluster. Combining the two types of experience can uncover more information, improve training quality, and ensure the acquisition of high-quality candidate network clusters.

[0054] The method provided in this application, which samples data from a private experience pool, can leverage the personalized experience of training nodes to improve the targeting of single-node training and uncover potential advantages. Training comment nodes using specific experience samples can improve their accuracy in evaluating the value of actions, thereby guiding action nodes to select better strategies and enhancing the decision-making ability of the network cluster. Collecting and merging public and private experience samples allows for a comprehensive consideration of cluster and individual experience, enriching training data, making action node training more comprehensive, and improving adaptability and network cluster performance. Adjusting the network cluster using multiple trained nodes can leverage the strengths of each node, making the structure and parameters more reasonable, and improving collaboration and performance in deep reinforcement learning decision-making.

[0055] The method provided in this application can clarify the characteristics of inter-group relationships by identifying groupings, providing a basis for training inter-group comment nodes and ensuring relevance and effectiveness. Determining matching inter-group comment nodes based on groupings allows them to be closely associated with corresponding group training nodes, better evaluating action value and improving inter-group training synergy. Sampling inter-group comment nodes from the private experience pool within each group focuses on common characteristics within the group, improving evaluation accuracy, promoting collaborative training of nodes within the group, and enhancing network cluster performance.

[0056] The method provided in this application obtains feedback data through interaction between action nodes and the environment, providing a basis for analysis and evaluation, and helping to adjust the training direction. The scoring of action strategies can quantify their merits and demerits, facilitating comparison and evaluation, and intuitively reflecting the performance of action nodes, providing a reference indicator for comprehensive evaluation and optimization. Transmitting action scores to the strategy evaluation node enables it to obtain comprehensive information, ensuring accurate and comprehensive evaluation results, and providing a basis for optimizing training nodes.

[0057] The method provided in this application ensures that the strategy evaluation node has key information about each node by receiving action scores, which affects the accuracy of performance judgment. Obtaining inter-group action scores through inter-group comment nodes allows for evaluation of action nodes from different perspectives, making the evaluation more comprehensive and objective. Evaluating training nodes by combining action scores and inter-group action scores comprehensively considers individual and cluster-level performance, providing targeted guidance for optimization and improving optimization effectiveness.

[0058] The method provided in this application provides a clear objective for network cluster optimization by identifying dominant and subordinate nodes. Parameter replacement operations can pass on the characteristics of dominant nodes to improve the performance of subordinate nodes, avoiding getting trapped in local optima and improving overall performance. Monitoring the training process can promptly detect anomalies, ensuring stability and reliability, and adjusting the optimization strategy accordingly to ensure that training progresses as expected. Determining the target action decision network by judging termination conditions can guarantee network performance quality, avoid overtraining or undertraining, and enable it to effectively execute decision-making tasks.

[0059] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0061] Figure 1 is a flowchart illustrating the optimization method for deep reinforcement learning decision-making according to an embodiment of this application;

[0062] Figure 2 is a schematic diagram of the network architecture of a candidate network cluster according to an embodiment of this application;

[0063] Figure 3 is a schematic diagram of the network architecture of training nodes in a candidate network cluster according to an embodiment of this application;

[0064] Figure 4 is a schematic diagram of the working principle of the action node in a multi-joint robot control scenario based on a physics simulation engine according to an embodiment of this application.

[0065] Figure 5 is a schematic diagram of the working principle of the strategy evaluation node according to an embodiment of this application;

[0066] Figure 6 is a schematic diagram illustrating the working principle of the inter-group comment node according to an embodiment of this application;

[0067] Figure 7 is a schematic diagram of collecting experience from a public experience pool according to an embodiment of this application;

[0068] Figure 8 is a flowchart illustrating another optimization method for deep reinforcement learning decision-making according to an embodiment of this application;

[0069] Figure 9 is a structural block diagram of an optimization device for deep reinforcement learning decision-making according to an embodiment of this application;

[0070] Figure 10 is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Specific Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] According to embodiments of this application, optimization methods, apparatus, computer devices, and media for deep reinforcement learning decision-making are provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0073] This embodiment provides an optimization method for deep reinforcement learning decision-making. Figure 1 is a flowchart of the optimization method for deep reinforcement learning decision-making according to an embodiment of this application. As shown in Figure 1, the process includes the following steps:

[0074] Step S11: Detect whether the candidate network cluster is in the policy evaluation state. The candidate network cluster includes multiple training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a public experience pool. Each training node includes n training nodes, where n≥1.

[0075] It should be noted that, as shown in Figure 2, the network architecture of the candidate network cluster includes: policy evaluation nodes, a public experience pool, training nodes (divided into m node groups, with n training nodes in each group), and corresponding m inter-group comment nodes. The policy evaluation state refers to the stage where the network cluster evaluates the action policies of each training node. In this stage, the performance of the training nodes in the environment can be used to determine the quality of their policies, thereby deciding whether to perform operations such as copying or removing nodes to optimize the performance of the entire network cluster.

[0076] In this embodiment of the application, before obtaining the evaluation results corresponding to each training node output by the policy evaluation node, the method further includes the following steps A1-A3:

[0077] Step A1: Use the action nodes in the candidate network cluster to interact with the corresponding preset environment to obtain environment reward data.

[0078] It should be noted that, as shown in Figure 3, the network architecture of each training node in the candidate network cluster includes an action node, a preset environment, a private experience pool, and a target comment node. The main function of the action node is to output action selections based on the current environmental state through its built-in action network. For example, as shown in Figure 4, in a multi-joint robot control scenario based on the physics simulation engine MuJoCo, the action node determines the robot's next action based on environmental state information such as the robot's current position, angle, and speed, including joint rotation angles and movement directions. The preset environment is the environment set for the decision optimization task. For multi-joint robots, it includes various terrains and obstacles, and its state changes according to the actions performed by the action node.

[0079] Specifically, when a robot performs an action in a preset environment, the environment provides feedback based on the action's execution. This feedback data is called environmental reward data. For example, during a robot's movement, if it successfully avoids an obstacle, the environment provides a positive reward value (positive experience); if it collides with an obstacle, it provides a negative reward value (negative experience). This reward data reflects the effectiveness of the robot's action strategy in the current environment.

[0080] Step A2: Analyze the action strategies output by the action nodes in the training nodes based on the environmental reward data to obtain action scores.

[0081] Specifically, environmental feedback data is a crucial basis for analyzing action strategies. By analyzing this data, we can understand how the action strategies adopted by the action nodes perform in a pre-defined environment. Action scores are a quantitative evaluation of the action strategies output by the action nodes. For example, if environmental feedback data shows that the robot can efficiently complete the task over a period of time (such as quickly reaching the target location and avoiding all obstacles), the corresponding action strategy will receive a high score; conversely, if the robot frequently makes mistakes (such as colliding with obstacles multiple times or failing to reach the target for an extended period), the action strategy will receive a low score. This score directly reflects the quality of the action strategy.

[0082] Step A3: Transmit the action score to the policy evaluation node in the candidate network cluster.

[0083] Specifically, as shown in Figure 5, the policy evaluation node is the unit that evaluates the action policies of all actors. In each training round, it obtains evaluation metrics such as cumulative reward, cumulative steps, or differences between policies, as well as the action score corresponding to each training node. Based on this, it decides whether to copy or remove a training node according to the policy copying and removal mechanism. Transmitting the action score to the policy evaluation node allows it to comprehensively evaluate the action policies of the candidate network cluster by integrating the performance of each action node, thereby making more accurate decisions by combining other performance information.

[0084] In this embodiment of the application, step S11 includes the following steps B1-B3:

[0085] Step B1: Obtain the current training round number of the candidate network cluster.

[0086] Specifically, in reinforcement learning training, training is conducted in rounds. In each round, the actor interacts with the environment and updates its own parameters and experience pool based on the results. The number of training rounds is a way of counting the training process, used to determine the stage and progress. Specifically, a counter can be used to record the number of training rounds completed. The counter increments by 1 upon completion of one round of training (including actor-environment interaction, critic training, and actor training based on new experiences). The current training round number can be obtained by querying the counter value.

[0087] Step B2: Determine whether the current number of training rounds has reached the preset number of training rounds per batch.

[0088] Specifically, when building a candidate network cluster, the number of training rounds per batch is preset, and its value can be determined based on factors such as the training task and model complexity. For example, it can be set to 50 for a simple multi-joint robot motion policy training task, and 200 for a task in a complex environment. It is a threshold that determines when to evaluate the policy. Determining whether the current number of training rounds has reached the preset value is to determine whether to enter the policy evaluation stage. If the threshold has not been reached, it means that the training is accumulating experience and optimizing the actors and critics. If the threshold has been reached, the training situation needs to be evaluated to consider whether to adjust the training nodes.

[0089] Step B3: If the number of training rounds per batch is reached, control the candidate network cluster to enter the policy evaluation state; if the number of training rounds per batch is not reached, continue to perform training operations until the number of training rounds per batch is reached.

[0090] Specifically, when the current training rounds reach the preset number of training rounds per batch, the candidate network cluster enters the policy evaluation state. At this time, the policy evaluation nodes comprehensively evaluate the action policies of each training node, integrating various information (such as actor action performance, environmental reward data, etc.) to prepare for subsequent node optimization. If the preset value is not reached, training continues. Actors interact with the environment to sample and obtain new round experiences, which are then stored. Critics train based on the new experience data, and actors continuously train themselves using the new experience and critic value estimates to optimize their action policies until the preset number of training rounds per batch is reached before entering the policy evaluation state.

[0091] Step S12: If in the policy evaluation state, obtain the evaluation results corresponding to each training node output by the policy evaluation node.

[0092] In this embodiment of the application, step S12 includes the following steps C1-C3:

[0093] Step C1: Receive action scores from the training nodes.

[0094] Specifically, the policy evaluation node can receive action scores transmitted from each training node, ensuring that each training node can accurately send its own action score to the policy evaluation node for subsequent comprehensive evaluation.

[0095] Step C2: Analyze the action strategies output by the action nodes in the training nodes using the inter-group comment nodes associated with the training nodes to obtain the inter-group action scores.

[0096] Specifically, the inter-group commentary node is an evaluation unit trained based on all experiences within the training node group. Unlike commentary nodes in a single training node, it can evaluate the action strategy from a macro perspective of the experiences of all nodes within the group. In the strategy evaluation state, for each training node, the action strategy output by the action node is further analyzed using its associated inter-group commentary nodes. A comprehensive evaluation is conducted by integrating relevant experiences from other nodes within the group and the performance of the action node itself, resulting in an inter-group action score. This can supplement the shortcomings of action scores obtained from single-node experiences and provide more comprehensive and multi-dimensional evaluation information.

[0097] As an example, as shown in Figure 6, multiple private experience pools (private experience pool 1, private experience pool 2, ..., private experience pool n) sample their respective comment nodes (commentator 1, commentator 2, ..., commentator n). These comment nodes then sample the inter-group comment nodes (inter-group commentator 1). Inter-group commentator 1 comprehensively considers the information obtained from each commentator and analyzes the action strategies output by the action nodes in each training node. It can utilize the information sampled from multiple private experience pools to comprehensively evaluate the action strategies from a macro perspective, thereby deriving the inter-group action value (inter-group action score). This scoring method based on multi-node experience within the group avoids the limitations of relying solely on the experience of a single node for evaluation, thus providing a richer and more accurate reference for strategy evaluation.

[0098] Step C3: Evaluate the training nodes based on action scores and inter-group action scores to obtain the corresponding evaluation results.

[0099] Specifically, after obtaining the training node's own action score and the inter-group action score, the policy evaluation node considers both comprehensively, for example, by using a weighted average (the weights are determined according to the actual situation) to combine the two scores into a comprehensive evaluation index according to rules. This result can accurately reflect the performance of the training node in the candidate network cluster, and is an important basis for the training node to optimize its operations (such as determining superior and inferior nodes), which helps to improve the performance of the network cluster and obtain a better action decision network.

[0100] Step S13: Optimize the training nodes in the candidate network cluster according to the evaluation results until the candidate network cluster meets the preset termination conditions to obtain the target action decision network.

[0101] In this embodiment of the application, step S13 includes the following steps D1-D4:

[0102] Step D1: Determine the dominant and disadvantageous nodes in the candidate network cluster based on the evaluation results.

[0103] Specifically, the evaluation results obtained through comprehensive assessment of each training node fully reflect the performance of each node in the current training phase and are an important basis for determining the dominant and subordinate nodes. The determination method typically involves setting thresholds or using a ranking method. For example, nodes with evaluation results above a certain threshold can be identified as dominant nodes, exhibiting excellent performance in interacting with the environment and executing action strategies, demonstrating high action efficiency and good strategy adaptability; nodes with evaluation results below the threshold are considered subordinate nodes, performing relatively weakly in various aspects and requiring further optimization and adjustment.

[0104] Step D2: Based on the first network parameters of the dominant node, perform a replacement operation on the second network parameters of the inferior node.

[0105] Specifically, in the training nodes, each node has network parameters that determine its behavior and performance. For example, the action policy network parameters of the action node (actor) affect action selection, and the action value estimation network parameters of the commentator node (commentator) affect the accuracy of value assessment. The first network parameter belongs to the dominant node, and the second network parameter belongs to the suboptimal node. The replacement operation introduces the good network parameters of the dominant node into the suboptimal node to improve its performance and optimize the cluster performance. The replacement operation replicates the first network parameter of the dominant node, adds a small amount of noise (to avoid overfitting), and then replaces the second network parameter of the suboptimal node, allowing the suboptimal node to inherit the characteristics of the dominant node while having room for variation to adapt to different training conditions.

[0106] Step D3: Monitor the training status of each training node in the candidate network cluster for each batch.

[0107] After optimizing the training nodes, it is necessary to continuously monitor the training status of each node in subsequent batches. Monitoring includes changes in empirical data related to interaction with the environment (such as reward values ​​and state transitions), updates to network parameters trained based on new empirical data, and changes in action policy performance (such as accuracy and efficiency). Monitoring helps understand the effectiveness of the optimization. If a node that was previously deemed weak after parameter replacement shows poor performance, further analysis is needed to identify the reasons and implement targeted improvements.

[0108] Step D4: Determine whether the candidate network cluster meets the preset termination condition based on the training situation. If the preset termination condition is met, the candidate network cluster is used as the target action decision network.

[0109] Specifically, a termination condition is preset before training, which is related to factors such as the performance of training nodes, the number of training epochs, and the number of batches. For example, it can be set to meet the condition when the performance differences of the action network of all training nodes are small and the changes in each batch of training are small, or when a certain number of training epochs or batches are reached. After monitoring the training status of each training node in each batch, it is compared with the preset termination condition. If the condition is met, it means that the candidate network cluster has been sufficiently trained and optimized, and it can be used as the target action decision network. This network contains optimized nodes, has ideal overall performance, and the output action strategy is also relatively optimal.

[0110] Step S14: Determine the target action node in the target action decision network, and use the action strategy corresponding to the target action node as the target action strategy.

[0111] In this embodiment, the methods for determining target action nodes in the optimized target action decision network include, but are not limited to: selecting based on the performance of each action node during previous training, such as high reward values ​​for interacting with the environment, high efficiency in executing action strategies, and good action accuracy; determining based on specific task requirements, selecting the fastest-responding action node for tasks requiring rapid response, and selecting the action node with the best execution accuracy for tasks emphasizing precision. After determining the target action node, its corresponding action strategy is identified as the target action strategy. This strategy, after continuous optimization and screening, is expected to achieve the optimal decision-making effect under the current task scenario and network architecture, helping to complete related decision optimization tasks, such as enabling multi-joint robots to complete walking, grasping, and other action tasks more efficiently and accurately.

[0112] In this embodiment of the application, before step S11, the following steps S21-S23 are also included:

[0113] Step S21: Initialize multiple node components of the training nodes in the original network cluster, including the original action node, the original comment node, the preset environment, and the private experience pool.

[0114] In this embodiment, the initial network cluster before the training process begins serves as the infrastructure for subsequent training and optimization operations. It includes multiple training nodes and related components (initial action nodes, initial comment nodes, preset environment, and private experience pool), each in its initial preset state. The initial action node outputs action selection based on the environmental state, such as determining robot actions in a multi-joint robot control scenario. The initial comment node evaluates the actions and environmental state to provide a basis for training optimization. The preset environment is set according to the task and changes based on the actions of the action nodes. The private experience pool stores the private experience data accumulated by the training nodes for subsequent component training and optimization.

[0115] Specifically, initialization involves assigning initial states or parameter values ​​to node components to ensure their proper functioning during subsequent training. This includes setting the initial action strategy network parameters for the original action node, determining the initial action value estimation network parameters for the original comment node, and setting initial state conditions for the preset environment. For example, initial weights can be assigned to the parameters of the original action node based on prior knowledge to initially select actions, and initial values ​​can be assigned to the parameters of the original comment node to initially evaluate actions and states.

[0116] Step S22: Use the original action node of each training node to interact with the corresponding preset environment to obtain round experience, and store the round experience in the corresponding private experience pool and public experience pool.

[0117] In this embodiment, the original action nodes of each training node interact with the corresponding preset environment based on the initial action policy network parameters. They output action selections according to the environment state, and the preset environment changes accordingly, providing a reward value. This generates round experience, including information such as environment state, action, and reward value. After obtaining the round experience, it needs to be stored for subsequent training optimization. This storage includes both the private experience pool of the corresponding training node for its own training optimization and the public experience pool for other nodes to utilize, thereby improving the network cluster training efficiency and achieving experience sharing and collaborative training. For example, other training nodes can combine public and private experience for more comprehensive optimization during action node training.

[0118] Step S23: Optimize the original network cluster based on the private experience pool and the public experience pool to obtain candidate network clusters.

[0119] In this embodiment of the application, step S23 includes the following steps E1-E5:

[0120] Step E1: Collect the first experience sample from the private experience pool of the training node.

[0121] Specifically, the private experience pool stores the private experiences obtained by training nodes through interaction sampling with the environment. This data is unique to each node and reflects its actions in a specific environment. The first experience sample is collected from this pool for use in training and optimizing related components. The collection method can be based on rules or random selection to obtain representative data as samples, providing a data foundation for the training of the original comment nodes.

[0122] Step E2: Train the original comment nodes using the first empirical samples until the training conditions are met, and obtain the target comment nodes.

[0123] Specifically, the original comment node is mainly used to evaluate the action of the action node and the current environmental state, providing an estimate of the action value and calculating the action gradient, thus providing a basis for judging the quality of actions during training. Training it using the first empirical samples is to enable it to more accurately provide reasonable value estimates for different states and action combinations, improving its action strategy optimization capabilities. During training, samples are input into the original comment node, which adjusts its internal parameters based on the sample information, its own network structure, and the algorithm, making the output value estimate closer to the real situation. This continues until preset training conditions are met, such as the error between the estimated value and the true value being less than a specific threshold or completing a certain number of iterations. Once training is complete, the target comment node is obtained, which will have a more accurate assessment of action value.

[0124] Step E3: Collect a second experience sample from the public experience pool, and merge the first experience sample with the second experience sample to obtain a merged experience sample.

[0125] Specifically, the public experience pool is a shared resource pool for nodes, storing environmental interaction experience data of all training nodes. It can provide diverse experiences for training nodes to improve training efficiency. As shown in Figure 7, when collecting the second experience sample from the public experience pool, samples are first taken from the pool and input into the experience filter of the training node. The filter is filtered according to the importance constraints related to the probability of action selection. The filtered experience is used for training of action nodes (actors). The action node combines the action value of comment nodes (commentators) and optimizes the action policy gradient based on the filtered experience to improve its capabilities. Then, it is merged with the first experience sample from the private experience pool to form a merged experience sample. By comprehensively utilizing public and private experiences, a richer data foundation is provided for the training of the original action node to optimize the strategy.

[0126] Step E4: Train the original action nodes using merged experience samples to obtain action nodes.

[0127] Specifically, the original action node outputs action selections based on the current environmental state through its built-in network. The purpose of training is to enable it to output appropriate actions more accurately and efficiently for different environments, thereby improving decision-making and task performance, such as helping multi-joint robots complete tasks precisely. During training, it inputs merged experience samples, combining information such as the environment, actions, and feedback in the samples with its own structure and algorithm to adjust internal parameters. After learning and adaptation, it can output better action selections to complete tasks after training is completed.

[0128] Step E5: Obtain the trained inter-group comment nodes, and adjust the original network cluster based on the action nodes, target comment nodes, and inter-group comment nodes to obtain the candidate network cluster.

[0129] In this embodiment, the trained action nodes possess optimized action selection capabilities, providing a better foundation for the network cluster to execute action tasks; the target comment nodes offer more accurate evaluations, providing reliable action value estimates and gradient calculations; and the inter-group comment nodes can evaluate action value from multiple dimensions. When adjusting the original network cluster, the characteristics of these three elements can be combined to adjust the relevant connection weights and parameters of the action nodes based on their performance; the evaluation results of the target comment nodes can be used to adjust the inter-node relationships and the degree to which other nodes utilize the evaluation information; and the inter-group comment nodes can adjust global parameters based on their value judgments. By comprehensively utilizing their characteristics to make detailed adjustments to the original network cluster in multiple aspects, a candidate network cluster is obtained, achieving a better state and laying the foundation for subsequent training and decision optimization.

[0130] In this embodiment of the application, obtaining the trained inter-group comment nodes includes the following steps E51-E54:

[0131] Step E51: Identify the grouping of training nodes in the original network cluster.

[0132] Specifically, the original network cluster contains multiple training nodes. Optimization using inter-group comment nodes requires obtaining the grouping information of these training nodes. Grouping can be based on specific rules or the needs of the actual scenario. For example, in a multi-joint robot control scenario, nodes can be grouped according to joint control requirements or task modules, facilitating the management and optimization of each group. Accurately identifying the grouping information lays the foundation for determining matching inter-group comment nodes and subsequent targeted training and adjustment operations.

[0133] Step E52: Determine multiple node groups based on the grouping situation, and determine the original inter-group comment nodes that match each node group according to the preset association relationship.

[0134] Specifically, after clarifying the grouping of training nodes, the original inter-group comment nodes matching each group can be determined based on preset associations. These associations are set at the initial stage of network cluster construction, taking into account the compatibility of training nodes and inter-group comment nodes in different groups in terms of functionality and data characteristics. For example, the group controlling the joint movements of a robot arm will be matched with original inter-group comment nodes that are good at evaluating the value of arm joint movements, so as to more accurately evaluate and optimize the movement value of nodes within the group in the future.

[0135] Step E53: Collect a third experience sample from the private experience pool within the node group.

[0136] Specifically, each defined node group has a corresponding private experience pool. Third-party experience samples are collected from this private experience pool to train matching original inter-group comment nodes. Data collection can be done according to rules or by random selection to obtain representative data samples, providing a targeted data foundation for training the original inter-group comment nodes and allowing them to better adapt to the characteristics and needs of the corresponding node groups.

[0137] Step E54: Use the third empirical sample to train the original inter-group comment nodes that match the node grouping, and obtain the trained inter-group comment nodes.

[0138] Specifically, third-party experience samples taken from the private experience pool within each node group are input into the matched original inter-group comment nodes. These nodes continuously adjust their internal parameters based on the state and action information in the samples, combined with their own network structure and algorithm, to output an action value estimate. This process continues until preset conditions are met, such as the error between the estimated value and the true value calculated based on the samples being less than a specific threshold, or a certain number of iterations being completed. When these conditions are met, the original inter-group comment node is considered to have completed training, resulting in a well-trained inter-group comment node. At this point, it will more accurately evaluate the action value within the corresponding group, providing better evaluation and optimization guidance for the trained nodes within the group.

[0139] Figure 8 is a flowchart of another deep reinforcement learning decision optimization method according to an embodiment of this application. As shown in Figure 8, the process includes: initialization phase, environment interaction phase, critic training phase, actor training phase, policy evaluation phase, condition judgment phase, policy replication phase, and continuous training and convergence judgment phase.

[0140] Initialization phase: Initialize the network parameters of various training-related nodes (training, inter-group comments, policy evaluation), experience pool (public), actors (i.e., action nodes) and commentators (i.e. evaluation nodes). These parameters will be continuously optimized in the future.

[0141] Environmental interaction phase: Actors take actions based on the environmental state, the environment provides feedback to generate new states, and the resulting experiences (environmental state, actions, rewards) are stored in private and public experience pools.

[0142] Commentator training phase: For commentators within a training node, their network parameters are optimized by sampling from a private experience pool to evaluate action value. For commentators outside the training node (i.e., inter-group commentary nodes), their network parameters are optimized by sampling from the experience of all nodes within the group to evaluate action value from a macro perspective and avoid localized problems.

[0143] Actor training phase: Actors obtain samples from private experience pools, obtain samples from public experience pools based on action selection probabilities through experience filters, calculate action gradients by combining action value evaluations from commentators within nodes and between groups, and then adjust network parameters to optimize action strategies.

[0144] Strategy evaluation phase: Actors interact with the environment to obtain the cumulative rewards (such as rewards and steps) of the current policy network, and report them as evaluation data to the strategy evaluation node.

[0145] Condition judgment phase: Check whether the number of training rounds has reached the preset value for a single batch. If not, return to the environment interaction phase to continue training. If it has, enter the policy replication phase.

[0146] Strategy replication phase: The strategy evaluation node determines the advantageous and disadvantageous nodes based on the evaluation data, replicates the network parameters of the advantageous nodes, adds a small amount of noise, and then replaces the parameters of the disadvantageous nodes.

[0147] Continuous training and convergence judgment phase: Repeat the above training process, monitor whether the cluster converges (small differences in node performance and parameters, small changes in batch training) or reaches the preset termination state, and output the optimal action strategy if the conditions are met.

[0148] This embodiment also provides an optimization apparatus for deep reinforcement learning decision-making, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0149] This embodiment provides an optimization device for deep reinforcement learning decision-making, as shown in Figure 9, including:

[0150] The detection module 91 is used to detect whether the candidate network cluster is in the policy evaluation state. The candidate network cluster includes multiple training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a common experience pool. Each training node includes n training nodes, where n≥1.

[0151] The acquisition module 92 is used to acquire the evaluation results corresponding to each training node output by the policy evaluation node if the policy evaluation state is in progress.

[0152] The optimization module 93 is used to optimize the training nodes in the candidate network cluster according to the evaluation results until the candidate network cluster meets the preset termination conditions and the target action decision network is obtained.

[0153] The determination module 94 is used to determine the target action node in the target action decision network and take the action strategy corresponding to the target action node as the target action strategy.

[0154] In an optional embodiment of this application, the apparatus further includes: a first interaction module, configured to initialize multiple node components of training nodes in the original network cluster, wherein the node components include original action nodes, original comment nodes, preset environments, and private experience pools; to use the original action nodes of each training node to interact with the corresponding preset environments to obtain round experience, and to store the round experience in the corresponding private experience pools and public experience pools; and to optimize the original network cluster based on the private experience pools and public experience pools to obtain candidate network clusters.

[0155] In an optional implementation of this application, the first interaction module is used to collect a first experience sample from the private experience pool of the training node; train the original comment node using the first experience sample until the training conditions are met to obtain the target comment node; collect a second experience sample from the public experience pool and merge the first experience sample with the second experience sample to obtain a merged experience sample; train the original action node using the merged experience sample to obtain the action node; obtain the trained inter-group comment node, and adjust the original network cluster based on the action node, the target comment node, and the inter-group comment node to obtain a candidate network cluster.

[0156] In an optional implementation of this application, the first interaction module is used to identify the grouping of training nodes in the original network cluster; determine multiple node groups according to the grouping; and determine the original inter-group comment nodes that match each node group according to a preset association relationship; collect third experience samples in the private experience pool within the node group; and use the third experience samples to train the original inter-group comment nodes that match the node group to obtain trained inter-group comment nodes.

[0157] In an optional embodiment of this application, the detection module 91 is used to obtain the current training round number of the candidate network cluster; determine whether the current training round number has reached the preset single batch training round number; if the single batch training round number has been reached, control the candidate network cluster to enter the policy evaluation state; if the single batch training round number has not been reached, continue to execute the training operation until the single batch training round number is reached.

[0158] In an optional embodiment of this application, the apparatus further includes: a second interaction module, configured to interact with the corresponding preset environment using the action nodes in the candidate network cluster to obtain environment reward data; analyze the action strategies output by the action nodes in the training nodes based on the environment reward data to obtain action scores; and transmit the action scores to the policy evaluation nodes in the candidate network cluster.

[0159] In one optional embodiment of this application, the second interaction module is configured to receive action scores from training nodes; analyze the action strategies output by action nodes in training nodes using the inter-group comment nodes associated with the training nodes to obtain inter-group action scores; and evaluate the training nodes based on the action scores and inter-group action scores to obtain corresponding evaluation results.

[0160] In an optional embodiment of this application, the optimization module 93 is used to determine the dominant and subordinate nodes in the candidate network cluster based on the evaluation results; perform a replacement operation on the second network parameters of the subordinate nodes based on the first network parameters of the dominant nodes; monitor the training status of each training node in the candidate network cluster for each batch; determine whether the candidate network cluster meets the preset termination conditions based on the training status; and when the preset termination conditions are met, use the candidate network cluster as the target action decision network.

[0161] Please refer to Figure 10, which is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application. As shown in Figure 10, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses and can be installed on a common motherboard or otherwise as needed. The processor can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system).

[0162] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0163] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0164] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0165] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0166] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0167] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0168] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification. Although embodiments of this application have been described in conjunction with the accompanying drawings, various modifications and variations can be made by those skilled in the art without departing from the spirit and scope of this application, and such modifications and variations fall within the scope defined by the appended claims.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An optimization method for deep reinforcement learning decision-making, characterized in that, The method includes: Detect whether the candidate network cluster is in the policy evaluation state, wherein the candidate network cluster includes multiple sets of training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a common experience pool, and each set of training nodes includes n training nodes, n≥1; If the policy evaluation state is in progress, the evaluation results corresponding to each training node output by the policy evaluation node are obtained. The training nodes in the candidate network cluster are optimized based on the evaluation results until the candidate network cluster meets the preset termination conditions, thereby obtaining the target action decision network. In the target action decision network, a target action node is determined, and the action strategy corresponding to the target action node is taken as the target action strategy.

2. The method according to claim 1, characterized in that, Before detecting whether the candidate network cluster is in the policy evaluation state, the method further includes: Initialize multiple node components of the training nodes in the original network cluster, wherein the node components include the original action node, the original comment node, the preset environment, and the private experience pool; Each training node interacts with its original action node and the corresponding preset environment to obtain round experience, and stores the round experience in the corresponding private experience pool and public experience pool. The original network cluster is optimized based on the private experience pool and the public experience pool to obtain a candidate network cluster.

3. The method according to claim 2, characterized in that, The optimization of the original network cluster based on the private experience pool and the public experience pool to obtain candidate network clusters includes: Collect the first experience sample from the private experience pool of the training node; The original comment node is trained using the first experience sample until the training conditions are met, and the target comment node is obtained. A second experience sample is collected from the public experience pool, and the first experience sample is merged with the second experience sample to obtain a merged experience sample; The original action nodes are trained using the merged experience samples to obtain action nodes; Obtain trained inter-group comment nodes, and adjust the original network cluster based on the action nodes, the target comment nodes, and the inter-group comment nodes to obtain a candidate network cluster.

4. The method according to claim 3, characterized in that, The process of obtaining the trained inter-group comment nodes includes: Identify the grouping of training nodes in the original network cluster; Based on the grouping situation, multiple node groups are determined, and the original inter-group comment nodes that match each node group are determined according to the preset association relationship; A third experience sample is collected from the private experience pool within the node group; The original inter-group comment nodes that match the node grouping are used to train the trained inter-group comment nodes.

5. The method according to claim 1, characterized in that, The process of detecting whether the candidate network cluster is in a policy evaluation state includes: Obtain the current training round number of the candidate network cluster; Determine whether the current training round number has reached the preset single batch training round number; If the number of training rounds per batch is reached, the candidate network cluster is controlled to enter the policy evaluation state. If the number of training rounds per batch is not reached, the training operation continues until the number of training rounds per batch is reached.

6. The method according to claim 1, characterized in that, Before obtaining the evaluation results corresponding to each training node output by the policy evaluation node, the method further includes: By utilizing the action nodes in the candidate network cluster to interact with the corresponding preset environment, environmental feedback data is obtained; The action strategies output by the action nodes in the training nodes are analyzed based on the environmental reward data to obtain action scores; The action score is transmitted to the policy evaluation node in the candidate network cluster.

7. The method according to claim 6, characterized in that, The step of obtaining the evaluation results corresponding to each training node output by the policy evaluation node includes: Receive action scores from the training nodes; By utilizing the inter-group comment nodes associated with the training nodes, the action strategies output by the action nodes in the training nodes are analyzed to obtain the inter-group action scores; The training node is evaluated based on the action score and the inter-group action score to obtain the corresponding evaluation result.

8. The method according to claim 1, characterized in that, The step of optimizing the training nodes in the candidate network cluster based on the evaluation results until the candidate network cluster meets a preset termination condition to obtain the target action decision network includes: Based on the evaluation results, the dominant and disadvantageous nodes in the candidate network cluster are determined; Based on the first network parameters of the advantageous node, a replacement operation is performed on the second network parameters of the disadvantaged node; Monitor the training status of each training node in the candidate network cluster for each batch; Based on the training results, it is determined whether the candidate network cluster meets the preset termination condition. If the preset termination condition is met, the candidate network cluster is used as the target action decision network.

9. An optimization device for deep reinforcement learning decision-making, characterized in that, The device includes: The detection module is used to detect whether the candidate network cluster is in the policy evaluation state. The candidate network cluster includes multiple sets of training nodes, multiple inter-group comment nodes, policy evaluation nodes, and a common experience pool. Each set of training nodes includes n training nodes, where n≥1. The acquisition module is used to acquire the evaluation results corresponding to each training node output by the policy evaluation node if the policy evaluation state is in progress. The optimization module is used to optimize the training nodes in the candidate network cluster according to the evaluation results until the candidate network cluster meets the preset termination conditions to obtain the target action decision network. The determination module is used to determine the target action node in the target action decision network and take the action strategy corresponding to the target action node as the target action strategy.

10. A computer device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 8.

Images