Mechanical arm path planning method and device, storage medium and computer equipment

Abstract

The invention relates to the technical field of path planning, and provides a mechanical arm path planning method and device, a storage medium and computer equipment. The method comprises the steps that the current environment state of a mechanical arm is obtained; inputting the current environment state into the trained path planning network; wherein the path planning network comprises a double-actor network and a three-reviewer network; the current environment state is processed through the path planning network, and the execution action of the mechanical arm is output; wherein the execution action is determined after two candidate actions are generated by a double-actor network and are evaluated by a three-commentator network; and controlling the mechanical arm to move according to the execution action. According to the embodiment of the invention, the path planning network integrating the double-actor network and the three-commentator network is adopted, so that the robustness and decision quality of path planning of the mechanical arm in a complex environment are effectively improved, and the risk of decision failure caused by misjudgment of a single network is reduced; and the stability and reliability of the overall path planning behavior of the mechanical arm are improved.

Description

Technical Field

[0001] This disclosure relates to the field of path planning technology, and more specifically, to a method, apparatus, storage medium, and computer device for path planning of a robotic arm. Background Technology

[0002] Path planning for robotic arms is a core technology for autonomous robot movement, which is related to the task execution efficiency in industrial scenarios such as intelligent manufacturing, flexible assembly, and warehousing and logistics. With the development of industrial automation and intelligence, the adaptive planning capability of robotic arms in unstructured environments has become a research hotspot.

[0003] Currently, path planning methods in related technologies (such as artificial potential field methods) are often difficult to apply in complex environments due to high computational complexity and easy getting trapped in local optima; sampling-based methods (such as RRT) have the ability to explore high-dimensional spaces, but they suffer from unstable path quality and poor convergence.

[0004] To address the aforementioned issues, deep reinforcement learning-based methods have emerged. These methods, through an end-to-end learning mechanism, enable agents to autonomously learn optimal policies in a high-dimensional state space. Among them, the Soft Actor-Critic (SAC) algorithm exhibits good performance due to its entropy-regularized balance exploration and utilization. However, the standard SAC algorithm still has inherent drawbacks in practical applications: First, its dual-criticism and minimum-value-update mechanism, while suppressing Q-value overestimation, easily leads to a systematic Q-value underestimation bias, affecting the accuracy of policy optimization; second, its single-agent network structure limits policy diversity, resulting in insufficient exploration efficiency in complex environments and a tendency to converge to local optima. Summary of the Invention

[0005] This disclosure provides at least one robotic arm path planning method, apparatus, storage medium, and computer device. By employing a path planning network that integrates a dual-actor network and a three-commentator network, the robustness and decision quality of the robotic arm's path planning in complex environments are effectively improved. This reduces the risk of decision failure due to a single network's misjudgment and increases the stability and reliability of the robotic arm's overall path planning behavior.

[0006] This disclosure provides a robotic arm path planning method, including: The current environmental state of the robotic arm is obtained; and the current environmental state is input into a trained path planning network; wherein the path planning network includes a two-actor network and a three-commentator network; The path planning network processes the current environmental state and outputs the execution action of the robotic arm; wherein the execution action is generated by the two-actor network into two candidate actions, which are then evaluated by the three-critic network to determine the final action. The movement of the robotic arm is controlled according to the action to be performed.

[0007] In some possible embodiments, processing the current environmental state through the path planning network and outputting the robotic arm's execution action includes: Based on the dual-actor network and the current environmental state, a first candidate action and a second candidate action are generated. The first candidate action is evaluated using the three-critic network, and a first Q value is determined based on the evaluation result; and the second candidate action is evaluated using the three-critic network, and a second Q value is determined based on the evaluation result. The first Q value is compared with the second Q value, and the candidate action with the higher Q value is determined as the action to be executed.

[0008] In some possible embodiments, determining the first Q value based on the evaluation results includes: For the first candidate action, the Q-value estimate of each of the three critic networks for the first candidate action is determined based on the evaluation results; Determine the maximum value of the Q-value estimates of the first candidate action by the first critic network and the second critic network; Based on the Q-value estimates of the first candidate action by the first critic network, the second critic network, and the third critic network, the average value of the Q-value estimates of the first candidate action is calculated. The first Q value is obtained by weighting and summing the maximum value and the average value according to a preset weighting coefficient; The determination of the second Q value based on the evaluation results includes: For the second candidate action, the Q-value estimate of each of the three critic networks for the second candidate action is determined based on the evaluation results; Determine the maximum value of the Q-value estimates of the second candidate action by the first critic network and the second critic network; Based on the Q-value estimates of the second candidate action by the first critic network, the second critic network, and the third critic network, the average value of the Q-value estimates of the second candidate action is calculated; The second Q value is obtained by weighting the maximum value and the average value according to a preset weighting coefficient.

[0009] In some possible embodiments, the path planning network is trained through the following steps: Initialize the first actor network, the second actor network, three online critic networks, and three target critic networks respectively, and create an experience replay pool; Based on the first actor network, the second actor network, and the three online commentator networks, the robotic arm is controlled to perform actions in the simulation environment, collect experience tuples containing state, action, reward, and next state, and store them in the experience replay pool; When the amount of data in the experience replay pool reaches a preset threshold, a small batch of experience tuples is randomly sampled. Based on the sampled small batch of experience tuples, an alternating odd-even round cross-update strategy is used to update the first actor network, the second actor network, and the three online commentator networks respectively, until the reward value of the path planning network in the training rounds converges or reaches the preset maximum number of training rounds, thus obtaining the trained path planning network. Specifically, after each update of the three online critic networks, the three target critic networks are soft-updated using a soft-update method; wherein each target critic network only tracks parameter changes of the online critic network corresponding to the target critic network.

[0010] In some possible embodiments, the alternating odd-even round cross-updating strategy for updating the first actor network, the second actor network, and the three online commentator networks includes: In odd-numbered update rounds: The first actor network is used to generate a first next action based on the next state, and the second actor network is used to generate a second next action based on the next state; Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; Based on the target Q-value, the parameters of the three online critic networks are updated respectively; and the Q-value of the output action of the second actor network is evaluated based on the updated three online critic networks, and the parameters of the second actor network are updated according to the evaluation result. In even-numbered update rounds: The first actor network is used to generate a first next action based on the next state, and the second actor network is used to generate a second next action based on the next state; Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; Based on the target Q-value, the parameters of the three online commentator networks are updated respectively; and the Q-value of the output action of the first actor network is evaluated based on the updated three online commentator networks, and the parameters of the first actor network are updated according to the evaluation result.

[0011] In some possible embodiments, calculating the target Q-value based on the first next action, the second next action, and the three target critic networks includes: For the first next action, calculate the Q-value estimates of the first next action by the three target commentator networks, and take the minimum value as the first conservative Q-value estimate; For the second next action, calculate the Q-value estimates of the second next action by the three target commentator networks, and take the minimum value as the second conservative Q-value estimate; The maximum value of the first conservative Q-value estimate and the second conservative Q-value estimate is taken as the unified conservative Q-value estimate; The target Q-value is calculated based on the unified conservative Q-value estimate, reward, and discount factors.

[0012] In some possible embodiments, after controlling the movement of the robotic arm, the method further includes: Collect the current experience tuple generated after the robotic arm performs the action, and store the current experience tuple in the experience replay pool; wherein, the current experience tuple includes the current environment state, the action performed, the current action reward, and the next environment state; The path planning network is continuously updated online based on the experience tuples in the experience replay pool.

[0013] This disclosure provides a robotic arm path planning device, including: A state acquisition module is used to acquire the current environmental state of the robotic arm and input the current environmental state into a trained path planning network; wherein the path planning network includes a two-actor network and a three-commentator network; The path planning module is used to process the current environmental state through the path planning network and output the execution action of the robotic arm; wherein the execution action is generated by the two-actor network into two candidate actions, which are then evaluated by the three-critic network to determine the final action. The motion control module is used to control the movement of the robotic arm according to the executed action.

[0014] In some possible embodiments, the path planning module is specifically used for: Based on the dual-actor network and the current environmental state, a first candidate action and a second candidate action are generated. The first candidate action is evaluated using the three-critic network, and a first Q value is determined based on the evaluation result; and the second candidate action is evaluated using the three-critic network, and a second Q value is determined based on the evaluation result. The first Q value is compared with the second Q value, and the candidate action with the higher Q value is determined as the action to be executed.

[0015] In some possible embodiments, the path planning module is specifically used for: For the first candidate action, the Q-value estimate of each of the three critic networks for the first candidate action is determined based on the evaluation results; Determine the maximum value of the Q-value estimates of the first candidate action by the first critic network and the second critic network; Based on the Q-value estimates of the first candidate action by the first critic network, the second critic network, and the third critic network, the average value of the Q-value estimates of the first candidate action is calculated. The first Q value is obtained by weighting and summing the maximum value and the average value according to a preset weighting coefficient; The path planning module is specifically used for: For the second candidate action, the Q-value estimate of each of the three critic networks for the second candidate action is determined based on the evaluation results; Determine the maximum value of the Q-value estimates of the second candidate action by the first critic network and the second critic network; Based on the Q-value estimates of the second candidate action by the first critic network, the second critic network, and the third critic network, the average value of the Q-value estimates of the second candidate action is calculated; The second Q value is obtained by weighting the maximum value and the average value according to a preset weighting coefficient.

[0016] In some possible embodiments, the status acquisition module is further configured to: Initialize the first actor network, the second actor network, three online critic networks, and three target critic networks respectively, and create an experience replay pool; Based on the first actor network, the second actor network, and the three online commentator networks, the robotic arm is controlled to perform actions in the simulation environment, collect experience tuples containing state, action, reward, and next state, and store them in the experience replay pool; When the amount of data in the experience replay pool reaches a preset threshold, a small batch of experience tuples is randomly sampled. Based on the sampled small batch of experience tuples, an alternating odd-even round cross-update strategy is used to update the first actor network, the second actor network, and the three online commentator networks respectively, until the reward value of the path planning network in the training rounds converges or reaches the preset maximum number of training rounds, thus obtaining the trained path planning network. Specifically, after each update of the three online critic networks, the three target critic networks are soft-updated using a soft-update method; wherein each target critic network only tracks parameter changes of the online critic network corresponding to the target critic network.

[0017] In some possible embodiments, the status acquisition module is further configured to: In odd-numbered update rounds: The first actor network is used to generate a first next action based on the next state, and the second actor network is used to generate a second next action based on the next state; Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; Based on the target Q-value, the parameters of the three online critic networks are updated respectively; and the Q-value of the output action of the second actor network is evaluated based on the updated three online critic networks, and the parameters of the second actor network are updated according to the evaluation result. In even-numbered update rounds: The first actor network is used to generate a first next action based on the next state, and the second actor network is used to generate a second next action based on the next state; Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; Based on the target Q-value, the parameters of the three online commentator networks are updated respectively; and the Q-value of the output action of the first actor network is evaluated based on the updated three online commentator networks, and the parameters of the first actor network are updated according to the evaluation result.

[0018] In some possible embodiments, the status acquisition module is further configured to: For the first next action, calculate the Q-value estimates of the first next action by the three target commentator networks, and take the minimum value as the first conservative Q-value estimate; For the second next action, calculate the Q-value estimates of the second next action by the three target commentator networks, and take the minimum value as the second conservative Q-value estimate; The maximum value of the first conservative Q-value estimate and the second conservative Q-value estimate is taken as the unified conservative Q-value estimate; The target Q-value is calculated based on the unified conservative Q-value estimate, reward, and discount factors.

[0019] In some possible embodiments, the motion control module is further configured to: Collect the current experience tuple generated after the robotic arm performs the action, and store the current experience tuple in the experience replay pool; wherein, the current experience tuple includes the current environment state, the action performed, the current action reward, and the next environment state; The path planning network is continuously updated online based on the experience tuples in the experience replay pool.

[0020] This disclosure provides a computer device including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the robotic arm path planning method as described in any of the above possible embodiments.

[0021] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robotic arm path planning method as described in any of the possible embodiments above.

[0022] The robotic arm path planning method, apparatus, storage medium, and computer equipment provided in this disclosure specifically include a dual-actor network that can generate two different candidate actions in parallel for the same environmental state. This provides diverse choices from the policy source, enhances the agent's exploration capabilities, and helps to escape local optima, thereby planning a better movement path when facing complex or uncertain environments. Simultaneously, the three-critic network provides multiple value judgments for action evaluation. By independently evaluating and comparing the Q-values ​​of two candidate actions, it can integrate the perspectives of multiple value functions and select the action with the better evaluation result as the final execution instruction.

[0023] Thus, this disclosure adopts a path planning network that integrates a two-actor network and a three-commentator network, which effectively improves the robustness and decision quality of the robotic arm's path planning in complex environments, reduces the risk of decision failure due to a single network's judgment error, and increases the stability and reliability of the robotic arm's overall path planning behavior.

[0024] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings referenced in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a robotic arm path planning method provided by an embodiment of this disclosure is shown; Figure 2 A flowchart of a path planning network execution method provided by an embodiment of this disclosure is shown; Figure 3 A flowchart of a path planning network training method provided by an embodiment of this disclosure is shown; Figure 4 A flowchart illustrating a method for calculating a target Q value provided by an embodiment of this disclosure is shown; Figure 5 A flowchart of a path planning network continuous update method provided by an embodiment of this disclosure is shown; Figure 6 A schematic diagram of the structure of a robotic arm path planning device provided in an embodiment of this disclosure is shown; Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0028] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0029] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] As a core technology for autonomous robot motion, path planning for robotic arms directly impacts the efficiency of task execution in industrial scenarios such as intelligent manufacturing, flexible assembly, and warehousing logistics. With the continuous development of industrial production, the working environments faced by robotic arms are becoming increasingly complex and diverse, with dynamic, high-dimensional, or partially observable environments becoming common. In such complex environments, traditional path planning methods reveal many limitations. For example, traditional methods such as the artificial potential field method, due to their high computational complexity, require significant computational resources and time to handle large-scale, high-dimensional path planning problems, and are prone to getting trapped in local optima, making it difficult to find the globally optimal path. Therefore, they are unsuitable for dynamic, high-dimensional, or partially observable environments. Sampling-based methods, such as the Rapidly-exploring Random Trees (RRT) algorithm, while capable of exploring high-dimensional spaces, produce inconsistent path quality, sometimes generating tortuous and unreasonable paths, and exhibiting poor convergence, making it difficult to quickly obtain a path that meets the requirements. With the continuous improvement of industrial automation and intelligence, the adaptive planning ability of robotic arms in unstructured environments has gradually become a research hotspot. How to enable robotic arms to plan reasonable paths efficiently and stably in complex and ever-changing environments has become a key problem that urgently needs to be solved.

[0031] Research has shown that Deep Reinforcement Learning (DRL) provides a theoretically feasible solution for adaptive path planning of robotic arms in complex scenarios by constructing a closed-loop system of environmental interaction and policy optimization. DRL methods, with their end-to-end learning mechanism, enable agents to autonomously explore and learn optimal policies in a high-dimensional state space. However, DRL methods face two typical problems in practical deployment: firstly, bias in action-value function estimation, stemming from the accumulation of function approximation errors, which causes the policy gradient update direction to deviate from the true optimal solution; secondly, insufficient exploration efficiency, as the agent is prone to getting trapped in local optima due to insufficient randomness in the exploration policy. These problems are particularly pronounced in high-dimensional continuous control tasks such as robotic arm path planning, where the state space is complex and the action space is continuous. To improve exploration efficiency, Haarnoja et al. proposed the Soft Actor-Critic (SAC) algorithm, which enhances policy randomness by introducing an entropy regularization term, achieving a better balance between exploration and exploitation. However, the SAC algorithm still suffers from Q-value underestimation bias and limited policy diversity due to single-agent networks in practical applications, which restricts its performance in complex path planning tasks.

[0032] Based on the above research, this disclosure provides a robotic arm path planning method, apparatus, storage medium, and computer device. Specifically, the dual-actor network can generate two different candidate actions in parallel for the same environmental state, providing diverse choices from the policy source, enhancing the agent's exploration capabilities, and helping to escape local optima, thereby planning a better movement path when facing complex or uncertain environments. Simultaneously, the three-critic network provides multiple value judgments for action evaluation. By independently evaluating and comparing the Q-values ​​of the two candidate actions, it can integrate the perspectives of multiple value functions and select the action with the better evaluation result as the final execution instruction.

[0033] In this embodiment, a path planning network integrating a dual-actor network and a three-commentator network is adopted, which effectively improves the robustness and decision quality of the robotic arm's path planning in complex environments, reduces the risk of decision failure due to a single network's judgment error, and increases the stability and reliability of the robotic arm's overall path planning behavior.

[0034] To facilitate understanding of this embodiment, the executing entity of the robotic arm path planning method provided in this disclosure will first be described in detail. The executing entity of the robotic arm path planning method provided in this disclosure is a computer device. This computer device can be a terminal device or a server. The terminal device can also be a mobile device, a user terminal, a terminal, a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. Optionally, this method can also be applied to an implementation environment composed of computer devices and servers.

[0035] The robotic arm path planning method provided in this application embodiment will be described in detail below with reference to the accompanying drawings. See also: Figure 1 The diagram shows a flowchart of a robotic arm path planning method provided in an embodiment of this disclosure. The method includes the following steps S101 to S103: S101, Obtain the current environmental state of the robotic arm; and input the current environmental state into the trained path planning network.

[0036] Here, the current environmental state refers to the set of information perceived by the robotic arm from its working environment when performing path planning tasks. This information is usually digitally expressed in the form of vectors. For example, in a robotic arm grasping task, the current environmental state may include the real-time angles and angular velocities of the robotic arm's six joints, the position and orientation of the end effector in three-dimensional space, the three-dimensional coordinates of the target object relative to the robotic arm's base, and the position and size of obstacles in the environment.

[0037] Understandably, the path planning network is a neural network model trained using deep reinforcement learning algorithms. This disclosure presents an improved model based on the Soft Actor-Critic (SAC) algorithm, with its core architecture consisting of a two-actor network and a three-critic network. The actor network, also known as the policy network, determines the output action based on the input state information. In this disclosure, the two-actor network refers to two actor networks with identical structures but different parameters, working in parallel to generate action choices from different policy perspectives. The critic network, also known as the value network, evaluates the long-term value of performing an action in a specific state. The three-critic network refers to three critic networks with identical structures but different parameters, collectively providing value references for action selection.

[0038] Specifically, after obtaining the current environmental state of the robotic arm, it can be input into a trained path planning network to utilize the complex mapping relationships learned by the network to convert the high-dimensional environmental state into optimized action commands. The training steps for the path planning network will be discussed in detail later.

[0039] S102, the current environmental state is processed through the path planning network, and the execution action of the robotic arm is output.

[0040] Understandably, by processing the current environmental state through a path planning network, an execution action for controlling the robotic arm can be output. The determination of this action is divided into two stages: first, candidate actions are generated, and then they are evaluated and selected. Specifically, to ensure the diversity of action selection and the accuracy of evaluation, when processing the current environmental state through the path planning network, reference is made to... Figure 2 As shown, the steps S201~S203 may be included: S201, Based on the dual-actor network and the current environmental state, generate a first candidate action and a second candidate action.

[0041] Here, during the candidate action generation phase, the two actor networks simultaneously receive the same current environment state and can each output an action based on their learned policies. For example, in the same scenario of approaching a target, the first actor network might output a set of joint angle increment commands that cause the robotic arm's end effector to quickly approach the target along a straight trajectory, while the second actor network might output a set of joint angle increment commands that cause the robotic arm's end effector to move along a curved trajectory to avoid potential collision risks in advance. This results in two candidate actions with different policy orientations.

[0042] Candidate action generation can be represented as a process using two independent actor networks. and Two candidate actions are generated based on the current environment state s (the first candidate action) Second candidate action ) s represents the current environment state.

[0043] S202, the first candidate action is evaluated using the three-critic network, and a first Q value is determined based on the evaluation result; and the second candidate action is evaluated using the three-critic network, and a second Q value is determined based on the evaluation result.

[0044] Furthermore, the three-critic network comprises a first critic network, a second critic network, and a third critic network. During the evaluation and selection phase, all three critic networks are used simultaneously to evaluate the value of the two candidate actions. Specifically, the first candidate action is combined with the current state to form a first state-action pair, which is then input into each of the three critic networks, with each network outputting a Q-value estimate. Similarly, the second candidate action is combined with the current state to form a second state-action pair, which is also input into the three critic networks, resulting in another set of three Q-value estimates. Here, the six Q-value estimates for the two candidate actions can be expressed as: ; Where i represents the candidate action number; j represents the commentator network number; and the three Q values ​​for the first candidate action can be obtained. The three Q values ​​for the second candidate action are: .

[0045] Understandably, the Q-value output by each commentator network represents its prediction of the long-term cumulative reward for that state-action pair. Based on the Q-value evaluation results of each commentator network for the first and second candidate actions, the final Q-value (i.e., the first Q-value and the second Q-value) corresponding to each candidate action can be obtained.

[0046] For example, in order to balance the contradiction between exploration and utilization, and at the same time reduce the bias that may be caused by a single network evaluation, the following steps (1) to (2) may be included when determining the final Q value (i.e. the first Q value and the second Q value) corresponding to each candidate action based on the evaluation results: (1) For the first candidate action, determine the Q-value estimate of each commentator network in the three commentator networks for the first candidate action based on the evaluation results; determine the maximum value of the Q-value estimate of the first candidate action by the first commentator network and the second commentator network; calculate the average value of the Q-value estimate of the first candidate action based on the Q-value estimates of the first commentator network, the second commentator network and the third commentator network; and sum the maximum value and the average value according to a preset weighting coefficient to obtain the first Q-value. (2) For the second candidate action, based on the evaluation results, determine the Q-value estimate of each of the three commentator networks for the second candidate action; determine the maximum value of the Q-value estimate of the first commentator network and the second commentator network for the second candidate action; based on the Q-value estimates of the first commentator network, the second commentator network and the third commentator network for the second candidate action, calculate the average value of the Q-value estimate of the second candidate action; and sum the maximum value and the average value according to a preset weighting coefficient to obtain the second Q-value.

[0047] Understandably, for the first candidate action, based on the evaluation results, the Q-value estimates of each of the three commentator networks for the first candidate action can be determined. These three Q-value estimates come from three independent commentator networks, based on the same state-action pair but potentially offering different value assessments. Next, the maximum value of the Q-value estimates of the first and second commentator networks for the first candidate action is determined. This step aims to capture the highest value assessment from the two commentator networks, reflecting a preference for potentially high-reward actions. Simultaneously, based on the Q-value estimates of the first, second, and third commentator networks for the first candidate action, the average value of the Q-value estimates for the first candidate action is calculated. This average value reflects the collective judgment of the three commentator networks, helping to smooth out potential evaluation errors from individual networks. Finally, the maximum value and the average value are weighted and summed according to preset weighting coefficients to obtain the first Q-value. The preset weighting coefficients are mainly used to adjust the relative importance of the highest value assessment and the average value assessment in the final decision, and can be set according to actual needs, such as 0.2, 0.3, etc.

[0048] Similarly, the same calculation process is performed for the second candidate action to obtain the second Q value corresponding to the second candidate action, so as to ensure that the two candidate actions are compared under the same standard, thus ensuring the fairness and consistency of the evaluation.

[0049] The formulas for calculating the first Q value and the second Q value can be expressed as follows: ; ; In the formula, This represents the preset weighting coefficient, which can be set to 0.2.

[0050] S203, compare the first Q value with the second Q value, and determine the candidate action with the higher Q value as the action to be executed.

[0051] Understandably, after obtaining the comprehensive Q-value evaluations of two candidate actions, by comparing the first Q-value with the second Q-value, the candidate action with the higher comprehensive Q-value can be selected as the final action to be executed. Thus, this selection mechanism based on multiple value network evaluation and comparison can filter out actions that are recognized by the majority of the commentator network and have optimistic exploration potential, thereby improving the quality and robustness of the decision-making process.

[0052] S103, control the movement of the robotic arm according to the executed action.

[0053] Specifically, after obtaining the action based on the path planning network, the control system can convert it into specific low-level control signals. For example, if the action is a six-dimensional angle increment vector in the joint space, the control system will calculate the target position of each joint and drive the joint motors to rotate through position servo control, so that the end effector of the robotic arm moves to the target position according to the planned path to complete the predetermined operation task.

[0054] The following is combined with Figure 3 The training process of the path planning network proposed in this disclosure is described in detail. The training of the path planning network may include the following steps S301 to S304: S301 initializes the first actor network, the second actor network, three online critic networks, and three target critic networks, and creates an experience replay pool.

[0055] Understandably, initialization refers to setting initial values ​​for the weight and bias parameters of these neural networks. This typically employs random initialization strategies, such as Xavier or He initialization, to ensure that each network has different feature extraction capabilities at the start of training. The first and second actor networks are policy networks with the same network structure but independently initialized parameters; they will learn different policies for mapping environmental states to actions. The three online critic networks are the main networks used to evaluate the value of state-action pairs, updating their parameters in each training step. The three target critic networks are copies of the corresponding online critic networks, with slower parameter updates, primarily used to provide stable training objectives. The experience replay pool is a data storage structure, typically implemented using a circular queue or priority queue, used to store experience data generated by the agent's interactions with the environment.

[0056] S302, based on the first actor network, the second actor network and the three online commentator networks, control the robotic arm to perform actions in the simulation environment, collect experience tuples containing state, action, reward and next state, and store them in the experience replay pool.

[0057] Understandably, by controlling the robotic arm to perform actions in a simulated environment, the agent can systematically explore the environment and collect training data to achieve the subsequent training task of the path planning network. In each interaction, the current environmental state can be obtained first, and then simultaneously input into the first and second actor networks. Each actor network outputs a candidate action based on its current policy parameters, forming two different action choices. Subsequently, a specific action selection mechanism is employed, such as using three online critic networks to evaluate the Q-values ​​of these two candidate actions, and selecting the final action based on the weighted comparison result (refer to step S102 above). After the robotic arm executes the selected action, the simulation environment calculates the action execution result according to the physics engine and returns the corresponding reward signal and a new environmental state.

[0058] In this context, "state" refers to the environmental observations of the robotic arm at a specific moment, such as the angles of each joint, the position of the end effector, and the coordinates of the target object. "Action" refers to the control commands output by the actor network, such as the angle increments or torque values ​​of each joint. "Reward" is a scalar value returned by the environment based on the action's execution result, used to evaluate the quality of the action. For example, a positive reward is given when the robotic arm's end effector approaches the target object, and a negative reward is given when it collides with an obstacle. It may also include auxiliary rewards to encourage smooth movement and energy conservation. The next state refers to the new state the environment transitions to after the action is performed, and can contain all the updated observation information of the environment under the influence of the action. These elements together constitute a complete experience tuple, recording a complete interaction process between the agent and the environment. These experience tuples are then sequentially stored in the experience replay pool, providing a data foundation for subsequent network training.

[0059] S303: When the amount of data in the experience replay pool reaches a preset threshold, randomly sample a small batch of experience tuples.

[0060] Here, the preset threshold can typically be set to several thousand to tens of thousands of experiences to ensure that the experience replay pool has sufficiently diverse samples for learning. Random sampling refers to unbiasedly selecting a small batch of experiences from the experience replay pool; the batch size can be set to values ​​such as 64, 128, or 256. In this way, the random sampling mechanism can break the temporal correlation between experience data, helping to improve the stability and efficiency of training. At the same time, small-batch sampling can fully utilize the parallel processing capabilities of modern computing devices, accelerating the training process.

[0061] S304, based on the sampled mini-batch experience tuples, an alternating odd-even round cross-update strategy is used to update the first actor network, the second actor network, and the three online commentator networks respectively, until the reward value of the path planning network in the training rounds converges or reaches the preset maximum number of training rounds, thus obtaining the trained path planning network.

[0062] Understandingly, the cross-update strategy involves updating the second actor network after updating the three online critic networks in odd-numbered update rounds, and then updating the first actor network after updating the three online critic networks in even-numbered update rounds. This alternating update mechanism ensures that both actor strategies can participate in goal construction by rotating the facilitator, avoiding single-strategy dominance, effectively reducing the coupling dependency between the actor and critic networks, preventing overfitting, and thus improving training stability. This update process is repeated until preset termination conditions are met: first, the average reward value of the path planning network tends to stabilize over multiple consecutive training rounds, i.e., reward value convergence, indicating that the network has learned an effective strategy; second, the training reaches the preset maximum number of training rounds, a setting primarily used to control training duration and avoid indefinite training. When either condition is met, the training process terminates, and the trained path planning network is obtained.

[0063] Specifically, in odd-numbered update rounds, the following (a) to (c) can be included: (a) Using the first actor network to generate a first next action based on the next state, and using the second actor network to generate a second next action based on the next state; (b) Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; (c) Based on the target Q-value, update the parameters of the three online critic networks respectively; and, based on the updated three online critic networks, evaluate the Q-value of the output action of the second actor network, and update the parameters of the second actor network according to the evaluation result.

[0064] Understandably, the network parameter update process in odd-numbered rounds can include the following: First, a dual-actor network is used to perform parallel policy deduction for the next state. The first actor network generates a first next action based on its policy parameters, and the second actor network generates a second next action based on its policy parameters. These two actions represent future action choices from different policy perspectives. Then, three target critic networks perform Q-value evaluations on these two future actions. Each target critic network outputs a Q-value estimate for each action, reflecting each critic network's prediction of the long-term cumulative reward of the state-action pair. Based on these Q-value estimates, a target Q-value can be calculated, which will serve as the learning objective for updating the online critic networks. Next, based on the calculated target Q-value, the parameters of all three online critic networks are updated synchronously by minimizing the difference between the predicted Q-value and the target Q-value. Finally, using these newly updated online critic networks, the expected value of the second actor network's output action in the current state is re-evaluated, and the parameters of the second actor network are updated according to the evaluation results, optimizing its policy towards achieving a higher Q-value evaluation.

[0065] For example, refer to Figure 4 As shown, the calculation of the target Q value may include the following steps S401~S404: S401, for the first next action, calculate the Q-value estimates of the first next action by the three target commentator networks, and take the minimum value as the first conservative Q-value estimate.

[0066] Here, taking the minimum value as the first conservative Q-value estimate for the first next action helps to suppress Q-value overestimation and provides a more conservative value assessment.

[0067] S402, for the second next action, calculate the Q-value estimates of the second next action by the three target commentator networks, and take the minimum value as the second conservative Q-value estimate.

[0068] Similarly, for the second next action, taking the minimum value as the second conservative Q-value estimate helps to suppress Q-value overestimation and provide a more conservative value assessment.

[0069] S403, take the maximum value of the first conservative Q-value estimate and the second conservative Q-value estimate as the unified conservative Q-value estimate.

[0070] Understandably, in order to achieve cross-guidance between strategies, after obtaining the first conservative Q-value estimate and the second conservative Q-value estimate, the maximum value can be taken as the unified conservative Q-value estimate. At the same time, the corresponding action is taken as the winning action. In this way, the more promising value estimate in the two-actor network is preserved, which reflects the idea of ​​optimistic exploration.

[0071] S404, Calculate the target Q value based on the unified conservative Q value estimate, reward and discount factors.

[0072] Specifically, after obtaining a uniformly conservative Q-value estimate, the immediate reward can be combined with a discounted estimate of future value to calculate the target Q-value. The immediate reward is a scalar value directly obtained from the empirical tuple, representing the immediate return of the current state-action pair. The discount factor is a hyperparameter between 0 and 1, typically set to 0.99 or a similar value, used to adjust the importance of future returns in the current decision; a higher discount factor indicates a greater emphasis on long-term gains, while a lower discount factor focuses more on short-term gains. The uniformly conservative Q-value estimate is a comprehensive assessment of the value of future states obtained through the aforementioned steps. Thus, the target Q-value considers both the current immediate feedback and the carefully evaluated expectation of future returns, providing a stable and reliable learning target for parameter updates in the online commentator network.

[0073] The target Q value can be calculated using the following formula: ; In the formula, Represented as the target Q value in odd-numbered update rounds; This is indicated as a reward; It can be expressed as a discount factor; Q( s' To unify the conservative estimate, Represented as the winning actions of the first actor network and the second actor network. probability; This is represented as a hyperparameter.

[0074] Here, after obtaining the target Q value, the parameters of the three online critic networks can be optimized and updated by minimizing the mean squared error loss, which can be expressed as: ; in, Let be an experience tuple, s represent the current state, a represent the action to be performed, and r represent the reward. Indicates the next state; Let be the optimization objective function for the i-th online critic network.

[0075] For example, after updating the parameters of the three online critic networks, the Q-value of the winning action output by the second actor network is re-evaluated using the updated three online critic networks, and the parameters of the second actor network are updated based on the evaluation results. During the update process, this can be achieved by maximizing the weighted sum of its expected Q-value and entropy, which can be expressed as: ; In the formula, This is expressed as the objective function of the second actor network; Expressed as expectation; This represents the second-actor network choosing an action in state s. The logarithm of the probability; This represents the action of the first online critic network in state s after the update. Q-value estimation; This represents the actions of the updated second online critic network under state s. Q-value estimation; Represented as the updated state of the i-th online critic network pair. s Next action Q-value estimation.

[0076] Here, the parameter updates for each network can employ optimization methods such as gradient descent and gradient ascent; no specific limitations are specified. The parameters of the online commentator network are optimized by minimizing the loss function. The parameters of the actor network are updated by maximizing the objective function. Updates are performed. These optimization processes can be implemented using algorithms such as stochastic gradient descent and the Adam optimizer; the specific choice of optimization algorithm does not affect the implementation of this method.

[0077] Similarly, in even-numbered update rounds, the following (I) to (III) may be included: (I) Using the first actor network to generate a first next action based on the next state, and using the second actor network to generate a second next action based on the next state; (II) Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; (III) Based on the target Q value, update the parameters of the three online critic networks respectively; and, based on the updated three online critic networks, evaluate the Q value of the winning action output by the first actor network, and update the parameters of the first actor network according to the evaluation result.

[0078] Understandably, the overall process of updating network parameters in even-numbered rounds remains symmetrical to that in odd-numbered rounds, but the objects being updated differ. First, a parallel policy deduction for the next state is performed using a dual-actor network. The first and second actor networks generate the first and second next actions, respectively, representing future action choices from different policy perspectives. Then, three target critic networks evaluate the Q-values ​​of these two future actions, using the same computational process as in odd-numbered rounds to obtain the target Q-value, which serves as the learning objective for updating the online critic networks. Next, based on the calculated target Q-value, the parameters of all three online critic networks are updated synchronously by minimizing the mean squared error between the predicted Q-value and the target Q-value. Finally, using these newly updated online critic networks, the expected value of the winning action output by the first actor network is re-evaluated, and the parameters of the first actor network are updated according to the evaluation results, optimizing its policy towards achieving a higher Q-value evaluation.

[0079] Here, the formula for calculating the target Q value in even-numbered update rounds can be expressed as: ; In the formula, This is represented as the target Q value in even-numbered update rounds; This represents the probability value of the winning action output by the second actor network.

[0080] Similarly, the expression for updating the three online critic networks using the target Q-value in even-number update rounds can be represented as: ; Similarly, the expression for the first actor network update process can be represented as: ; In the formula, This is expressed as the objective function of the first actor network; Expressed as expectation; This represents the second-actor network choosing an action in state s. The logarithm of the probability; This represents the action of the first online critic network in state s after the update. Q-value estimation; This represents the actions of the updated second online critic network under state s. Q-value estimation; Let the action of the i-th online critic network in state s be represented by the updated action of the i-th online critic network. Q-value estimation.

[0081] For example, after each update of the three online critic networks, the three target critic networks can be soft-updated using a soft-update method. Each target critic network only tracks the parameter changes of its corresponding online critic network. Specifically, the soft-update method is a key technique in deep reinforcement learning used to maintain training stability. Its core principle is to slowly and gradually bring the parameters of the target critic network closer to the parameters of its corresponding online critic network, rather than directly copying them. This ensures that the parameters of the target critic network used to calculate the target Q-value do not undergo drastic changes, thus providing a relatively stable and reliable learning target for the online critic network training and effectively preventing divergence or non-convergence during training due to excessive fluctuations in the target value.

[0082] Here, the specific implementation process of soft update can be represented as follows: ; in, Let j be the target critic network; Let j be the j-th online critic network; This represents the soft update coefficient, which can be set to 0.995; the arrow indicates an assignment operation.

[0083] The principle that each target critic network tracks only the parameter changes of its corresponding online critic network means that a strict one-to-one correspondence exists during the network parameter update process. Specifically, the first target critic network tracks only the parameter changes of the first online critic network, the second target critic network tracks only the parameter changes of the second online critic network, and the third target critic network tracks only the parameter changes of the third online critic network. This one-to-one tracking mechanism ensures that each target critic network can independently and accurately reflect the learning progress of its paired online critic network, thus providing a diverse and consistent estimation basis when calculating the target Q-value. Through this combination of soft updates and corresponding tracking mechanisms, the algorithm can significantly improve the stability and convergence of the entire training process while maintaining the necessary exploratory capabilities.

[0084] The proposed alternating odd-even round update strategy ensures that both actor strategies participate in target construction by rotating the facilitator, avoiding single-strategy dominance. This effectively reduces the coupling dependency between the actor network and the critic network, preventing overfitting and thus improving the stability of the training process. In odd rounds, the first actor network participates in the target Q-value calculation and updates the second actor network; in even rounds, the second actor network participates in the target calculation and updates the first actor network, forming two decoupled optimization loops.

[0085] In some possible embodiments, to enhance the adaptability and robustness of the path planning system in actual deployment and to achieve continuous learning and performance optimization in non-stationary environments, after controlling the robotic arm to perform the actions output by the path planning network, referencing... Figure 5 As shown, it may also include the following steps S501~S502: S501, collect the current experience tuple generated after the robotic arm performs the execution action, and store the current experience tuple in the experience replay pool.

[0086] Specifically, after the robotic arm executes the action output by the path planning network, the complete data of this interaction can be recorded simultaneously, forming a new experience tuple. This experience tuple contains four elements: the current environmental state before the action, the actual action performed by the robotic arm, the current action reward from the environment, and the observed next environmental state after the action. These real-time collected experience tuples are added to the experience replay pool, forming the data foundation for the network's continuous learning together with the historical experience data stored during the training phase. Here, the experience replay pool can be managed using a first-in-first-out queue structure or a priority sampling mechanism to ensure the timeliness and diversity of the data.

[0087] S502, based on the experience tuples in the experience replay pool, the path planning network is continuously updated online.

[0088] Understandably, the path planning network can periodically or under specific conditions (such as when a certain amount of new data is added to the experience replay pool) randomly sample small batches of experience tuples from the experience replay pool. Then, using the same network update mechanism as the training process described above—including an alternating odd-even round cross-update strategy, calculation of the target Q-value based on the target commentator network, optimization of the mean squared error loss of the online commentator network, and a weighted sum of maximizing the expected Q-value and entropy of the actor network—all network parameters in the path planning network (including the first actor network, the second actor network, and the three online commentator networks) are updated. Simultaneously, after each update of the online commentator network, the corresponding target commentator network parameters are also updated synchronously using a soft update method.

[0089] In this way, the continuously updated online mechanism disclosed herein can train the path planning system to overcome the limitations of the initial training environment. By continuously learning from new interactive experiences, it adaptively adjusts its policy network and value network. When the working environment changes, such as the introduction of new obstacles, changes in the dynamic characteristics of the robotic arm, or updates to task requirements, it can autonomously optimize its decision-making strategy, gradually improving its path planning performance in real and complex environments. Thus, a seamless transition from "offline training" to "online learning" is achieved, forming a complete lifelong learning system architecture. This effectively solves the performance degradation problem of traditional deep reinforcement learning methods when the environment changes, significantly improving the system's reliability and adaptability during long-term operation.

[0090] The robotic arm path planning method, apparatus, storage medium, and computer equipment provided in this disclosure adopt a path planning network that integrates a two-actor network and a three-commentator network. This effectively improves the robustness and decision quality of the robotic arm's path planning in complex environments, reduces the risk of decision failure due to a single network's judgment error, and increases the stability and reliability of the robotic arm's overall path planning behavior.

[0091] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0092] Based on the same inventive concept, this disclosure also provides a robotic arm path planning device corresponding to the robotic arm path planning method. Since the principle of the device in this disclosure for solving the problem is similar to the robotic arm path planning method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0093] Reference Figure 6 The diagram shown is a schematic of a robotic arm path planning device 600 provided in an embodiment of this disclosure. The device includes: The state acquisition module 601 is used to acquire the current environmental state of the robotic arm and input the current environmental state into the trained path planning network; wherein, the path planning network includes a two-actor network and a three-commentator network; The path planning module 602 is used to process the current environmental state through the path planning network and output the execution action of the robotic arm; wherein the execution action is generated by the two-actor network into two candidate actions, and determined after evaluation by the three-critic network; The motion control module 603 is used to control the movement of the robotic arm according to the executed action.

[0094] In some possible embodiments, the path planning module 602 is specifically used for: Based on the dual-actor network and the current environmental state, a first candidate action and a second candidate action are generated. The first candidate action is evaluated using the three-critic network, and a first Q value is determined based on the evaluation result; and the second candidate action is evaluated using the three-critic network, and a second Q value is determined based on the evaluation result. The first Q value is compared with the second Q value, and the candidate action with the higher Q value is determined as the action to be executed.

[0095] In some possible embodiments, the path planning module 602 is specifically used for: For the first candidate action, the Q-value estimate of each of the three critic networks for the first candidate action is determined based on the evaluation results; Determine the maximum value of the Q-value estimates of the first candidate action by the first critic network and the second critic network; Based on the Q-value estimates of the first candidate action by the first critic network, the second critic network, and the third critic network, the average value of the Q-value estimates of the first candidate action is calculated. The first Q value is obtained by weighting and summing the maximum value and the average value according to a preset weighting coefficient; The path planning module 602 is specifically used for: For the second candidate action, the Q-value estimate of each of the three critic networks for the second candidate action is determined based on the evaluation results; Determine the maximum value of the Q-value estimates of the second candidate action by the first critic network and the second critic network; Based on the Q-value estimates of the second candidate action by the first critic network, the second critic network, and the third critic network, the average value of the Q-value estimates of the second candidate action is calculated; The second Q value is obtained by weighting the maximum value and the average value according to a preset weighting coefficient.

[0096] In some possible embodiments, the status acquisition module 601 is further configured to: Initialize the first actor network, the second actor network, three online critic networks, and three target critic networks respectively, and create an experience replay pool; Based on the first actor network, the second actor network, and the three online commentator networks, the robotic arm is controlled to perform actions in the simulation environment, collect experience tuples containing state, action, reward, and next state, and store them in the experience replay pool; When the amount of data in the experience replay pool reaches a preset threshold, a small batch of experience tuples is randomly sampled. Based on the sampled small batch of experience tuples, an alternating odd-even round cross-update strategy is used to update the first actor network, the second actor network, and the three online commentator networks respectively, until the reward value of the path planning network in the training rounds converges or reaches the preset maximum number of training rounds, thus obtaining the trained path planning network. Specifically, after each update of the three online critic networks, the three target critic networks are soft-updated using a soft-update method; wherein each target critic network only tracks parameter changes of the online critic network corresponding to the target critic network.

[0097] In some possible embodiments, the status acquisition module 601 is further configured to: In odd-numbered update rounds: The first actor network is used to generate a first next action based on the next state, and the second actor network is used to generate a second next action based on the next state; Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; Based on the target Q-value, the parameters of the three online critic networks are updated respectively; and the Q-value of the output action of the second actor network is evaluated based on the updated three online critic networks, and the parameters of the second actor network are updated according to the evaluation result. In even-numbered update rounds: The first actor network is used to generate a first next action based on the next state, and the second actor network is used to generate a second next action based on the next state; Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; Based on the target Q-value, the parameters of the three online commentator networks are updated respectively; and the Q-value of the output action of the first actor network is evaluated based on the updated three online commentator networks, and the parameters of the first actor network are updated according to the evaluation result.

[0098] In some possible embodiments, the status acquisition module 601 is further configured to: For the first next action, calculate the Q-value estimates of the first next action by the three target commentator networks, and take the minimum value as the first conservative Q-value estimate; For the second next action, calculate the Q-value estimates of the second next action by the three target commentator networks, and take the minimum value as the second conservative Q-value estimate; The maximum value of the first conservative Q-value estimate and the second conservative Q-value estimate is taken as the unified conservative Q-value estimate; The target Q-value is calculated based on the unified conservative Q-value estimate, reward, and discount factors.

[0099] In some possible embodiments, the motion control module 603 is further configured to: Collect the current experience tuple generated after the robotic arm performs the action, and store the current experience tuple in the experience replay pool; wherein, the current experience tuple includes the current environment state, the action performed, the current action reward, and the next environment state; The path planning network is continuously updated online based on the experience tuples in the experience replay pool.

[0100] Based on the same technical concept, this disclosure also provides a computer device. (See also...) Figure 7 The diagram shows the structure of a computer device 700 provided in this embodiment of the present disclosure, including a processor 701, a memory 702, and a bus 703. The memory 702 stores execution instructions and includes a main memory 7021 and an external memory 7022. The main memory 7021, also called internal memory, is used to temporarily store computational data in the processor 701, as well as data exchanged with external memory 7022 such as a hard disk. The processor 701 exchanges data with the external memory 7022 through the main memory 7021.

[0101] In this embodiment, the memory 702 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 701. That is, when the computer device 700 is running, the processor 701 communicates with the memory 702 through the bus 703, so that the processor 701 executes the application code stored in the memory 702, and then executes the method described in any of the foregoing embodiments.

[0102] The memory 702 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0103] Processor 701 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0104] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the computer device 700. In other embodiments of this application, the computer device 700 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0105] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robotic arm path planning method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0106] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the robotic arm path planning method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0107] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A path planning method for a robotic arm, characterized in that, include: Obtain the current environmental state of the robotic arm; The current environment state is then input into the trained path planning network; wherein the path planning network includes a two-actor network and a three-commentator network; The path planning network processes the current environmental state and outputs the execution action of the robotic arm; wherein the execution action is generated by the two-actor network into two candidate actions, which are then evaluated by the three-critic network to determine the final action. The movement of the robotic arm is controlled according to the action to be performed.

2. The method according to claim 1, characterized in that, The process of processing the current environmental state through the path planning network and outputting the execution action of the robotic arm includes: Based on the dual-actor network and the current environmental state, a first candidate action and a second candidate action are generated. The first candidate action is evaluated using the three-critic network, and a first Q value is determined based on the evaluation result; and the second candidate action is evaluated using the three-critic network, and a second Q value is determined based on the evaluation result. The first Q value is compared with the second Q value, and the candidate action with the higher Q value is determined as the action to be executed.

3. The method according to claim 2, characterized in that, The three critic networks include a first critic network, a second critic network, and a third critic network; The determination of the first Q value based on the evaluation results includes: For the first candidate action, the Q-value estimate of each of the three critic networks for the first candidate action is determined based on the evaluation results; Determine the maximum value of the Q-value estimates of the first candidate action by the first critic network and the second critic network; Based on the Q-value estimates of the first candidate action by the first critic network, the second critic network, and the third critic network, the average value of the Q-value estimates of the first candidate action is calculated. The first Q value is obtained by weighting and summing the maximum value and the average value according to a preset weighting coefficient; The determination of the second Q value based on the evaluation results includes: For the second candidate action, the Q-value estimate of each of the three critic networks for the second candidate action is determined based on the evaluation results; Determine the maximum value of the Q-value estimates of the second candidate action by the first critic network and the second critic network; Based on the Q-value estimates of the second candidate action by the first critic network, the second critic network, and the third critic network, the average value of the Q-value estimates of the second candidate action is calculated; The second Q value is obtained by weighting the maximum value and the average value according to a preset weighting coefficient.

4. The method according to claim 2, characterized in that, The path planning network is trained through the following steps: Initialize the first actor network, the second actor network, three online critic networks, and three target critic networks respectively, and create an experience replay pool; Based on the first actor network, the second actor network, and the three online commentator networks, the robotic arm is controlled to perform actions in the simulation environment, collect experience tuples containing state, action, reward, and next state, and store them in the experience replay pool; When the amount of data in the experience replay pool reaches a preset threshold, a small batch of experience tuples is randomly sampled. Based on the sampled small batch of experience tuples, an alternating odd-even round cross-update strategy is used to update the first actor network, the second actor network, and the three online commentator networks respectively, until the reward value of the path planning network in the training rounds converges or reaches the preset maximum number of training rounds, thus obtaining the trained path planning network. Specifically, after each update of the three online critic networks, the three target critic networks are soft-updated using a soft-update method; wherein each target critic network only tracks parameter changes of the online critic network corresponding to the target critic network.

5. The method according to claim 4, characterized in that, The method employs an alternating odd-even round cross-update strategy to update the first actor network, the second actor network, and the three online commentator networks, respectively, including: In odd-numbered update rounds: The first actor network is used to generate a first next action based on the next state, and the second actor network is used to generate a second next action based on the next state; Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; Based on the target Q-value, the parameters of the three online critic networks are updated respectively; and the Q-value of the output action of the second actor network is evaluated based on the updated three online critic networks, and the parameters of the second actor network are updated according to the evaluation result. In even-numbered update rounds: The first actor network is used to generate a first next action based on the next state, and the second actor network is used to generate a second next action based on the next state; Calculate the target Q-value based on the first next action, the second next action, and the three target critic networks; Based on the target Q-value, the parameters of the three online commentator networks are updated respectively; and the Q-value of the output action of the first actor network is evaluated based on the updated three online commentator networks, and the parameters of the first actor network are updated according to the evaluation result.

6. The method according to claim 5, characterized in that, The calculation of the target Q-value based on the first next action, the second next action, and the three target critic networks includes: For the first next action, calculate the Q-value estimates of the first next action by the three target commentator networks, and take the minimum value as the first conservative Q-value estimate; For the second next action, calculate the Q-value estimates of the second next action by the three target commentator networks, and take the minimum value as the second conservative Q-value estimate; The maximum value of the first conservative Q-value estimate and the second conservative Q-value estimate is taken as the unified conservative Q-value estimate; The target Q-value is calculated based on the unified conservative Q-value estimate, reward, and discount factors.

7. The method according to claim 6, characterized in that, After controlling the movement of the robotic arm, the method further includes: Collect the current experience tuple generated after the robotic arm performs the action, and store the current experience tuple in the experience replay pool; wherein, the current experience tuple includes the current environment state, the action performed, the current action reward, and the next environment state; The path planning network is continuously updated online based on the experience tuples in the experience replay pool.

8. A robotic arm path planning device, characterized in that, include: The status acquisition module is used to acquire the current environmental status of the robotic arm; The current environment state is then input into the trained path planning network; wherein the path planning network includes a two-actor network and a three-commentator network; The path planning module is used to process the current environmental state through the path planning network and output the execution action of the robotic arm; wherein the execution action is generated by the two-actor network into two candidate actions, which are then evaluated by the three-critic network to determine the final action. The motion control module is used to control the movement of the robotic arm according to the executed action.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images