Multi-manipulator motion planning reinforcement learning training method based on structured action space, manipulator motion planning method

Abstract

The present invention provides a multi-manipulator motion planning reinforcement learning training method based on structured action space and a manipulator motion planning method. The multi-manipulator motion planning reinforcement learning training method based on structured action space includes: obtaining sample state postures of the multi-manipulators and task target coordinates corresponding to sample tasks required to be performed by the multi-manipulators; inputting the state postures and task target coordinates into a high-level learning module of a preset hierarchical reinforcement model to obtain discrete joint action values; inputting the state postures, task target coordinates and discrete joint action values ​​into a low-level learning module of a preset hierarchical reinforcement model to obtain continuous joint action values; adjusting the postures of the multi-manipulators based on the discrete joint action values ​​and the continuous joint action values ​​to obtain the adjusted postures of the multi-manipulators; determining reward values ​​based on the adjusted postures of the multi-manipulators; adjusting the low-level learning modules of the preset hierarchical reinforcement model based on the reward values ​​to obtain a trained hierarchical reinforcement model.

Description

Technical Field

[0001] The present invention relates to the technical field of robot motion planning, and more specifically to a multi-robot arm motion planning reinforcement learning training method and a robot arm motion planning method based on structured action space. Background Art

[0002] Robotic arms play an important role in industrial automation, healthcare services, space exploration and other fields. Usually, the working scene of a robotic arm is not a free space, but a constrained space with obstacles. Deep reinforcement learning has demonstrated its ability to learn complex data patterns in a range of decision-making areas, so it can be used to plan the control of multiple robotic arms.

[0003] Among related technologies, deep multi-agent reinforcement learning has been criticized for its low sample efficiency. Even for relatively simple problem settings, millions of interactions are usually required. In the control planning application of multiple robotic arms, the output is continuous joint movements. Although the movement accuracy is high, it takes a long time to find the corresponding feasible solution, and the efficiency is poor. Summary of the invention

[0004] In view of the above problems, the present invention provides a multi-manipulator motion planning reinforcement learning training method and a manipulator motion planning method based on structured action space.

[0005] According to a first aspect of the present invention, a multi-manipulator motion planning reinforcement learning training method based on a structured action space is provided, including: obtaining sample state postures of the multi-manipulator and task target coordinates corresponding to sample tasks that the multi-manipulator needs to perform, the above-mentioned task target coordinates representing the target placement position after the multi-manipulator grasps the target object or the target grasping position of the multi-manipulator grasping the target object; inputting the above-mentioned state postures and the above-mentioned task target coordinates into a high-level learning module of a preset hierarchical reinforcement model to obtain discrete joint action values ​​of the above-mentioned multi-manipulator, and the above-mentioned discrete joint action values ​​are in a discrete joint action value group. The preset value of the discrete joint action value group represents multiple preset action angles of the multi-manipulator in the structured action space; the state posture, the task target coordinates and the discrete joint action values ​​are input into the low-level learning module of the preset hierarchical reinforcement model to obtain the continuous joint action values ​​of the multi-manipulator; the posture of the multi-manipulator is adjusted based on the discrete joint action values ​​and the continuous joint action values ​​to obtain the adjusted posture of the multi-manipulator; the reward value is determined based on the adjusted posture of the multi-manipulator; the low-level learning module of the preset hierarchical reinforcement model is adjusted based on the reward value to obtain a trained hierarchical reinforcement model.

[0006] According to an embodiment of the present invention, the above method also includes: using a rapidly expanding random tree algorithm to calculate the robot arm joint path points based on the state posture of the above-mentioned multiple robot arms; extracting the discrete joint calculation action values ​​of the multiple robot arms from the above-mentioned robot arm joint path points; obtaining a loss value based on the above-mentioned discrete joint action values ​​and the above-mentioned discrete joint calculation action values; and adjusting the high-level learning module of the above-mentioned preset hierarchical reinforcement model based on the above-mentioned loss value.

[0007] According to an embodiment of the present invention, the reward value is determined based on the adjusted posture of the above-mentioned multiple robotic arms, including: in the event that no collision occurs between the robotic arms during the process of adjusting the posture of the above-mentioned multiple robotic arms, a reward is given for the above-mentioned adjusted posture; in the event that a collision occurs between the robotic arms during the process of adjusting the posture of the above-mentioned multiple robotic arms, a negative reward is given for the above-mentioned adjusted posture.

[0008] According to an embodiment of the present invention, the above method also includes: rewarding the above-mentioned adjusted posture when the robot arm reaches the task target coordinates within a preset time; rewarding the above-mentioned adjusted posture when the robot arm end direction coordinates of the above-mentioned adjusted posture are closer to the above-mentioned task target coordinates than the robot arm end direction coordinates of the above-mentioned state posture.

[0009] According to an embodiment of the present invention, the above-mentioned reward for the above-mentioned adjusted posture includes: constructing a state transition reward value based on the above-mentioned state posture, the above-mentioned adjusted posture, the above-mentioned discrete joint action value and the above-mentioned continuous joint action value; and placing the above-mentioned state transition reward value into an experience recovery pool.

[0010] According to an embodiment of the present invention, the above-mentioned low-level learning module includes a policy network and an evaluation network. The above-mentioned low-level learning module that adjusts the above-mentioned preset hierarchical reinforcement model based on the above-mentioned reward value includes: calculating the policy update formula based on the multiple state transition reward values ​​in the above-mentioned experience recovery pool; updating the above-mentioned policy network based on the above-mentioned policy update formula; updating the above-mentioned evaluation network based on the above-mentioned multiple state transition reward values.

[0011] The second aspect of the present invention provides a method for robot arm motion planning, including: obtaining the target state posture of multiple robot arms and the target coordinates corresponding to the target tasks to be performed by the above-mentioned multiple robot arms; inputting the above-mentioned target state posture and the above-mentioned target coordinates into a hierarchical reinforcement model to obtain motion planning instructions, and the above-mentioned hierarchical reinforcement model is obtained based on the method of the above-mentioned first aspect; performing motion planning for the above-mentioned multiple robot arms based on the above-mentioned motion planning instructions.

[0012] The third aspect of the present invention provides a multi-manipulator motion planning reinforcement learning training device based on a structured action space, including: a first acquisition module, used to obtain sample state postures of the multi-manipulator and task target coordinates corresponding to sample tasks that need to be performed by the multi-manipulator, the above-mentioned task target coordinates represent the target placement position after the multi-manipulator grasps the target object or the target grasping position of the multi-manipulator grasping the target object; a discrete joint action value module, used to input the above-mentioned state posture and the above-mentioned task target coordinates into the high-level learning module of the preset hierarchical reinforcement model to obtain the discrete joint action values ​​of the above-mentioned multi-manipulator, the above-mentioned discrete joint action values ​​are preset values ​​in the discrete joint action value group, and the above-mentioned discrete joint action values ​​are preset values ​​in the discrete joint action value group. The joint action value group represents multiple preset action angles of the above-mentioned multi-manipulator in the above-mentioned structured action space; the continuous joint action value module is used to input the above-mentioned state posture, the above-mentioned task target coordinates and the above-mentioned discrete joint action values ​​into the low-level learning module of the preset hierarchical reinforcement model to obtain the continuous joint action values ​​of the above-mentioned multi-manipulator; the posture adjustment module is used to adjust the posture of the above-mentioned multi-manipulator based on the above-mentioned discrete joint action values ​​and the above-mentioned continuous joint action values ​​to obtain the adjusted posture of the multi-manipulator; the determination module is used to determine the reward value based on the adjusted posture of the above-mentioned multi-manipulator; the model adjustment module is used to adjust the low-level learning module of the above-mentioned preset hierarchical reinforcement model based on the above-mentioned reward value to obtain a trained hierarchical reinforcement model.

[0013] The fourth aspect of the present invention provides a robotic arm motion planning device, comprising: a second acquisition module, used to acquire the target state posture of multiple robotic arms and the target coordinates corresponding to the target tasks to be performed by the above-mentioned multiple robotic arms; an input module, used to input the above-mentioned target state posture and the above-mentioned target coordinates into a hierarchical reinforcement model to obtain motion planning instructions, and the above-mentioned hierarchical reinforcement model is obtained based on the method of the above-mentioned first aspect; a motion planning module, used to perform motion planning for the above-mentioned multiple robotic arms based on the above-mentioned motion planning instructions.

[0014] A fifth aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] The sixth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0016] The seventh aspect of the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0017] According to an embodiment of the present invention, by splitting the joint movements of the robotic arm into discrete joint movements and continuous joint movements for hierarchical calculation, the characteristics of high exploration efficiency and fast convergence speed of discrete movements can be fully utilized to quickly obtain discrete joint movement values ​​through high-level learning modules. Through low-level learning modules, iteration is performed on the basis of discrete joint movement values ​​to quickly obtain corresponding continuous joint movement values, thereby improving movement accuracy. This greatly improves training efficiency while ensuring movement accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0019] Figure 1 An application scenario diagram of a multi-robotic arm motion planning reinforcement learning training method and device based on a structured action space according to an embodiment of the present invention is shown.

[0020] Figure 2 A diagram showing an application scenario of a robot arm motion planning method and device according to an embodiment of the present invention is shown.

[0021] Figure 3 A flowchart of a multi-manipulator motion planning reinforcement learning training method based on a structured action space according to an embodiment of the present invention is shown.

[0022] Figure 4 A schematic diagram of the continuous action space is shown.

[0023] Figure 5 A structured hybrid action space according to an embodiment of the present invention is shown.

[0024] Figure 6 A schematic diagram of the structure of a hierarchical reinforcement learning architecture according to an embodiment of the present invention is shown.

[0025] Figure 7 A training framework diagram of multi-robotic arm motion planning reinforcement learning training based on structured action space according to an embodiment of the present invention is shown.

[0026] Figure 8 A reward value curve diagram according to an embodiment of the present invention is shown.

[0027] Fig. 9 A schematic diagram of reward function comparison according to an embodiment of the present invention is shown.

[0028] Fig.10 A comparison chart of ablation experiments between expert data and hierarchical reinforcement learning according to an embodiment of the present invention is shown.

[0029] Fig.11 A discrete action interval comparison graph according to an embodiment of the present invention is shown.

[0030] Fig.12 A flow chart of a robot arm motion planning method according to an embodiment of the present invention is shown.

[0031] Fig.13 A structural block diagram of a multi-manipulator motion planning reinforcement learning training device based on structured action space according to an embodiment of the present invention is shown.

[0032] Fig.14 A structural block diagram of a robot arm motion planning device according to an embodiment of the present invention is shown.

[0033] Fig.15 A block diagram of an electronic device suitable for implementing a multi-manipulator motion planning reinforcement learning training method based on a structured action space according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0034] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0035] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0036] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0037] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0038] Robotic arms play a key role in industrial automation, healthcare services, space exploration, and other fields. Typically, the working scene of a robotic arm is not a free space, but a constrained space with obstacles. In most industrial environments, multiple robotic arms usually need to work in the same workspace, and the moving robotic arms are contained in dynamic obstacles. The main goal of motion planning for each robotic arm is to derive a trajectory that avoids collisions with dynamic obstacles such as other arms. However, due to the complex degrees of freedom, non-fixed base positions, and tightly coupled workspaces of these multi-arm systems, effective motion planning remains an ongoing challenge. Therefore, it is of great significance to study efficient obstacle avoidance motion planning for multi-robotic arms.

[0039] Traditional motion planning algorithms consist of four categories, including graph search algorithms, sampling-based algorithms such as fast exploration random trees, interpolation curve algorithms (such as straight lines and circles), and reaction-based algorithms. However, these algorithms are difficult to meet both success rate and efficiency.

[0040] In recent years, driven by the rapid development of artificial intelligence, deep reinforcement learning has demonstrated its ability to learn complex data patterns in a range of decision-making fields, including computer vision, games, robotic arm collaboration, and motion control. In this context, motion planning is conceptualized as a sequential decision problem, and a neural network is used as an approximation of the decision function. This motion planning algorithm takes the real-time state of the robotic arm as the input of the decision function and outputs the corresponding action selection.

[0041] Although deep reinforcement learning methods have been gradually applied to control planning applications of multi-robot arms, deep reinforcement learning and deep multi-agent reinforcement learning methods have been criticized for their low sample efficiency, and even for relatively simple problem settings, millions of interactions are usually required. This inefficiency hinders their widespread adoption in real-world industrial applications. A major obstacle is the challenge of exploration, especially how to explore the environment and collect valuable experience to enhance policy learning and seek the best solution.

[0042] In view of this, an embodiment of the present invention provides a multi-manipulator motion planning reinforcement learning training method based on a structured action space, the method comprising: obtaining sample state postures of the multi-manipulator and task target coordinates corresponding to sample tasks that the multi-manipulator needs to perform, the task target coordinates representing the target placement position after the multi-manipulator grasps the target object or the target grasping position of the multi-manipulator grasping the target object; inputting the sample state postures and task target coordinates into a high-level learning module of a preset hierarchical reinforcement model to obtain discrete joint action values ​​of the multi-manipulator, the discrete joint action values ​​are preset values ​​in a discrete joint action value group, and the discrete joint action value group represents multiple preset action angles of the multi-manipulator in the structured action space; inputting the sample state postures, task target coordinates and discrete joint action values ​​into a low-level learning module of a preset hierarchical reinforcement model to obtain continuous joint action values ​​of the multi-manipulator; adjusting the posture of the multi-manipulator based on the discrete joint action values ​​and the continuous joint action values ​​to obtain the adjusted posture of the multi-manipulator; determining a reward value based on the adjusted posture of the multi-manipulator; adjusting the low-level learning module of the preset hierarchical reinforcement model based on the reward value to obtain a trained hierarchical reinforcement model.

[0043] Figure 1 An application scenario diagram of a multi-robotic arm motion planning reinforcement learning training method and device based on a structured action space according to an embodiment of the present invention is shown.

[0044] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105, and the network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0045] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0046] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. A simulation environment of multiple robotic arms can be established through the rude engines on the first terminal device 101, the second terminal device 102, and the third terminal device 103, and training and testing of multi-robotic arm motion planning can be performed through the simulation environment.

[0047] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0048] It should be noted that the multi-manipulator motion planning reinforcement learning training method based on structured action space provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the multi-manipulator motion planning reinforcement learning training device based on structured action space provided in the embodiment of the present invention can generally be set in the server 105. The multi-manipulator motion planning reinforcement learning training method based on structured action space provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the multi-manipulator motion planning reinforcement learning training device based on structured action space provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0049] It should be understood that Figure 1 The number of the first terminal device, the second terminal device, the third terminal device, the network and the server in the embodiment is only for illustration. According to the implementation requirements, there may be any number of the first terminal device, the second terminal device, the third terminal device, the network and the server.

[0050] Figure 2 A diagram showing an application scenario of a robot arm motion planning method and device according to an embodiment of the present invention is shown.

[0051] like Figure 2As shown, the application scenario 200 according to this embodiment may include a first robotic arm 201, a second robotic arm 202, a third robotic arm 203, a first target object 204, a second target object 205, a third target object 206, and a target object placement container 207. The first robotic arm 201 may be used to grasp the first target object 204, the second robotic arm 202 may be used to grasp the second target object 205, and the third robotic arm 203 may be used to grasp the third target object 206. The target object placement container 207 is used to place the target object.

[0052] The first robot arm 201, the second robot arm 202, and the third robot arm 203 may be, for example, a UR5 multi-axis robot arm with six degrees of freedom, but the present invention does not limit this.

[0053] It should be understood that Figure 2 The number of the mechanical arms, target objects and target object placement containers in the embodiment is only illustrative. According to the implementation requirements, there can be any number of mechanical arms, target objects and target object placement containers.

[0054] The following will be based on Figure 1 The scene described by Figure 3~Figure 11 The multi-manipulator motion planning reinforcement learning training method based on structured action space in an embodiment of the present invention is described in detail.

[0055] Figure 3 A flowchart of a multi-manipulator motion planning reinforcement learning training method based on a structured action space according to an embodiment of the present invention is shown.

[0056] like Figure 3 As shown, the multi-robotic arm motion planning reinforcement learning training method based on structured action space of this embodiment includes operations S310 to S360.

[0057] In operation S310 , sample state poses of the multi-manipulator and task target coordinates corresponding to sample tasks that the multi-manipulator needs to perform are obtained.

[0058] The task target coordinates represent the target placement position after the multi-manipulator grasps the target object or the target grasping position of the multi-manipulator grasping the target object.

[0059] According to an embodiment of the present invention, the above-mentioned sample state posture includes a base posture and an initial posture. A target joint configuration that will not collide can be selected for each robotic arm, and the target end effector posture coordinates of the task can be calculated through the target joint configuration of forward kinematics. The generated robotic arm initial posture, base position and target end effector posture coordinates are used as sample data as a training data set.

[0060] According to an embodiment of the present invention, a physical engine may be used to construct a simulation environment, and a corresponding six-degree-of-freedom UR5 multi-axis robotic arm may be used to perform simulation processing, thereby training the model.

[0061] In operation S320, the sample state pose and task target coordinates are input into a high-level learning module of a preset hierarchical reinforcement model to obtain discrete joint action values ​​of the multi-manipulator.

[0062] The discrete joint action values ​​are preset values ​​in a discrete joint action value group, and the discrete joint action value group represents multiple preset action angles of the multi-manipulator in a structured action space.

[0063] Figure 4 A schematic diagram of the continuous action space is shown.

[0064] like Figure 4 As shown in the figure, the action space of each joint of the six-axis robot is [-180°, +180°], where the solid line represents the discrete action distribution in the continuous action space, and the rest are continuous action spaces.

[0065] Figure 5 A structured hybrid action space according to an embodiment of the present invention is shown.

[0066] like Figure 5 As shown, every 45° is a discrete action space (i.e., discrete joint action value), and the dotted line represents the exploration range of the small-range continuous action space. The exploration range of the continuous action space is a fan-shaped range centered on each discrete action space. The following generates continuous joint action values ​​in the small-range continuous action space adjacent to the discrete joint action value. Figure 5 The structured hybrid action space shown is only exemplary, and the number of discrete action spaces in the structured hybrid action space of the present invention is not limited.

[0067] In operation S330, the sample state posture, task target coordinates and discrete joint action values ​​are input into a low-level learning module of a preset hierarchical reinforcement model to obtain continuous joint action values ​​of the multi-manipulator.

[0068] Figure 6 A schematic diagram of the structure of a hierarchical reinforcement learning architecture according to an embodiment of the present invention is shown.

[0069] like Figure 6 As shown, in a multi-robot environment, each robot i has a high-level learning module and a low-level learning module ,in represents the parameters of the high-level learning module, Represents the parameters of the low-level learning module. , input the sample state pose of the robot And the coordinates of the mission target , thus obtaining the discrete joint action values ​​of all joints ; For the low-level learning module, input sample state pose 、Mission target coordinates and the above discrete joint action values , thus obtaining the continuous joint action values ​​of all joints .

[0070] In operation S340 , the posture of the multi-manipulator is adjusted based on the discrete joint action values ​​and the continuous joint action values ​​to obtain an adjusted posture of the multi-manipulator.

[0071] According to an embodiment of the present invention, the sum of the discrete joint action value and the continuous joint action value may be used as the joint action value of the robot arm at the current time step to interact with the environment, thereby adjusting the posture of the multi-robot arm.

[0072] In operation S350 , a reward value is determined based on the adjusted posture of the multi-manipulator.

[0073] In operation S360, a lower-level learning module of a preset hierarchical reinforcement model is adjusted based on the reward value to obtain a trained hierarchical reinforcement model.

[0074] According to an embodiment of the present invention, the reward value can be put into the experience recycling pool, and when the network needs to be updated later, the strategy formula is calculated to update the strategy network, and the evaluation network is updated by the reward value, so that the evaluation network converges to the true value.

[0075] According to an embodiment of the present invention, by splitting the joint movements of the robotic arm into discrete joint movements and continuous joint movements for hierarchical calculation, the characteristics of high exploration efficiency and fast convergence speed of discrete movements can be fully utilized to quickly obtain discrete joint movement values ​​through high-level learning modules. Through low-level learning modules, iteration is performed on the basis of discrete joint movement values ​​to quickly obtain corresponding continuous joint movement values, thereby improving movement accuracy. This greatly improves training efficiency while ensuring movement accuracy.

[0076] According to an embodiment of the present invention, the above method also includes: using a rapidly expanding random tree algorithm to calculate the joint path points of the robotic arms based on the state posture of multiple robotic arms; extracting the discrete joint calculation action values ​​of the multiple robotic arms from the joint path points of the robotic arms; obtaining the loss value based on the discrete joint action values ​​and the discrete joint calculation action values; and adjusting the high-level learning module of the preset hierarchical reinforcement model based on the loss value.

[0077] According to an embodiment of the present invention, an expert data-guided approach can be used to accelerate high-level strategy convergence, wherein the expert data is a robotic arm joint path point that can reach the target position generated by a fast expansion random number algorithm, and the aforementioned robotic arm joint path points are extracted to obtain discrete joint calculation action values, which are then cross-entropy lost with the discrete joint action values ​​output by the high-level learning module to obtain a loss value, and the high-level learning module of the preset hierarchical model is adjusted based on the loss value.

[0078] Figure 7 A training framework diagram of multi-robotic arm motion planning reinforcement learning training based on structured action space according to an embodiment of the present invention is shown.

[0079] like Figure 7 As shown in the figure, the high-level learning module includes an LSTM (Long Short-Term Memory) encoder, a fully connected layer, and a Softmax normalization layer. The high-level learning module is a discrete action decision layer, so it is necessary to perform classification regression and output the probability of discrete joint action values. The Softmax normalization layer is used to ensure that the sum of the probabilities of all discrete actions is 1; the low-level learning module includes an LSTM encoder, a fully connected layer, and a tanh activation function layer. The low-level learning module is a continuous action decision layer, so it is necessary to perform fitting regression and output the mean and variance of continuous actions. The output continuous joint action value is obtained by random sampling of the output Gaussian distribution, s t represents the sample state pose and task target coordinates, a t d represents the discrete joint action value, a t c Represents continuous joint action values.

[0080] According to an embodiment of the present invention, discrete joint calculation action values ​​are obtained by calculating through a fast expansion random number algorithm, and under the guidance of the discrete joint calculation action values, the high-level learning module can quickly converge to a near-optimal value, thereby solving the non-stationarity problem caused by simultaneous strategy changes, and then promoting the convergence of low-level learning modules, improving the stability of parallel training, and then improving the training speed.

[0081] According to an embodiment of the present invention, the above-mentioned determination of reward value based on the adjustment posture of multiple robotic arms includes: when the multiple robotic arms do not collide with each other during the process of adjusting the posture of the multiple robotic arms, rewarding the adjustment posture; when the multiple robotic arms collide with each other during the process of adjusting the posture of the multiple robotic arms, negative rewarding the adjustment posture.

[0082] According to an embodiment of the present invention, the above method also includes: rewarding the adjusted posture when the robotic arm reaches the task target coordinates within a preset time; rewarding the adjusted posture when the direction coordinates of the robotic arm end of the adjusted posture are closer to the task target coordinates than the direction coordinates of the robotic arm end of the state posture.

[0083] According to an embodiment of the present invention, since the existing reinforcement learning algorithm has the problem of sparse rewards, additional intrinsic rewards can be added to reduce errors and collisions. The collisions between the above-mentioned robotic arms may include collisions between robotic arms, collisions between robotic arms themselves, and collisions between robotic arms and the ground.

[0084] According to an embodiment of the present invention, a reward may be given for posture adjustment when the distance between the end effector of the robot arm and the task target coordinates is smaller than that in the previous time step.

[0085] According to an embodiment of the present invention, by giving rewards for avoiding collisions during the process of adjusting the posture of the robotic arm, damage to the robotic arm can be reduced and the target task can be completed quickly. By giving rewards when the robotic arm reaches the task target coordinates within a preset time, the robotic arm can complete the target task quickly and accurately. By giving rewards when the direction coordinates of the end of the robotic arm are closer to the task target coordinates, the robotic arm can be guided to correctly position and orient in space. By giving rewards or negative rewards for the posture adjustment of the robotic arm, the operating efficiency of multiple robotic arms can be improved, and errors and collisions during operation can be reduced, thereby accelerating the convergence of low-level learning modules, and further accelerating the convergence of the preset hierarchical reinforcement model, and then quickly obtaining a trained hierarchical reinforcement model.

[0086] According to an embodiment of the present invention, the above-mentioned reward for adjusting posture includes: constructing a state transition reward value based on the state posture, adjustment posture, discrete joint action values ​​and continuous joint action values; and placing the state transition reward value into an experience recovery pool.

[0087] According to an embodiment of the present invention, the above-mentioned low-level learning module includes a policy network and an evaluation network. The low-level learning module that adjusts the preset hierarchical reinforcement model based on the reward value includes: calculating the policy update formula based on multiple state transition reward values ​​in the experience recovery pool; updating the policy network based on the policy update formula; updating the evaluation network based on multiple state transition reward values.

[0088] According to an embodiment of the present invention, the state transition reward value will be stored in the experience recycling pool, which is a temporary storage area for saving the most recent state transitions and corresponding reward values. The experience recycling pool allows the model to learn from past experience instead of relying solely on real-time feedback, which can make the model converge faster.

[0089] Figure 8 A reward value curve diagram according to an embodiment of the present invention is shown.

[0090] like Figure 8 As shown, the blue line is the discrete hybrid action of the present invention, the orange line is the pure discrete action, the green line is the continuous action, the horizontal axis is the time step (i.e. the number of iterations in the model learning process), and the vertical axis is the reward value. It can be seen that the discrete hybrid action proposed in the present invention can converge faster than the pure discrete action and the continuous action, and obtain a higher reward value.

[0091] Fig. 9 A schematic diagram of reward function comparison according to an embodiment of the present invention is shown.

[0092] like Fig. 9 As shown, the red line is the position residual of the hierarchical reinforcement model with multiple reward values ​​proposed in the present invention, the green line is the position residual of the hierarchical reinforcement model without reward value, the blue line is the collision curve of the hierarchical reinforcement model with multiple reward values ​​proposed in the present invention, and the yellow line is the collision curve of the hierarchical reinforcement model without reward value, wherein the horizontal axis is the time step, and the vertical axis is the position residual and collision. It can be seen that the position residual of the present invention converges faster and has fewer collisions.

[0093] Fig.10 A comparison chart of ablation experiments between expert data and hierarchical reinforcement learning according to an embodiment of the present invention is shown.

[0094] like Fig.10 As shown, the horizontal axis is the time step, the vertical axis is the success rate and the cross entropy loss, the blue line is the cross entropy loss curve of the hierarchical reinforcement model without expert data guidance, the yellow line is the cross entropy loss curve of the hierarchical reinforcement model guided by expert data of the present invention, the red line is the success rate curve of the hierarchical reinforcement model guided by expert data of the present invention, and the green line is the success rate curve of the hierarchical reinforcement model without expert data guidance. It can be clearly seen that the present invention can converge faster and has a higher success rate.

[0095] Fig.11 A discrete action interval comparison graph according to an embodiment of the present invention is shown.

[0096] like Fig.11 As shown, the abscissa represents the time step, the ordinate represents the success rate, the blue line is the success rate curve when the maximum value of the discrete space is 0.3, the orange line is the success rate curve when the maximum value of the discrete space is 0.4, the green line is the success rate curve when the maximum value of the discrete space is 0.5, the red line is the success rate curve when the maximum value of the discrete space is 0.6, the purple line is the success rate curve when the maximum value of the discrete space is 0.7, and the brown line is the success rate curve when the maximum value of the discrete space is 0.9. It can be clearly seen that the success rate is highest when the maximum value of the discrete space is set to 0.5.

[0097] Fig.12 A flow chart of a robot arm motion planning method according to an embodiment of the present invention is shown.

[0098] like Fig.12 As shown, the robot arm motion planning method of this embodiment includes operations S1210 to S1230.

[0099] In operation S1210 , a target state posture of the multi-manipulator and a target coordinate corresponding to a target task to be performed by the multi-manipulator are acquired.

[0100] According to an embodiment of the present invention, the target coordinates may be, for example, range coordinates.

[0101] Exemplarily, after the robot arm grasps the target object, the target object can be placed within the range of the pallet, and the coordinates corresponding to the range of the pallet can be the above-mentioned target coordinates.

[0102] In operation S1220, the target state pose and target coordinates are input into the hierarchical reinforcement model to obtain a motion planning instruction.

[0103] Among them, the hierarchical reinforcement model is obtained based on the above-mentioned multi-robotic arm motion planning reinforcement learning training method based on structured action space.

[0104] In operation S1230 , motion planning is performed on the multi-manipulator based on the motion planning instruction.

[0105] Based on the above-mentioned multi-manipulator motion planning reinforcement learning training method based on structured action space, the present invention also provides a multi-manipulator motion planning reinforcement learning training device based on structured action space. Fig.13 The device is described in detail.

[0106] Fig.13 A structural block diagram of a multi-manipulator motion planning reinforcement learning training device based on structured action space according to an embodiment of the present invention is shown.

[0107] like Fig.13 As shown, the multi-robotic arm motion planning reinforcement learning training device 1300 based on structured action space of this embodiment includes a first acquisition module 1310, a discrete joint action value module 1320, a continuous joint action value module 1330, a posture adjustment module 1340, a determination module 1350 and a model adjustment module 1360.

[0108] The first acquisition module 1310 is used to acquire the sample state posture of the multi-manipulator and the task target coordinates corresponding to the sample tasks that the multi-manipulator needs to perform, and the task target coordinates represent the target placement position after the multi-manipulator grasps the target object or the target grasping position of the multi-manipulator grasping the target object. In one embodiment, the first acquisition module 1310 can be used to perform the operation S310 described above, which will not be repeated here.

[0109] The discrete joint action value module 1320 is used to input the sample state posture and task target coordinates into the high-level learning module of the preset hierarchical reinforcement model to obtain the discrete joint action values ​​of the multi-manipulator, the discrete joint action values ​​are preset values ​​in the discrete joint action value group, and the discrete joint action value group represents multiple preset action angles of the multi-manipulator in the structured action space. In one embodiment, the discrete joint action value module 1320 can be used to perform the operation S320 described above, which will not be repeated here.

[0110] The continuous joint action value module 1330 is used to input the sample state posture, task target coordinates and discrete joint action values ​​into the low-level learning module of the preset hierarchical reinforcement model to obtain the continuous joint action values ​​of the multi-manipulator. In one embodiment, the continuous joint action value module 1330 can be used to perform the operation S330 described above, which will not be repeated here.

[0111] The posture adjustment module 1340 is used to adjust the posture of the multi-manipulator based on the discrete joint action value and the continuous joint action value to obtain the adjusted posture of the multi-manipulator. In one embodiment, the posture adjustment module 1340 can be used to perform the operation S340 described above, which will not be repeated here.

[0112] The determination module 1350 is used to determine the reward value based on the adjusted posture of the multi-manipulator. In one embodiment, the determination module 1350 can be used to perform the operation S350 described above, which will not be repeated here.

[0113] The model adjustment module 1360 is used to adjust the lower-level learning module of the preset hierarchical reinforcement model based on the reward value to obtain a trained hierarchical reinforcement model. In one embodiment, the model adjustment module 1360 can be used to perform the operation S360 described above, which will not be repeated here.

[0114] According to an embodiment of the present invention, any multiple modules among the first acquisition module 1310, the discrete joint action value module 1320, the continuous joint action value module 1330, the posture adjustment module 1340, the determination module 1350 and the model adjustment module 1360 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the first acquisition module 1310, the discrete joint action value module 1320, the continuous joint action value module 1330, the posture adjustment module 1340, the determination module 1350 and the model adjustment module 1360 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the first acquisition module 1310, the discrete joint action value module 1320, the continuous joint action value module 1330, the posture adjustment module 1340, the determination module 1350 and the model adjustment module 1360 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0115] Based on the above-mentioned robot arm motion planning method, the present invention also provides a robot arm motion planning device. Fig.14 The device is described in detail.

[0116] Fig.14 A structural block diagram of a robot arm motion planning device according to an embodiment of the present invention is shown.

[0117] like Fig.14 As shown, the robot arm motion planning device 1400 of this embodiment includes a second acquisition module 1410 , an input module 1420 and a motion planning module 1430 .

[0118] The second acquisition module 1410 is used to acquire the target state posture of the multi-manipulator and the target coordinates corresponding to the target task to be performed by the multi-manipulator. In one embodiment, the second acquisition module 1410 can be used to perform the operation S1210 described above, which will not be repeated here.

[0119] The input module 1420 is used to input the target state posture and target coordinates into the hierarchical reinforcement model to obtain motion planning instructions. The hierarchical reinforcement model is obtained based on the multi-manipulator motion planning reinforcement learning training method based on the structured action space. In one embodiment, the input module 1420 can be used to perform the operation S1220 described above, which will not be repeated here.

[0120] The motion planning module 1430 is used to perform motion planning for the multi-manipulator based on the motion planning instruction. In one embodiment, the motion planning module 1430 can be used to perform the operation S1230 described above, which will not be described in detail here.

[0121] According to an embodiment of the present invention, any multiple modules of the second acquisition module 1410, the input module 1420 and the motion planning module 1430 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the second acquisition module 1410, the input module 1420 and the motion planning module 1430 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the second acquisition module 1410, the input module 1420 and the motion planning module 1430 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0122] Fig.15 A block diagram of an electronic device suitable for implementing a multi-manipulator motion planning reinforcement learning training method based on a structured action space according to an embodiment of the present invention is shown.

[0123] like Fig.15As shown, the electronic device 1500 according to an embodiment of the present invention includes a processor 1501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1502 or the program loaded from the storage part 1508 to the random access memory (RAM) 1503. The processor 1501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1501 may also include an onboard memory for caching purposes. The processor 1501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0124] In RAM 1503, various programs and data required for the operation of electronic device 1500 are stored. Processor 1501, ROM 1502 and RAM 1503 are connected to each other via bus 1504. Processor 1501 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 1502 and / or RAM 1503. It should be noted that the program can also be stored in one or more memories other than ROM 1502 and RAM 1503. Processor 1501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0125] According to an embodiment of the present invention, the electronic device 1500 may further include an input / output (I / O) interface 1505, which is also connected to the bus 1504. The electronic device 1500 may further include one or more of the following components connected to the input / output (I / O) interface 1505: an input portion 1506 including a keyboard, a mouse, etc.; an output portion 1507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1508 including a hard disk, etc.; and a communication portion 1509 including a network interface card such as a LAN card, a modem, etc. The communication portion 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to the input / output (I / O) interface 1505 as needed. A removable medium 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1510 as needed, so that a computer program read therefrom is installed into the storage portion 1508 as needed.

[0126] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0127] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 1502 and / or RAM 1503 described above and / or one or more memories other than ROM 1502 and RAM 1503.

[0128] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the multi-manipulator motion planning reinforcement learning training method based on structured action space provided by the embodiment of the present invention.

[0129] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 1501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0130] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1509, and / or installed from the removable medium 1511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0131] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1509, and / or installed from the removable medium 1511. When the computer program is executed by the processor 1501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0132] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0133] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0135] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A multi-manipulator motion planning reinforcement learning training method based on structured action space, characterized in that: The method comprises: Acquire sample state poses of the multi-manipulator and task target coordinates corresponding to sample tasks to be performed by the multi-manipulator, wherein the task target coordinates represent a target placement position after the multi-manipulator grasps a target object or a target grasping position of the multi-manipulator grasping the target object; Input the sample state posture and the task target coordinates into a high-level learning module of a preset hierarchical reinforcement model to obtain discrete joint action values ​​of the multi-manipulator, wherein the discrete joint action values ​​are preset values ​​in a discrete joint action value group, and the discrete joint action value group represents a plurality of preset action angles of the multi-manipulator in the structured action space; Inputting the sample state posture, the task target coordinates and the discrete joint action values ​​into a low-level learning module of a preset hierarchical reinforcement model to obtain the continuous joint action values ​​of the multi-manipulator; Adjusting the posture of the multi-manipulator based on the discrete joint action value and the continuous joint action value to obtain an adjusted posture of the multi-manipulator; Determining a reward value based on the adjusted posture of the multi-manipulator; A low-level learning module of the preset hierarchical reinforcement model is adjusted based on the reward value to obtain a trained hierarchical reinforcement model.

2. The method according to claim 1, characterized in that The method further comprises: Using a fast expanding random tree algorithm, the robot arm joint path points are calculated based on the state and posture of the multi-robot arm; Extracting the joint path points of the robotic arms to obtain discrete joint calculation action values ​​of the multi-robotic arms; Obtaining a loss value based on the discrete joint action value and the discrete joint calculated action value; A high-level learning module of the preset hierarchical reinforcement model is adjusted based on the loss value.

3. The method according to claim 1, characterized in that The step of determining a reward value based on the adjusted posture of the multi-manipulator comprises: In the process of adjusting the posture of the multi-manipulator arms, if no collision occurs between the manipulator arms, rewarding the adjusted posture; When the multi-manipulator arms collide with each other during the posture adjustment process, a negative reward is given to the posture adjustment.

4. The method according to claim 3, characterized in that The method further comprises: When the robot arm reaches the task target coordinates within a preset time, rewarding the adjusted posture; When the robot arm end direction coordinates of the adjusted posture are closer to the task target coordinates than the robot arm end direction coordinates of the state posture, the adjusted posture is rewarded.

5. The method according to claim 3, characterized in that: The rewarding the adjusted posture includes: Constructing a state transition reward value based on the state posture, the adjusted posture, the discrete joint action value and the continuous joint action value; The state transfer reward value is placed into the experience recovery pool.

6. The method according to claim 5, characterized in that The low-level learning module includes a strategy network and an evaluation network, and the low-level learning module for adjusting the preset hierarchical reinforcement model based on the reward value includes: Calculate a strategy update formula based on the multiple state transition reward values ​​in the experience recycling pool; Updating the policy network based on the policy update formula; The evaluation network is updated based on the plurality of state transition reward values.

7. A robot arm motion planning method, characterized in that: The method comprises: Acquire a target state position and posture of the multi-manipulator and a target coordinate corresponding to a target task to be performed by the multi-manipulator; Inputting the target state posture and the target coordinates into a hierarchical reinforcement model to obtain motion planning instructions, wherein the hierarchical reinforcement model is obtained based on the method described in any one of claims 1 to 6; Perform motion planning for the multi-manipulator based on the motion planning instruction.

8. A multi-manipulator motion planning reinforcement learning training device based on structured action space, characterized in that: The device comprises: A first acquisition module is used to acquire a sample state posture of a multi-manipulator and a task target coordinate corresponding to a sample task to be performed by the multi-manipulator, wherein the task target coordinate represents a target placement position after the multi-manipulator grasps a target object or a target grasping position of the multi-manipulator grasping the target object; A discrete joint action value module is used to input the sample state posture and the task target coordinates into a high-level learning module of a preset hierarchical reinforcement model to obtain discrete joint action values ​​of the multi-manipulator, wherein the discrete joint action values ​​are preset values ​​in a discrete joint action value group, and the discrete joint action value group represents a plurality of preset action angles of the multi-manipulator in the structured action space; A continuous joint action value module, used for inputting the sample state posture, the task target coordinates and the discrete joint action values ​​into a low-level learning module of a preset hierarchical reinforcement model to obtain the continuous joint action values ​​of the multi-manipulator; A posture adjustment module, used for adjusting the posture of the multi-manipulator based on the discrete joint action value and the continuous joint action value to obtain an adjusted posture of the multi-manipulator; A determination module, configured to determine a reward value based on the adjusted posture of the multi-manipulator; A model adjustment module is used to adjust the lower-level learning module of the preset hierarchical reinforcement model based on the reward value to obtain a trained hierarchical reinforcement model.

9. A robot arm motion planning device, characterized in that: The device comprises: A second acquisition module is used to acquire a target state posture of the multi-manipulator and a target coordinate corresponding to a target task to be performed by the multi-manipulator; An input module, used for inputting the target state posture and the target coordinates into a hierarchical reinforcement model to obtain a motion planning instruction, wherein the hierarchical reinforcement model is obtained based on the method described in any one of claims 1 to 6; A motion planning module is used to perform motion planning for the multi-manipulator based on the motion planning instruction.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

Images