Double-mechanical-arm cooperative motion planning method and system, electronic equipment and storage medium

Abstract

The invention discloses a double-mechanical-arm cooperative motion planning method based on deep reinforcement learning, and the method comprises the steps: building a workbench simulation environment, and enabling double mechanical arms to carry out cooperative motion planning and testing in the environment; a deep reinforcement learning model of the double-mechanical-arm motion planning system is established, the model makes a decision according to the current state of the double mechanical arms in each time step so as to output a motion instruction of the double mechanical arms, and after the state of the double mechanical arms is updated, a reward value for executing the motion instruction is obtained through calculation; generating an expert track and storing the expert track in an experience playback pool as an expert sample; the states, the action instructions and the reward values formed by different time steps are stored in an experience playback pool in a grouping mode to serve as experience samples; and regularly extracting samples from the experience playback pool for batch training and updating model parameters. The invention further provides a system, electronic equipment and a storage medium. According to the method, the double-mechanical-arm cooperative movement task with the tail end posture constraint can be efficiently completed in a limited space.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent motion planning for robotic arms, and in particular to a method, system, electronic device and storage medium for collaborative motion planning of dual robotic arms based on deep reinforcement learning. Background Art

[0002] Traditional motion planning algorithms, such as artificial potential field method, rapidly expanding random tree, probabilistic roadmap method, etc., usually rely on pre-set models and environmental information. These methods work well when dealing with simple, static environments, but show obvious shortcomings when facing complex, dynamic work scenes. First, traditional algorithms have poor adaptability to environmental changes and are difficult to cope with real-time updates and emergencies. Secondly, the computational efficiency is often low, and the planning time is long in large-scale, multi-degree-of-freedom robotic arm systems. In addition, these algorithms are difficult to avoid collisions when multiple robotic arms work together, and lack the guarantee of a global optimal solution, resulting in work efficiency and accuracy being affected.

[0003] Deep reinforcement learning combines the advantages of deep learning and reinforcement learning to learn the optimal strategy through trial and error in an unsupervised environment. Compared with traditional methods, deep reinforcement learning does not rely on precise mathematical models and has stronger environmental adaptability and robustness. In the industrial field, as far as the collaborative motion planning of dual robotic arms based on this is concerned, the current method cannot solve tasks such as workpiece assembly with installation direction requirements in highly overlapping workspaces due to the lack of constraints on the posture of the end effector; at the same time, it is more difficult to train intelligent agents to complete such tasks. Summary of the invention

[0004] In view of the above technical problems, the present invention provides a dual-manipulator collaborative motion planning method based on deep reinforcement learning. Accordingly, the present invention also provides a dual-manipulator collaborative motion planning system based on deep reinforcement learning, an electronic device and a storage medium.

[0005] According to one aspect of the present invention, a dual-manipulator collaborative motion planning method based on deep reinforcement learning is provided, comprising the following steps:

[0006] Build a workbench simulation environment for dual robotic arms to perform collaborative motion planning and testing;

[0007] A deep reinforcement learning model for the dual-manipulator motion planning system is established, and decisions are made according to the current state of the dual-manipulator at each time step to output the action instructions of the dual-manipulator, and the reward value for executing the action instructions is calculated after the state of the dual-manipulator is updated; the state is defined according to the current joint angle of the manipulator, the current joint angular velocity, the current position and direction of the end effector in the world coordinate system, the current position and direction error between the end effector and the target, whether the end effector reaches the target within the allowable error range, and the minimum distance between the dual manipulators; the action is defined according to the joint angle increment of the dual manipulator; a reward function is designed based on five aspects: position control of the two end effectors, posture control of the two end effectors, reward for successful completion of the task, self-collision avoidance of the dual manipulators, and speed constraints of the two end effectors, and the reward value is calculated through the reward function;

[0008] Generate expert trajectories and store them in the experience replay pool as expert samples;

[0009] The states, action instructions, and reward values ​​formed at different time steps are grouped and stored in the experience replay pool as experience samples;

[0010] A batch of samples are periodically extracted from the experience replay pool for batch training and updating the model parameters until the training is completed, thereby realizing the collaborative motion planning of the dual robotic arms.

[0011] In one embodiment of the present invention, the posture of the end effector is represented by the Tate Blaine angle, and the reward function related to the posture control is defined as:

[0012]

[0013] in, , , Respectively represent the roll, pitch and yaw errors, corresponding to the rotation of the three coordinate axes x, y and z. Indicates the degree of influence of the posture.

[0014] In one embodiment of the present invention, the safety status of the system is described by the minimum distance between the two robotic arms. A vector-based method is introduced to calculate the minimum distance between the two arms. The reward function related to the self-collision avoidance of the two robotic arms is defined as:

[0015]

[0016] in: Indicates the minimum distance between the two robotic arms. Indicates the safety threshold.

[0017] In one embodiment of the present invention, the experience replay pool is divided into an exploration buffer and an expert buffer. The exploration buffer is used to store experience samples, and the expert buffer is used to store expert samples.

[0018] In one embodiment of the present invention, a dynamic sampling mechanism is used to balance expert guidance and self-exploration, and the number of expert samples extracted is periodically reduced to zero as training progresses.

[0019] In one embodiment of the present invention, before extracting samples for training and updating model parameters, the method further includes:

[0020] Perform similarity filtering on the samples, remove and return similar samples.

[0021] In one embodiment of the present invention, samples are similarly filtered based on the Euclidean distance, which is defined as follows:

[0022]

[0023] in, , They represent different experience samples respectively;

[0024] Set the threshold to ε, when When similar experience samples are found, they are removed and sent back.

[0025] According to another aspect of the present invention, a dual-manipulator collaborative motion planning system based on deep reinforcement learning is provided, comprising:

[0026] The environment building module is used to build a workbench simulation environment for the dual robotic arms to perform collaborative motion planning and testing in the environment;

[0027] A model creation module is used to establish a deep reinforcement learning model for the dual-manipulator motion planning system, make decisions based on the current state of the dual-manipulator at each time step to output action instructions for the dual-manipulator, and calculate the reward value for executing the action instructions after the state of the dual-manipulator is updated;

[0028] The expert sample generation module generates expert trajectories and stores them in the experience replay pool as expert samples;

[0029] The experience sample generation module groups the states, action instructions, and reward values ​​formed at different time steps and stores them in the experience replay pool as experience samples; and

[0030] The training module periodically extracts a batch of samples from the experience replay pool for batch training and updates the model parameters until the training is completed, thereby realizing the collaborative motion planning of the dual robotic arms.

[0031] According to another aspect of the present invention, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0032] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0033] processor;

[0034] A storage medium stores a computer program thereon, wherein the computer program executes the steps of the method described above when executed by the processor.

[0035] The present invention can efficiently complete the task of dual-manipulator coordinated motion with end-point posture constraints in a limited space. Specifically:

[0036] 1) Compared with the prior art, the present invention not only takes into account the motion path of the robot arm, but also uses the reward function to control the posture of the dual robot arms in real time and describe the safety status of the system. It can accurately control the posture of the end effector without relying on accurate kinematic models and complex inverse kinematics solution processes. It has higher flexibility and adaptability, which can enhance the adaptability and collaborative motion accuracy of the dual robot arms in complex task scenarios.

[0037] 2) Different from the traditional experience replay method, the present invention can significantly improve the validity of data by introducing expert guidance strategy and dual experience structure. Further through dynamic sampling mechanism and similarity filtering, the selection process of training samples can be optimized, thereby improving learning efficiency and model convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0039] Figure 1 It is a flow chart of a dual-robotic-arm collaborative motion planning method based on deep reinforcement learning in one embodiment of the present invention.

[0040] Figure 2 It is a module diagram of a dual-robotic-arm collaborative motion planning system based on deep reinforcement learning in one embodiment of the present invention.

[0041] Figure 3 It is a schematic diagram of the structure of an electronic device in one embodiment of the present invention.

[0042] Figure 4It is a schematic diagram of the structure of a computer-readable storage medium in one embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the description of the present invention more detailed and complete, reference may be made to the accompanying drawings and various embodiments described below, in which the same numbers represent the same or similar components. On the other hand, well-known components and steps are not described in the embodiments to avoid unnecessary limitations on the present invention. In the embodiments and claims, the description of "connection" may generally refer to a component being indirectly coupled to another component through other components, or a component being directly connected to another component without passing through other components. In the embodiments and claims, the description of "online" may generally refer to a component being indirectly connected to another component through other components for wired and / or wireless communication, or a component being physically connected to another component without passing through other components. In the embodiments and claims, unless the article is specifically limited in the text, "a" and "the" may generally refer to a single or multiple. "About", "approximately" or "roughly" as used herein is used to modify any quantity that can be slightly changed, but such slight changes do not change its essence. In the embodiments, if there is no special explanation, the error range of the numerical value modified by "about", "approximately" or "substantially" is generally allowed within 20%, preferably within 10%, and more preferably within 5%.

[0044] According to one aspect of the present invention, a dual-manipulator collaborative motion planning method based on deep reinforcement learning is provided. Figure 1 , the method comprises the following steps:

[0045] S1, build a workbench simulation environment for the dual robotic arms to perform collaborative motion planning and testing in this environment.

[0046] Specifically, the construction of the workbench simulation environment mainly includes clarifying the needs of the simulation environment, selecting an appropriate simulation platform, creating a workbench model, building a dual-manipulator model, configuring sensors and interfaces, configuring physical engine parameters, testing and verification, etc. In one embodiment of the present invention, a simplified workbench simulation environment is built through the PyBullet simulation platform, so that the dual-manipulator can perform collaborative motion planning and testing in this environment. PyBullet is an open source real-time physics simulation library that is widely used in robot simulation, reinforcement learning, scientific computing, and computer graphics. Based on the Bullet physics engine, PyBullet provides an easy-to-use Python interface to facilitate developers to quickly build a physical simulation environment. It also provides accurate dynamic simulation, including rigid body dynamics, collision detection and processing, joint and constraint mechanics, etc. Due to its flexibility and ease of use, it is widely used in robotics research, artificial intelligence training, game development and other fields.

[0047] S2, establish a deep reinforcement learning model for the dual-manipulator motion planning system, make decisions based on the current state of the dual-manipulator at each time step to output the action instructions of the dual-manipulator, and calculate the reward value for executing the action instructions after the state of the dual-manipulator is updated. In the implementation process of the present invention, the state can be defined according to the current joint angle of the manipulator, the current joint angular velocity, the current position and direction of the end effector in the world coordinate system, the current position and direction error between the end effector and the target, whether the end effector reaches the target within the error allowable range, and the minimum distance between the dual-manipulator.

[0048] The action can be defined according to the joint angle increment of the dual manipulators. A reward function can be designed based on five aspects: position control of two end effectors, posture control of two end effectors, reward for successful task completion, self-collision avoidance of the dual manipulators, and speed constraints of two end effectors, and the reward value can be calculated through the reward function.

[0049] Specifically, the trajectory generation problem of the robot arm is usually described as a Markov decision process, and deep reinforcement learning can be used to solve this problem. The Markov decision process consists of a five-tuple (S, A, P, R, γ), where S is a finite state set, A is a finite action set, P represents the state transition probability distribution, 𝑅 represents the reward function, and γ∈[0,1] is the discount factor.

[0050] For status is defined as follows:

[0051]

[0052] Among them, taking robot arm 1 as an example, Represents the current joint angle of the robot arm, is the current joint angular velocity, i is the i-th joint. is the current position and orientation of the end effector in the world coordinate system. It is the current position and direction error between the end effector and the target, and it directly displays the current posture error. Boolean It represents whether the end effector reaches the target within the allowable error range. It is the minimum distance between the two arms, reflecting the safety of the system. In the following text, only the variables of robot arm 1 are explained, and the same is true for robot arm 2, which will not be repeated.

[0053] Further, action It can be defined as:

[0054]

[0055] in, Represents the joint angle increment of robot arm 1.

[0056] Furthermore, at each time step t, the agent obtains the current state of the environment through sensors Make decisions based on state transition probabilities Select Action After executing the action, the agent receives a numerical instant reward At the same time, the environment is affected by the action of the intelligent agent and enters the next new state , and then the agent continues to interact with the environment according to the new state. The ultimate goal of the deep reinforcement learning algorithm is to find an optimal strategy to maximize the cumulative reward from the beginning to the end of training. Strategy and cumulative rewards can be defined as

[0057]

[0058]

[0059] in, is the immediate reward obtained at time t, and T is the total number of steps in the round.

[0060] In the implementation process of the present invention, a dual-arm collaborative assembly task with end posture constraints is considered. The reward function is designed based on the position control of the two end effectors, the posture control of the two end effectors, the reward for successful completion of the task, the self-collision avoidance of the dual manipulators and the speed constraints of the two end effectors, and the reward value is calculated through the reward function. The first three aspects are responsible for guiding the agent to complete the task, and the last two aspects provide protection for system safety. Therefore, the present invention additionally considers the end effector posture constraint and the self-collision avoidance of the two arms in the reward function. Compared with the previous consideration of only position control and obstacle avoidance outside the system, it has more practical application significance and is suitable for dual-arm operation tasks with requirements for the end posture. The reward function related to the position control of the end effector uses the Euclidean distance to represent the position error. In the motion planning process, it is hoped that the reward value will increase as the position error decreases, and when the position error is large, the reward can be rapidly reduced to accelerate the convergence speed. In order to avoid collision with the target, when the end effector reaches the position near the target and the direction error is within the allowable range, it is determined that the target point has been reached. When both arms meet the condition, the task is considered completed and a task completion reward of 1 is given. When only one end effector meets the condition, a reward of 0.01 is given. The end effector speed constraint is also used. Excessive end effector speed may cause deviation from the target point or even collision with the target, causing damage to the system. Taking safety and efficiency into consideration, when the end effector is far away from the target, the speed constraint should not be too large; when it is close to the target, the speed constraint should be strengthened; if it is less than the safety threshold , giving the agent a penalty, similar to the self-collision avoidance reward function below.

[0061] Furthermore, the posture of the end effector can be represented by the Tait-Bryan angle, and the reward function related to the posture control is defined as:

[0062]

[0063] in, , , Respectively represent the roll, pitch and yaw errors, corresponding to the rotation of the three coordinate axes x, y and z. Indicates the degree of influence of the posture.

[0064] Compared with quaternions, Tait-Bryan angles have one less dimension and run through the three axes of rectangular coordinates. Different objects always point to the same fixed coordinate system, which is different from the classic Euler angles. Therefore, the directional deviation between the end effector posture and the main axis defined by the reward function in the present invention is simpler, intuitive and clearer. In addition, it is very common to directly use the translation and rotation components of the end effector as actions, but this may lead to the gimbal deadlock problem. The present invention proposes a method without inverse kinematics, setting the joint angle increment of the robot arm as the action. At each time step t, the direction of the end effector changes very little after a slight rotation of the joint. This joint space-based trajectory generation method does not involve traditional Euler angles and does not directly rotate the end effector, thereby avoiding the gimbal deadlock problem. Through the proposed directional control reward function, the agent attempts to find the target solution through several consecutive time steps.

[0065] In the implementation process of the present invention, the safety status of the system can be described by the minimum distance between the two mechanical arms. A vector-based method is introduced to calculate the minimum distance between the two arms. Less than the safety threshold When the minimum distance is , the system will enter a dangerous state and collision is likely to occur. At this time, a penalty is given to the agent. When it is relatively large, the penalty should be ignored to avoid the two arms deliberately moving away from each other in order to reduce the penalty, resulting in mission failure. Therefore, the reward function related to the self-collision avoidance of the two manipulators can be defined as:

[0066]

[0067] in: Indicates the minimum distance between the two robotic arms. Indicates the safety threshold.

[0068] S3, generates expert trajectories and stores them in the experience replay pool as expert samples.

[0069] Specifically, in the early stages of training, a deep reinforcement learning system with poor parameters finds it difficult to complete tasks, and the self-exploration experience pool is mostly low-quality data, while the expert guidance strategy can provide successful experience, guide the agent to obtain high rewards and collect high-quality data. Before training the agent, the expert demonstration trajectories obtained by the centralized BiRRT algorithm can be generated and stored at one time, thereby ensuring the quality and quantity of the expert demonstration data, which is convenient for immediate use at the beginning of training. These trajectories not only cover multiple areas of the target configuration space, but also ensure the feasibility and optimization degree of the path, providing reliable demonstration data for subsequent deep reinforcement learning. Therefore, the present invention introduces an expert guidance strategy in deep reinforcement learning to improve learning efficiency and accelerate convergence.

[0070] S4, groups the states, action instructions, and reward values ​​formed at different time steps and stores them in the experience replay pool as experience samples.

[0071] Specifically, the agent and the environment are first initialized. Then the agent observes the current state of the environment through sensors and interfaces, obtains necessary information, converts the observed information into a state vector, and inputs it into a deep reinforcement learning network. According to the current state, the agent selects and executes actions based on the strategy. Then the state is updated and the reward value of the current action is calculated according to the reward function. Finally, the state, action instructions, reward values ​​and other data formed by the interaction between the agent and the environment at different time steps are grouped and stored in the experience replay pool.

[0072] S5, regularly extracting a batch of samples from the experience replay pool for batch training and updating model parameters until the training is completed, so as to realize the collaborative motion planning of the dual robotic arms.

[0073] Specifically, the SAC algorithm can be used to update the model parameters. The SAC algorithm is based on the maximum entropy reinforcement learning framework. It encourages the algorithm to explore by introducing an entropy regularization term, thereby improving the performance of the algorithm. By continuously interacting with the environment, the agent gradually learns the optimal strategy. During the training process, the performance of the agent gradually improves, and it can efficiently complete the dual-manipulator collaborative motion task with end-point posture constraints in a limited space.

[0074] In one embodiment of the present invention, the experience replay pool is divided into an exploration buffer and an expert buffer, wherein the exploration buffer is used to store experience samples and the expert buffer is used to store expert samples. After the similar experience samples are removed, they will be sent back to the corresponding buffer.

[0075] Specifically, the experience replay pool is used to store historical information of the interaction between the agent and the environment, from which the agent periodically extracts experience samples for learning and then updates the network model. The quality of the samples greatly affects the training efficiency of the network. The present invention adopts a dual experience structure combined with expert experience. Experts will not affect the ultimate optimality of the strategy, because the agent can continue to improve its optimality through successful trajectories beyond expert demonstrations collected through self-exploration.

[0076] In one embodiment of the present invention, a dynamic sampling mechanism is used to balance expert guidance and self-exploration, and the number of expert samples extracted is periodically reduced to zero as training progresses.

[0077] Specifically, after storing enough experience samples, it is necessary to extract small batches of samples from the experience replay buffer for network update. Different from the commonly used experience replay methods such as PER and HER, a dynamic sampling mechanism more suitable for dual experience replay buffer is designed to balance expert guidance and self-exploration and improve learning efficiency. and Expert Buffer Randomly select and The size of the data, and The size of changes dynamically over time. The definition is as follows:

[0078]

[0079] where N is the size of the mini-batch drawn from the experience replay buffer for each time step t, is the attenuation step length. and All are non-negative integers.

[0080] In the early stages of training, the expert demonstration trajectories provide a variety of additional successful experiences, which can effectively guide the agent to find higher cumulative rewards and enable it to collect some high-quality data through self-exploration. As training progresses, more and more high-quality samples are stored in In order to prevent overfitting, the sampling needs to become more and more unbiased, It periodically decreases to zero, after which the agent is able to learn and optimize solely through data from self-exploration.

[0081] In one embodiment of the present invention, before extracting samples for training and updating model parameters, the method further includes:

[0082] Perform similarity filtering on the samples, remove and return similar samples.

[0083] Specifically, before using empirical samples to update network parameters, in order to further improve training efficiency and enhance model generalization ability, samples can be similarly filtered based on Euclidean distance, which is defined as follows:

[0084]

[0085] in, , They represent different experience samples.

[0086] Set the threshold to ε, when When , similar experience samples will be eliminated and sent back, which can reduce redundant calculations and update operations during training, thereby making the model converge faster.

[0087] In summary, the present invention can efficiently complete the task of dual-manipulator coordinated motion with end-point posture constraints in a limited space. Specifically:

[0088] 1) Compared with the prior art, the present invention not only takes into account the motion path of the robot arm, but also uses the reward function to control the posture of the dual robot arms in real time and describe the safety status of the system. It can accurately control the posture of the end effector without relying on accurate kinematic models and complex inverse kinematics solution processes. It has higher flexibility and adaptability, which can enhance the adaptability and collaborative motion accuracy of the dual robot arms in complex task scenarios.

[0089] 2) Different from the traditional experience replay method, the present invention can significantly improve the validity of data by introducing expert guidance strategy and dual experience structure. Further through dynamic sampling mechanism and similarity filtering, the selection process of training samples can be optimized, thereby improving learning efficiency and model convergence speed.

[0090] According to another aspect of the present invention, a dual-robotic arm collaborative motion planning system based on deep reinforcement learning is provided. Figure 2 is a module diagram of a dual-manipulator collaborative motion planning system based on deep reinforcement learning in one embodiment of the present invention. Figure 2As shown, the system 300 includes an environment building module 310, a model creation module 320, an expert sample generation module 330, an experience sample generation module 340 and a training module 350. The environment building module 310 is used to build a workbench simulation environment for the dual robotic arms to perform collaborative motion planning and testing in the environment. The model creation module is used to establish a deep reinforcement learning model for the dual-manipulator motion planning system. The model makes decisions based on the current state of the dual-manipulator at each time step to output the action instructions of the dual-manipulator, and calculates the reward value for executing the action instructions after the state of the dual-manipulator is updated; the state is defined according to the current joint angle of the manipulator, the current joint angular velocity, the current position and direction of the end effector in the world coordinate system, the current position and direction error between the end effector and the target, whether the end effector reaches the target within the error allowable range, and the minimum distance between the dual-manipulator; the action is defined according to the joint angle increment of the dual-manipulator; the reward function is designed based on the position control of the two end effectors, the posture control of the two end effectors, the reward for successful completion of the task, the self-collision avoidance of the dual-manipulator and the speed constraint of the two end effectors, and the reward value is calculated through the reward function. The expert sample generation module 330 generates an expert trajectory and stores it in the experience replay pool as an expert sample. The experience sample generation module 340 groups the states, action instructions, and reward values ​​formed at different time steps and stores them in the experience replay pool as experience samples. The training module 350 periodically extracts a batch of samples from the experience replay pool for batch training and updates model parameters until the training is completed, thereby realizing the collaborative motion planning of dual robotic arms. The present invention can efficiently complete the collaborative motion task of dual robotic arms with end posture constraints in a limited space.

[0091] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0092] processor;

[0093] A storage medium stores a computer program thereon, wherein the computer program executes the steps of the method described above when executed by the processor.

[0094] Refer to the following Figure 3 To describe an electronic device 600 in one embodiment of the present invention. Figure 3 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0095] like Figure 4As shown, the electronic device 600 is in the form of a general computing device. The components of the device 600 may include, but are not limited to: at least one processor 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processor 610), a display unit 640, etc.

[0096] The storage unit stores program codes, which can be executed by the processor 610, so that the processor 610 executes the steps of various exemplary embodiments of the present invention described in the electronic prescription circulation processing method section of this specification. For example, the processor 610 can execute the steps in the above method.

[0097] The storage unit 620 may include a storage medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0098] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0099] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0100] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and may also communicate with one or more devices that enable tenants to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. In addition, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network 102 adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems. The present invention can efficiently complete dual-manipulator collaborative motion tasks with end-point posture constraints in a limited space.

[0101] According to another aspect of the present invention, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0102] refer to Figure 4 In one embodiment, the program product 800 for implementing the above method may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a server. However, those skilled in the art should understand that the program product referred to in the present invention is not limited thereto, and the computer storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0103] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0104] In addition, a part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer storage media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer storage medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0105] Communication media include media by which communication signals containing, for example, computer readable instructions, data structures, program modules or other data are transmitted from one system to another. Communication media may include guided transmission media (such as cables and wires, for example, fiber optic, coaxial, etc.) and wireless (unguided transmission) media that can propagate energy waves, such as acoustic, electromagnetic, RF, microwave and infrared. Computer readable instructions, data structures, program modules or other data may be embodied as a modulated data signal in, for example, a wireless medium (such as a carrier wave or similar mechanism such as embodied as part of spread spectrum technology). The term "modulated data signal" refers to a signal whose one or more characteristics are changed or set in such a manner as to encode information in the signal. Modulation may be analog, digital, or a hybrid modulation technique.

[0106] By way of example and not limitation, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include (but are not limited to) volatile memory, such as random access memory (RAM, DRAM, SRAM); and non-volatile memory, such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or later developed that can store computer-readable information or data for use by a computer system.

[0107] In summary, the present invention can efficiently complete the dual-manipulator collaborative motion task with end-point posture constraints in a limited space.

[0108] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A dual-manipulator collaborative motion planning method based on deep reinforcement learning, characterized in that: include: Build a workbench simulation environment for dual robotic arms to perform collaborative motion planning and testing; A deep reinforcement learning model for the dual-manipulator motion planning system is established. The model makes decisions based on the current state of the dual-manipulator at each time step to output action instructions for the dual-manipulator, and calculates the reward value for executing the action instructions after the state of the dual-manipulator is updated. The state is defined according to the current joint angle of the manipulator, the current joint angular velocity, the current position and direction of the end effector in the world coordinate system, the current position and direction error between the end effector and the target, whether the end effector reaches the target within the allowable error range, and the minimum distance between the dual manipulators; the action is defined according to the joint angle increment of the dual manipulators; a reward function is designed based on five aspects: position control of the two end effectors, posture control of the two end effectors, reward for successful completion of the task, self-collision avoidance of the dual manipulators, and speed constraints of the two end effectors, and the reward value is calculated through the reward function; Generate expert trajectories and store them in the experience replay pool as expert samples; The states, action instructions, and reward values ​​formed at different time steps are grouped and stored in the experience replay pool as experience samples; A batch of samples are periodically extracted from the experience replay pool for batch training and updating the model parameters until the training is completed, thereby realizing the collaborative motion planning of the dual robotic arms.

2. The method according to claim 1, characterized in that: The posture of the end effector is represented by the Tate-Bryan angle, and the reward function associated with the posture control is defined as: in, , , Respectively represent the roll, pitch and yaw errors, corresponding to the rotation of the three coordinate axes x, y and z. Indicates the degree of influence of the posture.

3. The method according to claim 1, characterized in that: The safety status of the system is described by the minimum distance between the two manipulators. A vector-based method is introduced to calculate the minimum distance between the two arms. The reward function related to the self-collision avoidance of the two manipulators is defined as: in: Indicates the minimum distance between the two robotic arms. Indicates the safety threshold.

4. The method according to claim 1, characterized in that: The experience replay pool is divided into an exploration buffer and an expert buffer. The exploration buffer is used to store experience samples, and the expert buffer is used to store expert samples.

5. The method according to claim 1, characterized in that: A dynamic sampling mechanism is adopted to balance expert guidance and self-exploration, and the number of expert samples extracted is periodically reduced to zero as training progresses.

6. The method according to claim 1, characterized in that: Before extracting samples for training and updating model parameters, it also includes: Perform similarity filtering on the samples, remove and return similar samples.

7. The method according to claim 6, characterized in that: The samples were similarly filtered based on the Euclidean distance, which is defined as follows: in, , They represent different experience samples respectively; Set the threshold to ε, when When similar experience samples are found, they are removed and sent back.

8. A dual-manipulator collaborative motion planning system based on deep reinforcement learning, characterized in that: include: The environment building module is used to build a workbench simulation environment for the dual robotic arms to perform collaborative motion planning and testing in the environment; A model creation module is used to establish a deep reinforcement learning model for the dual-manipulator motion planning system. The model makes decisions based on the current state of the dual-manipulator at each time step to output action instructions for the dual-manipulator, and calculates the reward value for executing the action instructions after the state of the dual-manipulator is updated; The state is defined according to the current joint angle of the manipulator, the current joint angular velocity, the current position and direction of the end effector in the world coordinate system, the current position and direction error between the end effector and the target, whether the end effector reaches the target within the allowable error range, and the minimum distance between the dual manipulators; the action is defined according to the joint angle increment of the dual manipulators; a reward function is designed based on five aspects: position control of the two end effectors, posture control of the two end effectors, reward for successful completion of the task, self-collision avoidance of the dual manipulators, and speed constraints of the two end effectors, and the reward value is calculated through the reward function; The expert sample generation module generates expert trajectories and stores them in the experience replay pool as expert samples; The experience sample generation module groups the states, action instructions, and reward values ​​formed at different time steps and stores them in the experience replay pool as experience samples. as well as The training module periodically extracts a batch of samples from the experience replay pool for batch training and updates the model parameters until the training is completed, thereby realizing the collaborative motion planning of the dual robotic arms.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are executed.

10. An electronic device, characterized in that: The electronic device comprises: processor; A storage medium having a computer program stored thereon, wherein the computer program, when executed by the processor, performs the steps of the method according to any one of claims 1 to 7.

Images