Mechanical arm power assisting method and device, electronic equipment and storage medium
By constructing the link coordinate system model of the robot arm and applying the multi-intelligent body deep reinforcement learning algorithm, the problem of low intelligence level of robot arm in live working environment is solved, the construction efficiency and operation quality are improved, and the power of the robot arm is enhanced Assistive ability.
Patent Information
- Application Number
- CN202410880700.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-02
- Publication Date
- 2024-12-06
AI Technical Summary
In live working environment, the low intelligence of the robotic arm leads to a decrease in construction efficiency and operation quality, especially the reduction of workers' environmental perception ability and operation ability of the robotic arm affects the working efficiency and quality of the robotic arm.
By obtaining joint information and connection information of the robot arm, a connecting rod coordinate system model is constructed, and power assist is carried out based on the multi-intelligent body deep reinforcement learning algorithm and power assist optimization model to improve the intelligence level of the robot arm and the accuracy of power output. sex.
It improves the intelligence level of the robot arm, improves construction efficiency and operation quality, avoids power output errors caused by manual operation, and enhances the operation ability of the robot arm in complex environments.
Smart Images

Figure CN119077722A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for power assistance of a robotic arm. Background Art
[0002] In recent years, industrial robotic arms have been increasingly used in industrial production and in complex environments. Trajectory planning, as the basis for motion control of live-working robotic arms, has also received widespread attention. Its performance is of decisive significance to the working efficiency, motion smoothness and energy consumption of live-working industrial robots.
[0003] With the development of technology, the difficulty of construction is increasing, and workers' perception and judgment of the surrounding environment and ability to operate robotic arms will decline. Especially in the context of live working, the low intelligence of robotic arms seriously affects construction efficiency and work quality. Summary of the invention
[0004] The embodiments of the present application provide a robot arm power assist system, method, electronic device and storage medium, which can improve the intelligence level of the robot arm, thereby improving construction efficiency and work quality.
[0005] In a first aspect, an embodiment of the present application provides a robot arm power assist method, comprising:
[0006] Acquire joint information of a target robotic arm and connection information between the joints of the robotic arm;
[0007] Based on the joint information and the connection information, construct a link coordinate system model of the robotic arm;
[0008] Determining a power-assisted optimization model of the target robotic arm according to the connecting rod coordinate system model and a preset function;
[0009] Based on a preset multi-agent deep reinforcement learning algorithm and a power assistance optimization model of the target robotic arm, the target robotic arm is powered.
[0010] Optionally, in some embodiments of the present application, constructing a link coordinate system model of the target robotic arm based on the joint information and the connection information includes:
[0011] Determining a base of the target robotic arm;
[0012] According to the joint information and the connection information, a link coordinate system model of the target robotic arm is constructed based on the base.
[0013] Optionally, in some embodiments of the present application, constructing a link coordinate system model of the target robotic arm based on the joint information and the connection information and taking the base as a reference includes:
[0014] Establishing a link coordinate system of the target robotic arm according to the joint information and the connection information;
[0015] Determine the position vector of the end link of the target robot arm according to the joint information and the link coordinate system;
[0016] Based on the position vector and the base, a link coordinate system model of the target robotic arm is constructed.
[0017] Optionally, in some embodiments of the present application, determining the power-assisted optimization model of the target manipulator according to the connecting rod coordinate system model and a preset function includes:
[0018] According to the link coordinate system model and the preset function, obtaining random joint configuration information of the target robotic arm;
[0019] Determine a reachable position of the target robotic arm under the configuration of the random joint configuration information;
[0020] A power-assisted optimization model of the target robotic arm is determined according to a reachable position of the target robotic arm under the configuration of the random joint configuration information.
[0021] Optionally, in some embodiments of the present application, determining the power-assisted optimization model of the target robotic arm according to the reachable position of the target robotic arm under the configuration of the random joint configuration information includes:
[0022] Mapping the reachable position of the target manipulator under the configuration of the random joint configuration information to the same space, and returning to execute the step of determining the reachable position of the target manipulator under the configuration of the random joint configuration information;
[0023] Determining a working space of the target robotic arm according to the mapped reachable positions;
[0024] A preset objective function is obtained, and a power-assisted optimization model of the target robotic arm is determined according to the objective function and the workspace.
[0025] Optionally, in some embodiments of the present application, the power assistance for the target robotic arm based on the preset multi-agent deep reinforcement learning algorithm and the power assistance optimization model of the target robotic arm includes:
[0026] Determine each joint in the target robotic arm as an intelligent agent;
[0027] Based on the preset multi-agent deep reinforcement learning algorithm, output the auxiliary strategy corresponding to the agent;
[0028] In response to the real-time position change of the end effector of the target robotic arm, the target robotic arm is powered assisted according to the power-assisted optimization model of the target robotic arm and the assistance strategy corresponding to the intelligent agent.
[0029] Optionally, in some embodiments of the present application, the outputting of the auxiliary strategy corresponding to the agent based on the preset multi-agent deep reinforcement learning algorithm includes:
[0030] Initializing an agent network corresponding to the agent;
[0031] Get the preset reward function;
[0032] Training the agent network corresponding to the agent according to a preset multi-agent deep reinforcement learning algorithm and a reward function;
[0033] According to the trained agent network, an auxiliary strategy corresponding to the agent is output.
[0034] In a second aspect, an embodiment of the present application provides a mechanical arm power assist device, comprising:
[0035] An acquisition module, used to determine the joint information of the target robotic arm and the connection information between the joints of the robotic arm;
[0036] A construction module, used to construct a link coordinate system model of the robotic arm based on the joint information and the connection information;
[0037] A determination module, used to determine a power-assisted optimization model of the target robotic arm according to the connecting rod coordinate system model and a preset function;
[0038] An auxiliary module is used to provide power assistance to the target robotic arm based on a preset multi-agent deep reinforcement learning algorithm and a power assistance optimization model of the target robotic arm.
[0039] Correspondingly, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of any of the above methods when executing the program.
[0040] The present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements any of the above methods when the computer program is executed by a processor.
[0041] The present application provides a method, device, electronic device and storage medium for power assistance of a manipulator. After acquiring the joint information of a target manipulator and the connection information between the joints of the manipulator, a connecting rod coordinate system model of the manipulator is constructed based on the joint information and the connection information. Then, according to the connecting rod coordinate system model and a preset function, a power assistance optimization model of the target manipulator is determined. Finally, based on a preset multi-agent deep reinforcement learning algorithm and the power assistance optimization model of the target manipulator, the target manipulator is powered. In the power assistance scheme of the manipulator provided in the present application, a connecting rod coordinate system model of the manipulator is constructed based on the joint information and the connection information. Then, according to the connecting rod coordinate system model and a preset function, a power assistance optimization model of the target manipulator is determined. Finally, based on a preset multi-agent deep reinforcement learning algorithm and the power assistance optimization model of the target manipulator, intelligent power assistance is performed when the target manipulator performs a task, so as to avoid the manipulator providing an erroneous power output due to manual operation, thereby affecting the execution of the task. Thus, the intelligence level of the manipulator is improved, thereby improving the construction efficiency and the work quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0043] Figure 1 It is a schematic diagram of the flow of the robot arm power assistance method provided in an embodiment of the present application;
[0044] Figure 2 It is a schematic diagram of the structure of the connecting rod coordinate system model in the robot arm power assistance method provided in the embodiment of the present application;
[0045] Figure 3 is a schematic diagram of communication between intelligent agents in the robot arm power assistance method provided in an embodiment of the present application;
[0046] Figure 4 is a schematic diagram of the structure of a mechanical arm power assist device provided in an embodiment of the present application;
[0047] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of systems and methods consistent with the examples detailed in the attached claims, or with some aspects of the present application.
[0049] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover descriptions such as non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.
[0050] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0051] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.
[0052] Embodiments of the present application provide a robot arm power assistance method, device, electronic device and storage medium.
[0053] Among them, the robotic arm power assist device can be deployed in an electronic device, the electronic device can include a computer device and a mobile communication device, the computer device can include a personal computer, a laptop computer, a tablet computer, etc., the mobile communication device can include a smart phone, etc. In addition, the electronic device can communicate with the server via wired or wireless means, the server can include an independently operated server or a distributed server, or can include a server cluster composed of multiple servers, and the terminal can include a mobile phone, a tablet computer or a personal computer (PC, Personal Computer).
[0054] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.
[0055] The present application provides a method for power assistance of a robotic arm, including: obtaining joint information of a target robotic arm and connection information between the joints of the robotic arm; constructing a connecting rod coordinate system model of the robotic arm based on the joint information and the connection information; determining a power assistance optimization model of the target robotic arm according to the connecting rod coordinate system model and a preset function; and performing power assistance on the target robotic arm based on a preset multi-agent deep reinforcement learning algorithm and the power assistance optimization model of the target robotic arm.
[0056] See also Figure 1 , Figure 1 : is a schematic diagram of the process of the robot arm power assistance method provided by the present application, and the robot arm power assistance method specifically includes the following process:
[0057] 101. Obtain joint information of the target robotic arm and connection information between the joints of the robotic arm.
[0058] For example, the robot arm can be photographed by a camera, and then all joints and connecting rods of the target robot arm can be identified in the photographed image. Alternatively, the joint information of the target robot arm and the connection information between the joints of the robot arm can be obtained by identifying the technical documents provided by the manufacturer.
[0059] For example, for a 6-DOF industrial robot, number its joints from Joint 1 to Joint 6, and record the type of each joint, such as rotational joint or linear joint. For another example, use an angle ruler or electronic measuring tool to measure the maximum rotation angle of Joint 1. For example, the maximum rotation angle of Joint 1 may be ±180 degrees. Use a tape measure or laser rangefinder to measure the length of the connecting rod between Joint 1 and Joint 2, assuming the measurement result is 500 mm.
[0060] 102. Based on the joint information and connection information, construct the link coordinate system model of the robotic arm.
[0061] For example, the base of the robot arm (usually the part connected to the ground or workbench) is selected as the reference point, and the link coordinate system model of the robot arm is constructed based on the joint information and connection information. The link coordinate system model is a display form used in robotics to describe the geometric relationship and kinematic characteristics of the robot links and joints. Optionally, in some embodiments of the present application, the step of "constructing the link coordinate system model of the robot arm based on the joint information and connection information" may specifically include:
[0062] Determine the base of the target robotic arm;
[0063] According to the joint information and connection information, the link coordinate system model of the target robot arm is constructed based on the base.
[0064] Specifically, the base of the robot arm (usually the part connected to the ground) is used as a reference point to establish a global coordinate system, and a unique number is assigned to each joint and link of the robot arm, such as a unique number assigned according to their order in the robot arm. For each joint, a local coordinate system is defined. This coordinate system usually has three axes: the Z axis is along the joint axis, the X axis is usually perpendicular to the Z axis and points to the extension line of the link, and the Y axis is determined by the Z axis and the X axis. To meet the right-hand coordinate system rule, four parameters are defined for each joint and link using the Denavit-Hartenberg (DH) convention: joint angle θ, link length L, offset distance d, and link torsion angle α. Then, according to the DH parameters and the connection mode of the joint (such as spherical, hinge, etc.), the relative position and direction between the coordinate systems are determined to obtain the link coordinate system model, that is, optionally, in some embodiments of the present application, the step of "constructing the link coordinate system model of the target robot arm based on the joint information and connection information and the base", specifically can include:
[0065] Establishing a link coordinate system of the target robotic arm according to the joint information and the connection information;
[0066] According to the joint information and the link coordinate system, the position vector of the end link of the target robot arm is determined;
[0067] Based on the position vector and the base, the link coordinate system model of the target robotic arm is constructed.
[0068] For example, specifically, determine each joint and connecting rod and number them, take the base as the standard, and the joint at the front end of each connecting rod is its fixed coordinate system; then, determine the Z axis and origin of each coordinate system, the Z axis is the axial direction of each joint axis, and the origin is the intersection of joint axis i+1 and joint axis i or the intersection of their common perpendicular line and the Z axis of coordinate system {i}; then, determine the X axis of each coordinate system, X i Axis along Z i With Z i+1 Direction of the common perpendicular line, if the two joint axes intersect, let X i Perpendicular to the plane where the two joint axes are located; then, determine the Y axis of each coordinate system according to the right-hand rule; finally, make the coordinate systems {0} and {1} coincide, and select the coordinate system {n} so that the connecting rod parameter is 0 as much as possible.
[0069] After establishing the MDH (Modified Denavit-Hartenberg) link coordinate system, the DH parameter table of the robot arm can be determined, as shown in the following table:
[0070]
[0071] According to the principle of forward kinematics, the transformation matrix between the initial and final coordinate systems is:
[0072]
[0073] Among them (p x ,p y ,p z ) is the position vector of the end link of the robot. The total change between the end joint of the robot and the base is determined by the total transformation matrix, and the link coordinate system model of the robot is established, such as Figure 2 shown.
[0074] 103. Determine the power-assisted optimization model of the target robotic arm based on the connecting rod coordinate system model and the preset function.
[0075] For example, according to the link coordinate system model and the preset function, the reachable position of the end effector of the robot arm is determined, which is limited by the joint angle and the length of the link. Then, according to the multiple reachable positions of the end effector of the target robot arm, the power-assisted optimization model of the target robot arm is determined. That is, optionally, in some embodiments of the present application, the step of "determining the power-assisted optimization model of the target robot arm according to the link coordinate system model and the preset function" may specifically include:
[0076] According to the link coordinate system model and the preset function, the random joint configuration information of the target robot arm is obtained;
[0077] Determine the reachable position of the target robot under the configuration of the random joint configuration information;
[0078] According to the reachable position of the target manipulator under the configuration of random joint configuration information, a power-assisted optimization model of the target manipulator is determined.
[0079] For example, specifically, according to the position and posture of the end effector, determine the area that the robot arm can reach under the current joint configuration, repeat the above process multiple times, randomly change the joint configuration, and record the position of the end effector, use the Monte Carlo algorithm to approximate the entire workspace, and then map the calculated end effector position to a common reference coordinate system to visualize the workspace of the robot arm. Then, determine the space as a multivariate function of the parameters of each mechanism on the target robot arm, construct the objective function and constraints, and the objective function may include maximizing the workspace or minimizing energy consumption, etc. Finally, according to the objective function and the workspace, determine the power-assisted optimization model of the target robot arm, that is, optionally, in some embodiments of the present application, the step of "determining the power-assisted optimization model of the target robot arm according to the reachable position of the target robot arm under the configuration of random joint configuration information" may specifically include:
[0080] Mapping the reachable position of the target manipulator under the configuration of the random joint configuration information to the same space, and returning to the step of determining the reachable position of the target manipulator under the configuration of the random joint configuration information;
[0081] Determine the workspace of the target robot arm based on the mapped reachable positions;
[0082] Obtain a preset objective function, and determine a power-assisted optimization model for the target robotic arm based on the objective function and the workspace.
[0083] The workspace of the robot is the set of all position points in the reachable space of the end effector. Due to constraints such as structure and angle changes, the robot cannot reach all working positions in the designed space, so its actual workspace can be expressed as follows:
[0084] S(P)=S f (P)-S p (P) (1)
[0085] Where S f (P) represents the flexible workspace of the robot arm, S p (P) represents the attached workspace of the robot. The range of the workspace is determined by the parameters of the robot, so the workspace power assistance model can be regarded as a multivariate function of the parameters of each mechanism on the exoskeleton robot, so there is the following objective function:
[0086] F(H,L,θ)=maxS(P) (2)
[0087] The constraints are the angles of each joint and the length of each link. Assuming that the robot has N joints, we have:
[0088] θ imin ≤θ i ≤θ imax (i=1,2,…,N) (3)
[0089] L imin ≤L i ≤L imax (i=1,2,…,N) (4)
[0090] The robot workspace model can be solved by the Monte Carlo algorithm. Monte Carlo algorithm is a statistical simulation method that solves approximate solutions to problems based on statistical experiments and random simulations of random variables. By setting parameters and value ranges, the robot traverses all possible values, and the area reached by the end effector is the workspace.
[0091] The process of the Monte Carlo algorithm includes: defining the range of joint angles and link lengths; randomly generating a set of joint angles and link configurations; calculating the position of the end effector under the current configuration; recording the reachable positions; repeating the above process multiple times to cover the possible workspace; analyzing the recorded data to determine the range and shape of the workspace.
[0092] 104. Based on the preset multi-agent deep reinforcement learning algorithm and the power assistance optimization model of the target robotic arm, the target robotic arm is powered.
[0093] The multi-agent deep reinforcement learning algorithm is a method that combines deep learning, reinforcement learning, and multi-agent systems to solve the problem of cooperation and competition among multiple agents in a complex environment. This algorithm usually involves the following key concepts:
[0094] Agent: In a multi-agent system, each agent can perceive part of the state of the environment and take actions in the environment. State: The state of the environment in which the agent is located, which can be a partial observation (local state) or a complete observation (global state). Action: The actions that the agent can take in each state. The action space can be discrete or continuous. Reward: The feedback obtained after the agent takes a certain action, which is used to guide the learning process of the agent. Policy: The mapping of the agent from state to action. The policy can be deterministic or stochastic. Value Function: Predicts the cumulative reward obtained after adopting a certain policy. Q-Function: Predicts the expected return of taking a specific action in a given state. Deep Learning Model: Use deep neural networks to approximate policies or value functions because they can handle high-dimensional input data. Reinforcement Learning Algorithms: Such as Q-Learning, Policy Gradient Methods, Actor-Critic Methods, etc., are used for agents to learn optimal policies through interaction with the environment. Multi-Agent Learning: Consider the interaction between multiple agents, which may involve cooperation, competition, or mixed modes. Communication mechanism: The information exchange between agents can be direct communication or indirect interaction through the environment.
[0095] In the present application, each joint of the target robotic arm can be regarded as an intelligent agent, and the auxiliary strategy corresponding to the intelligent agent is output through the preset multi-agent deep reinforcement learning algorithm to perform power assistance on the target robotic arm. That is, optionally, in some embodiments of the present application, the step of "power assistance to the target robotic arm based on the preset multi-agent deep reinforcement learning algorithm and the power assistance optimization model of the target robotic arm" can specifically include:
[0096] Determine each joint in the target robot arm as an intelligent agent;
[0097] Based on the preset multi-agent deep reinforcement learning algorithm, output the auxiliary strategy corresponding to the intelligent agent;
[0098] In response to the real-time position change of the end effector of the target robot arm, the target robot arm is powered assisted according to the power assistance optimization model of the target robot arm and the assistance strategy corresponding to the intelligent agent.
[0099] A multi-agent learning system is a collection of multiple agents. Each agent interacts with the environment through a reward and punishment mechanism to obtain the optimal strategy. At this time, the change of the global state is related to the joint actions of each agent. The joint parameters of the exoskeleton robot arm are regarded as an agent, and the scope of the final workspace is the collection of the decision-making behaviors of each agent.
[0100] To this end, this application uses a multi-agent deep reinforcement learning algorithm based on SAC (Soft Actor Critic, a deep reinforcement learning algorithm) to achieve optimization of robot arm power assistance. Based on the SAC algorithm model, it introduces the centralized training and decentralized execution framework Centralized Training with Decentralized Execution (CTDE) for expansion, further introduces the agent communication mechanism, and constructs the Multi Agent Soft Actor Critic (MASAC) algorithm, which has good applicability to unstable environments.
[0101] The SAC algorithm uses entropy regularization to encourage exploration, balancing exploration and exploitation by adding an entropy term to the reward function.
[0102] Different from the traditional hard policy (deterministic policy), SAC learns a soft policy, that is, for a given state, it can output a probability distribution of actions.
[0103] SAC consists of two main components: Actor and Critic. The Actor is responsible for generating actions, and the Critic is responsible for evaluating the performance of the current policy. The Actor is a parameterized policy that outputs a probability distribution of all possible actions in a given state. The Critic evaluates the performance of the current policy relative to the optimal policy. SAC uses two Critic networks (double Critic structure) to estimate the value function more stably. The Critic network learns the expected reward of taking an action in a given state. Encouraging policy diversity by minimizing the entropy term promotes exploration. SAC uses the policy gradient theorem to update the Actor network so that the generated actions can not only obtain high rewards but also maintain high entropy. The Critic network is updated by minimizing the consistency error of the outputs of the two Critic networks. The SAC algorithm designs a variety of mechanisms to improve the stability of the training process, such as using reparameterization techniques and soft update strategies.
[0104] For example, specifically, each joint of the robotic arm is regarded as an independent agent. Each agent observes the state information of its joint such as angle, velocity and acceleration. The action space of each agent is defined as the torque output of the motor or the change in the rotation angle of the joint. A reward function is designed to motivate the agents to work together, for example, based on the degree to which the end effector of the robotic arm approaches the target position. A MASAC algorithm based on Soft Actor-Critic (SAC) is implemented to enable each agent to learn the optimal strategy. The agents share the position information and action effects of the end effector during the centralized training phase to learn the global optimal strategy. Information exchange between agents is achieved through communication mechanisms (such as gate mechanisms and GRU principles) to coordinate actions. Each agent learns the best action under different states according to the MASAC algorithm to maximize the cumulative reward. When the position of the end effector changes, each agent responds quickly according to the current state and the learned strategy, that is, optionally, in some embodiments of the present application, the step of "in response to the real-time position change of the end effector of the target robotic arm, according to the power assistance optimization model of the target robotic arm and the assistance strategy corresponding to the agent, the target robotic arm is powered", specifically may include:
[0105] Initialize the agent network corresponding to the agent;
[0106] Get the preset reward function;
[0107] Train the agent network corresponding to the agent according to the preset multi-agent deep reinforcement learning algorithm and reward function;
[0108] According to the trained agent network, the auxiliary strategy corresponding to the agent is output.
[0109] For each agent, initialize its neural network parameters. These networks include Actor network and Critic network. Actor network is used to generate actions, Critic network is used to evaluate actions, and a reward function is defined, which gives positive or negative feedback based on the performance of the agent (such as approaching the goal, completing the task, etc.). Use a multi-agent deep reinforcement learning algorithm, such as MASAC, combined with the reward function to train the network of each agent. The agent learns the optimal strategy through interaction with the environment.
[0110] Among them, the objective function of the maximum entropy algorithm SAC algorithm is:
[0111]
[0112] The SAC algorithm updates the ac-tor network by minimizing the KL divergence, and the following formula is the minimization target:
[0113]
[0114] The sampling process is moved out of the computational graph by re-parameterizing, and the optimization goal after re-parameterizing is:
[0115] J π (θ) = E[λlog(π θ (a t (s t ,ξ)|s t ))-Q β (s t ,a t (s t ,ξ))] (7)
[0116] Where E is the mathematical expectation, T is the total step length of each iteration, and s t ,a t are the state and action of the agent at time t, respectively, π is the trajectory distribution under strategy π, r(s t ,a t ) is the reward mechanism, H(·) is the policy entropy value, θ, β are network parameters, and D is the experience replay pool.
[0117] By introducing CTDE, the SAC algorithm is improved. During the training process, the critic network adds information from other individuals, but during the execution process, the actor network only uses the private observations of the agent to make decisions. For N agents, the objective function of the i-th agent updating the actor network is:
[0118]
[0119] The critic network parameters are updated by the Bellman error:
[0120]
[0121] After the network parameters are updated, the target actor and target critic networks are soft-updated through the hyperparameter τ to ensure the stability of the algorithm:
[0122]
[0123] In the MASAC network, each agent communicates with each other based on the gate mechanism and GRU principle. The communication model is shown in the figure below. Before execution, each agent accesses the communication device and performs read and write operations to obtain information and update the device content. Figure 3 As shown, agent 1 communicates with agent i through a communication device, and the action information of agent 1 is transmitted to agent i through the communication device. Agent i adjusts its own actions and strategies according to the action information transmitted by agent 1, and transmits the adjusted actions to agent 1. Agent 1 adjusts its own actions and strategies according to the action information transmitted by agent i, thereby realizing the linkage between agents.
[0124] After training, each agent's Actor network is able to output the best action in a given state to form an auxiliary strategy. These strategies guide the agent on how to choose the best action based on the current state and goal. During the training process, the performance of the agent is continuously evaluated, and the reward function or network structure is adjusted as needed. The trained agent network is applied to the actual robot arm control problem to achieve power assistance. In actual applications, the agent can continue to optimize its strategy based on real-time feedback to adapt to the dynamically changing environment.
[0125] Through this process, each agent is able to learn how to collaborate with other agents in complex environments to achieve common goals, such as improving the efficiency and precision of a robotic arm’s operations.
[0126] From the above, it can be seen that the embodiment of the present application provides a method for power assistance of a mechanical arm. After obtaining the joint information of the target mechanical arm and the connection information between the joints of the mechanical arm, a connecting rod coordinate system model of the mechanical arm is constructed based on the joint information and the connection information. Then, according to the connecting rod coordinate system model and the preset function, the power assistance optimization model of the target mechanical arm is determined. Finally, based on the preset multi-agent deep reinforcement learning algorithm and the power assistance optimization model of the target mechanical arm, the target mechanical arm is powered. Assist. In the scheme of power assistance of the mechanical arm provided in the present application, based on the joint information and the connection information, a connecting rod coordinate system model of the mechanical arm is constructed. Then, according to the connecting rod coordinate system model and the preset function, the power assistance optimization model of the target mechanical arm is determined. Finally, based on the preset multi-agent deep reinforcement learning algorithm and the power assistance optimization model of the target mechanical arm, intelligent power assistance is performed when the target mechanical arm performs a task, so as to avoid the mechanical arm providing erroneous power output due to manual operation, thereby affecting the execution of the task. Thus, the intelligence level of the mechanical arm is improved, thereby improving the construction efficiency and operation quality.
[0127] In order to better implement the robot arm power assist method of the embodiment of the present application, the embodiment of the present application also provides a robot arm power assist device, wherein the meaning of the terms is the same as that in the above-mentioned robot arm power assist system, and the specific implementation details can refer to the description in the system embodiment.
[0128] See also Figure 4 , Figure 4 A schematic diagram of the structure of a mechanical arm power assist device provided in an embodiment of the present application, wherein the mechanical arm power assist device may specifically include an acquisition module 201, a construction module 202, a determination module 203 and an auxiliary module 204, which may be specifically as follows:
[0129] The acquisition module 201 is used to determine the joint information of the target robotic arm and the connection information between the joints of the robotic arm.
[0130] The construction module 202 is used to construct a link coordinate system model of the robot arm based on the joint information and the connection information.
[0131] Optionally, in some embodiments of the present application, the construction module 202 may specifically include:
[0132] A first determining unit, used to determine a base of a target robotic arm;
[0133] The construction unit is used to construct a link coordinate system model of the target robot arm based on the joint information and the connection information and the base.
[0134] Optionally, in some embodiments of the present application, the construction unit can be specifically used to: establish a link coordinate system of the target robotic arm based on joint information and connection information; determine the position vector of the end link of the target robotic arm based on the joint information and the link coordinate system; and construct a link coordinate system model of the target robotic arm based on the position vector and the base.
[0135] The determination module 203 is used to determine the power-assisted optimization model of the target robotic arm according to the connecting rod coordinate system model and the preset function.
[0136] Optionally, in some embodiments of the present application, the determination module 203 may specifically include:
[0137] An acquisition unit, used for acquiring random joint configuration information of the target robotic arm according to the link coordinate system model and a preset function;
[0138] A second determination unit, used to determine a reachable position of the target manipulator under the configuration of the random joint configuration information;
[0139] The third determination unit is used to determine the power-assisted optimization model of the target robotic arm according to the reachable position of the target robotic arm under the configuration of the random joint configuration information.
[0140] Optionally, in some embodiments of the present application, the third determination unit can be specifically used to: map the reachable position of the target robotic arm under the configuration of the random joint configuration information to the same space, and return to execute the step of determining the reachable position of the target robotic arm under the configuration of the random joint configuration information; determine the working space of the target robotic arm based on the mapped reachable position; obtain a preset objective function, and determine the power-assisted optimization model of the target robotic arm based on the objective function and the working space.
[0141] The auxiliary module 204 is used to provide power assistance to the target robotic arm based on a preset multi-agent deep reinforcement learning algorithm and a power assistance optimization model of the target robotic arm.
[0142] Optionally, in some embodiments of the present application, the auxiliary module 204 may specifically include:
[0143] A fourth determination unit, used for determining each joint in the target robotic arm as an intelligent agent;
[0144] The output unit is used to output the auxiliary strategy corresponding to the intelligent agent based on the preset multi-agent deep reinforcement learning algorithm;
[0145] The auxiliary unit is used to respond to the real-time position change of the end effector of the target robot arm and provide power assistance to the target robot arm according to the power assistance optimization model of the target robot arm and the assistance strategy corresponding to the intelligent agent.
[0146] Optionally, in some embodiments of the present application, the output unit can be specifically used to: initialize the agent network corresponding to the agent; obtain a preset reward function; train the agent network corresponding to the agent according to a preset multi-agent deep reinforcement learning algorithm and a reward function; and output the auxiliary strategy corresponding to the agent based on the trained agent network.
[0147] The embodiment of the present application provides a power assist device for a robotic arm. After the acquisition module 201 acquires the joint information of the target robotic arm and the connection information between the joints of the robotic arm, the construction module 202 constructs a connecting rod coordinate system model of the robotic arm based on the joint information and the connection information. Then, the determination module 203 determines the power assist optimization model of the target robotic arm according to the connecting rod coordinate system model and the preset function. Finally, the auxiliary module 204 performs power assist on the target robotic arm based on the preset multi-agent deep reinforcement learning algorithm and the power assist optimization model of the target robotic arm. In the power assist scheme for the robotic arm provided in the present application, a connecting rod coordinate system model of the robotic arm is constructed based on the joint information and the connection information. Then, the power assist optimization model of the target robotic arm is determined according to the connecting rod coordinate system model and the preset function. Finally, based on the preset multi-agent deep reinforcement learning algorithm and the power assist optimization model of the target robotic arm, intelligent power assist is performed when the target robotic arm performs a task, so as to avoid the robotic arm providing erroneous power output due to manual operation, thereby affecting the execution of the task. Thus, the intelligence level of the robotic arm is improved, thereby improving the construction efficiency and operation quality.
[0148] In addition, the present application also provides an electronic device, such as Figure 5 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:
[0149] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0150] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.
[0151] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and the robot arm power assistance method by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.
[0152] The electronic device also includes a power supply 303 for supplying power to each component. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 303 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.
[0153] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0154] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby realizing various functions, as follows:
[0155] Acquire the joint information of the target robotic arm and the connection information between the joints of the robotic arm; construct the connecting rod coordinate system model of the robotic arm based on the joint information and the connection information; determine the power-assisted optimization model of the target robotic arm according to the connecting rod coordinate system model and the preset function; perform power-assisted operation on the target robotic arm based on the preset multi-agent deep reinforcement learning algorithm and the power-assisted optimization model of the target robotic arm.
[0156] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0157] After acquiring the joint information of the target mechanical arm and the connection information between the mechanical arm joints, the embodiment of the present application constructs a connecting rod coordinate system model of the mechanical arm based on the joint information and the connection information, then determines the power-assisted optimization model of the target mechanical arm according to the connecting rod coordinate system model and the preset function, and finally, based on the preset multi-agent deep reinforcement learning algorithm and the power-assisted optimization model of the target mechanical arm, the target mechanical arm is powered. In the scheme of power assistance of the mechanical arm provided in the present application, based on the joint information and the connection information, a connecting rod coordinate system model of the mechanical arm is constructed, then determines the power-assisted optimization model of the target mechanical arm according to the connecting rod coordinate system model and the preset function, and finally, based on the preset multi-agent deep reinforcement learning algorithm and the power-assisted optimization model of the target mechanical arm, intelligent power assistance is performed when the target mechanical arm performs a task, so as to avoid the mechanical arm providing erroneous power output due to manual operation, thereby affecting the execution of the task, thereby improving the intelligence level of the mechanical arm, thereby improving the construction efficiency and operation quality.
[0158] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0159] To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the robot arm power assistance methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:
[0160] Acquire the joint information of the target robotic arm and the connection information between the joints of the robotic arm; construct the connecting rod coordinate system model of the robotic arm based on the joint information and the connection information; determine the power-assisted optimization model of the target robotic arm according to the connecting rod coordinate system model and the preset function; perform power-assisted operation on the target robotic arm based on the preset multi-agent deep reinforcement learning algorithm and the power-assisted optimization model of the target robotic arm.
[0161] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0162] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0163] Since the instructions stored in the storage medium can execute the steps in any one of the robot arm power assistance methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the robot arm power assistance methods provided in the embodiments of the present application can be achieved. Please see the previous embodiments for details and will not be repeated here.
[0164] The above is a detailed introduction to a robotic arm power assistance method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for power assistance of a robotic arm, It is characterized in that include: Acquire joint information of a target robotic arm and connection information between the joints of the robotic arm; Based on the joint information and the connection information, construct a link coordinate system model of the robotic arm; Determining a power-assisted optimization model of the target robotic arm according to the connecting rod coordinate system model and a preset function; Based on a preset multi-agent deep reinforcement learning algorithm and a power assistance optimization model of the target robotic arm, the target robotic arm is powered.
2. The robot arm power assist method according to claim 1, It is characterized in that The step of constructing a link coordinate system model of the target robotic arm based on the joint information and the connection information includes: Determining a base of the target robotic arm; According to the joint information and the connection information, a link coordinate system model of the target robotic arm is constructed based on the base.
3. The robot arm power assist method according to claim 2, It is characterized in that The method of constructing a link coordinate system model of the target mechanical arm based on the joint information and the connection information and taking the base as a reference includes: Establishing a link coordinate system of the target robotic arm according to the joint information and the connection information; Determine the position vector of the end link of the target robot arm according to the joint information and the link coordinate system; Based on the position vector and the base, a link coordinate system model of the target robotic arm is constructed.
4. The robot arm power assist method according to claim 1, It is characterized in that Determining the power-assisted optimization model of the target robotic arm according to the connecting rod coordinate system model and the preset function includes: According to the link coordinate system model and the preset function, obtaining random joint configuration information of the target robotic arm; Determine a reachable position of the target robotic arm under the configuration of the random joint configuration information; A power-assisted optimization model of the target robotic arm is determined according to a reachable position of the target robotic arm under the configuration of the random joint configuration information.
5. The robot arm power assist method according to claim 4, It is characterized in that The step of determining the power-assisted optimization model of the target robotic arm according to the reachable position of the target robotic arm under the configuration of the random joint configuration information comprises: Mapping the reachable position of the target manipulator under the configuration of the random joint configuration information to the same space, and returning to execute the step of determining the reachable position of the target manipulator under the configuration of the random joint configuration information; Determining a working space of the target robotic arm according to the mapped reachable positions; A preset objective function is obtained, and a power-assisted optimization model of the target robotic arm is determined according to the objective function and the workspace.
6. The robot arm power assist method according to claim 1, It is characterized in that The method of performing power assistance on the target robotic arm based on the preset multi-agent deep reinforcement learning algorithm and the power assistance optimization model of the target robotic arm includes: Determine each joint in the target robotic arm as an intelligent agent; Based on the preset multi-agent deep reinforcement learning algorithm, output the auxiliary strategy corresponding to the agent; In response to the real-time position change of the end effector of the target robotic arm, the target robotic arm is powered assisted according to the power-assisted optimization model of the target robotic arm and the assistance strategy corresponding to the intelligent agent.
7. The robot arm power assist method according to claim 6, It is characterized in that The output of the auxiliary strategy corresponding to the agent based on the preset multi-agent deep reinforcement learning algorithm includes: Initializing an agent network corresponding to the agent; Get the preset reward function; Training the agent network corresponding to the agent according to a preset multi-agent deep reinforcement learning algorithm and a reward function; According to the trained agent network, an auxiliary strategy corresponding to the agent is output.
8. A mechanical arm power assist device, It is characterized in that include: An acquisition module, used to determine the joint information of the target robotic arm and the connection information between the joints of the robotic arm; A construction module, used to construct a link coordinate system model of the robotic arm based on the joint information and the connection information; A determination module, used to determine a power-assisted optimization model of the target robotic arm according to the connecting rod coordinate system model and a preset function; An auxiliary module is used to provide power assistance to the target robotic arm based on a preset multi-agent deep reinforcement learning algorithm and a power assistance optimization model of the target robotic arm.
9. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program according to the steps of the robot arm power assistance method as described in any one of claims 1 to 7.
10. A storage medium, It is characterized in that A computer program is stored which can be loaded by a processor and execute the robot arm power assistance method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Mechanical arm path planning method and device, electronic equipment and storage medium
CN113459109A
Mechanical arm anti-interference motion planning method based on multi-agent reinforcement learning
CN114083539A
Mechanical arm fault-tolerant obstacle avoidance method based on deep reinforcement learning
CN116021515A
Multi-agent deep reinforcement learning training method based on PPO
CN116306979A
Multi-mechanical-arm and multi-task target obtaining method, device and equipment and storage medium
CN116690588A
