Wheel type mobile mechanical arm coordination control method, system and device

Abstract

The invention relates to the technical field of unmanned system multi-agent control, in particular to a wheel type mobile mechanical arm coordination control method, system and device. Comprising the following steps that a wheeled mobile mechanical arm model is established, and a target reachability area is solved; the operability index of the wheeled mobile mechanical arm is calculated; calculating the maximum linear velocity and angular velocity of the wheel type mobile mechanical arm to obtain a velocity ellipsoid model of the wheel type mobile mechanical arm; designing a multi-agent combined state space, and designing a multi-agent combined action space; constructing a control algorithm flow; setting a task scene of a coordination control algorithm of the wheeled mobile manipulator, setting training hyper-parameters, and designing task evaluation indexes; and carrying out training setting, and recording the success rate of the task and the validity of the process when the training period is finished. According to the method, the problem that the model structure is complex and depends on kinematics equation solving or objective function optimization under the high-dimensional condition is solved. According to the invention, the technical effect of reducing complexity calculation is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent control of unmanned systems, and in particular to a coordinated control method, system and device for a wheeled mobile robotic arm. Background Art

[0002] The wheeled mobile manipulator is a new type of intelligent mobile robot system with active operation capability, which is composed of one or more manipulators installed on a wheeled unmanned platform. Compared with the traditional wheeled unmanned platform, the wheeled mobile manipulator can enhance the terrain passing ability of the wheeled unmanned platform, assist in improving the stability of the wheeled unmanned platform, and greatly improve the autonomous ability of the wheeled unmanned platform, making the wheeled unmanned platform no longer a simple means of transportation, but an important part of the current robot application. At present, the main application scenarios of wheeled mobile manipulators are: home services, factory manufacturing, medical assistance, off-road road passability detection, dangerous object recovery, mine clearance and explosion disposal, search and rescue and disaster relief, spraying and disinfection, casualty evacuation, combat support, etc.

[0003] Although the combination of wheeled ground unmanned platforms and robotic arms has obvious advantages, it also brings new challenges. Among the many challenges, a major problem of wheeled mobile robotic arms is how to achieve coordinated control of the vehicle and the arm to complete the operation task. In the past few decades, researchers have conducted a lot of research on the coordinated control methods of wheeled mobile robotic arms, but most of the proposed algorithms are traditional control methods, which rely on solving kinematic equations or optimizing objective functions in the case of complex robot model structures and high dimensions, which leads to high computational costs. With the breakthroughs in reinforcement and imitation learning in recent years, new possibilities have been brought to the solution of the above problems. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a coordinated control method, system and device for a wheeled mobile mechanical arm to achieve coordinated control of a wheeled mobile mechanical arm. The present invention provides a coordinated control method for a wheeled mobile mechanical arm, comprising the following steps: S1: establishing a wheeled mobile manipulator model, and solving a target reachability area according to the wheeled mobile manipulator model; S2: According to the wheeled mobile manipulator model, calculating the Jacobian matrix of the wheeled mobile manipulator and the operability index of the wheeled mobile manipulator; S3: Calculate the maximum linear velocity and the maximum angular velocity of the wheeled mobile robotic arm according to the Jacobian matrix of the wheeled mobile robotic arm, and obtain a velocity ellipsoid model of the wheeled mobile robotic arm; S4: constructing a joint state space, a joint action space and a reward function according to the target reachability area, the operability index and the velocity ellipsoid model; S5: constructing a coordinated control algorithm flow of a wheeled mobile manipulator based on the DE-MADDPG algorithm according to the joint state space, the joint action space and the reward function; S6: Setting the task scenario of the coordinated control algorithm process of the wheeled mobile manipulator based on the IsaacSim platform, setting the training hyperparameters, and designing the task evaluation indicators; S7: Perform training settings based on the task scenario of the wheeled mobile robotic arm coordinated control algorithm process, the training hyperparameters and the task evaluation indicators, record the success rate of the task at the end of each training cycle, and judge the effectiveness of the wheeled mobile robotic arm coordinated control algorithm process.

[0005] According to a coordinated control method for a wheeled mobile mechanical arm provided by the present invention, step S1 comprises: S11: With the center of mass of the unmanned vehicle platform as the origin, the horizontal direction directly behind the unmanned vehicle platform as the positive direction of the x-axis, and the vertical direction upward to the ground as the positive direction of the z-axis, the positive direction of the y-axis can form an axial scanning coordinate system of the right-hand system with the positive directions of the x-axis and the z-axis. The axial scanning coordinate system is a wheeled mobile robotic arm model; S12: Look up the wheeled mobile robotic arm operation manual to obtain the working radius of the wheeled mobile robotic arm ; S13: According to the z-axis coordinate of the target point , the z-axis coordinate of the first joint center of the wheeled mobile robot Calculate the angle between the straight line perpendicular to the ground starting from the target point and the straight line extending from the target to the center of the first joint of the robot arm : ; S14: Get the distance from the center of mass of the unmanned platform to the first joint of the robotic arm on the x-axis. , calculate the reachability area diameter Size: The size of the reachable area is the reachable area diameter A sphere with a diameter of .

[0006] According to a coordinated control method for a wheeled mobile mechanical arm provided by the present invention, step S2 comprises: S21: Calculate the Jacobian matrix of the wheeled mobile manipulator according to the wheeled mobile manipulator model : in, is the end effector position Joint angle The partial guide, is the degree of freedom of the operating space, is the degree of freedom of the joint space; S22: According to the Jacobian matrix of the wheeled mobile manipulator , calculate the operability index in, represents the determinant evaluation, for The transposed matrix of .

[0007] According to a coordinated control method for a wheeled mobile mechanical arm provided by the present invention, step S3 comprises: S31: According to the Jacobian matrix of the wheeled mobile manipulator , The upper part of represents the linear velocity of the end effector of the wheeled mobile robot. The lower part of represents the angular velocity of the end effector of the wheeled mobile robot. Translated as: in, is the linear velocity part, is the angular velocity part; S32: Determine the linear velocity of the end of the wheeled mobile robot : in, yes The transposed matrix of S33: Determine the angular velocity of the end of the wheeled mobile robot , in, yes The transposed matrix of and Draw the velocity ellipsoid model.

[0008] According to a coordinated control method for a wheeled mobile mechanical arm provided by the present invention, step S4 comprises: S41: According to the structure of the wheeled mobile robotic arm, the intelligent agents are divided into two agents: an unmanned vehicle agent and a robotic arm agent. According to the reachability region and the velocity ellipsoid model, the joint state space of the wheeled mobile manipulator is constructed as ,in, is the displacement of the unmanned vehicle agent relative to the starting position and the yaw in the global frame, are the linear and angular velocities of the joints of the robot agent, is the position coordinate of the end effector of the robot arm, is the target position coordinate; , is the set of real numbers; S42: Constructing a joint action space of the unmanned vehicle agent and the robotic arm agent according to the reachability region and the velocity ellipsoid model ,in is the action space of the unmanned vehicle agent, including the linear velocity and angular velocity of the unmanned vehicle agent. is the action space of the robot agent, including the linear velocity and angular velocity of the robot agent, and the data of the joint action space is normalized to ; S43: The reward function includes a global reward function, a collision penalty function, a reachability area local reward function and a maneuverability local reward function. Constructing the reward function includes the following steps: S431: Constructing a global reward function : in, is the first control coefficient of the reward range, is the second control coefficient of the reward amplitude, is the distance between the end effector of the wheeled mobile robot and the target point, represents the distance between two points, Indicates that the function takes the minimum value; S432: Construct collision penalty function : ; S433: Constructing a local reward function for the reachability region of a wheeled mobile manipulator : in, is the distance between the unmanned vehicle platform and the target point; S434: Constructing a local reward function for maneuverability of a wheeled mobile manipulator : in, is the operating threshold.

[0009] According to a coordinated control method for a wheeled mobile mechanical arm provided by the present invention, step S5 comprises: S51: Initialize the global evaluation network , initialize the global target evaluation network , initialize the action network of the unmanned vehicle agent , the action network of the robot agent , Evaluation network of unmanned vehicle agent and the evaluation network of the robot agent , generate the target position within the set range and set the initial state of the wheeled mobile manipulator; S52: Get the initial state of the simulation environment: , , ,in, Indicates that at time step When is the state of the joint state space, Indicates that at time step The action space of the unmanned vehicle agent is Indicates that at time step The action space of the robot agent is represents the policy function of the unmanned vehicle agent, Indicates that at time step The observed input data of the unmanned vehicle agent is Indicates that at time step is the policy function of the robot agent, Indicates that at time step The observation input data of the robot agent; S53: Perform joint actions ,in, , at time step Get the collision penalty function when , at time step The local reward function of the reachable area when and at time step The local reward function for maneuverability ; S54: Build storage sample , and store the stored samples into the experience pool, where Indicates that at time step The state of the joint state space when S55: training a global evaluation network, randomly sampling from the experience pool Samples make up a batch of samples : in, , is the sample ordinal number, , the symbol in the formula Indicates an assignment operation, that is, assigning the result of the expression on the right side of the equal sign to the variable on the left side. Represented in the joint state space Take the following The joint action of samples The collision reward function obtained after represents the action taken by the unmanned vehicle agent at the next time step, represents the action taken by the robot agent in the next time step, Indicates that the unmanned vehicle agent is in The observation data for the next time step in samples, Indicates that the robot agent is in the The observation data for the next time step in samples, is the policy function of the unmanned vehicle agent under the new policy at the next time step, is the policy function of the robot agent under the new policy at the next time step; For the The joint state space at the next time step when the sample is S56: Calculating target value and minimizing loss : in, It is The target value of samples, Indicated in In the samples, the agent The global reward obtained at the time step, the agents include the unmanned vehicle agent and the robotic arm agent, , is the discount factor, Indicates that the new strategy Next, use the action network parameters The action value function, Indicated in In the samples, the agent The observed data at the next time step, Indicates that the new strategy Next, the intelligent agent Observation data in the new state actions taken in It is an intelligent agent The global evaluation network uses the global evaluation network parameters to estimate the value of a given state and action; use and Update the global evaluation network and global target evaluation network : in, Indicates a replacement operation. is the evaluation network learning rate, is the global target network parameter, is the inertia update rate; is the loss function About evaluating network parameters The gradient of S57: Initialize the agent local evaluation network and agent local goal evaluation network , randomly sampled from the experience pool Samples form a small batch of samples ,in, is the second sample ordinal number, ;in, , In the joint state space Take the following The joint action of samples The collision reward function obtained after Indicates that the unmanned vehicle agent is in The observation data for the next time step in samples, Indicates that the robot agent is in the The observation data for the next time step in samples; For the The joint state space at the next time step when the sample is S58: Calculate local target value and local minimization loss : It is The local target value of samples, Indicated in In the samples, the agent The global reward obtained at the time step, It is an intelligent agent The global evaluation network uses the global evaluation network parameters to estimate the value of a given state and action; use and Update the agent's local evaluation network and local target evaluation network : in, is the local target network parameter; is the loss function On the evaluation of local evaluation network parameters The gradient of S59: Computational Agents i Strategy parameters : in, is the update rule for the policy parameters, It’s a strategy About strategy parameters The gradient of Among the samples, Observation data of an agent Agents under Joint action , It is a global evaluation network On the joint actions of agents The gradient of It is The local evaluation function of an agent On the joint actions of agents The gradient of It is Among the samples, Observation data of each agent; S510: Update and : in, is the agent strategy parameter for the next time step.

[0010] According to a coordinated control method for a wheeled mobile mechanical arm provided by the present invention, step S6 comprises: S61: Set up task scenarios, including simple environments without obstacles and complex environments with multiple obstacles; S62: Setting training hyper parameters, the training hyper parameters include: discount factor, experience pool size, inertia update rate, simulation time step, maximum time step of a single round, evaluation network learning rate, action network learning rate; S63: Design task evaluation indicators include average operability, average operability and success rate.

[0011] According to a coordinated control method for a wheeled mobile mechanical arm provided by the present invention, step S7 comprises: S71: The training is set as follows: the maximum length of the round is set according to the task scenario, training hyperparameters and task evaluation indicators, the initial position of the wheeled mobile manipulator is kept consistent, the design and call of the coordinated control algorithm of the wheeled mobile manipulator is implemented based on the RLGames library, two independent multi-layer perceptrons are used for DE-MADDPG, the input of the coordinated control algorithm of the wheeled mobile manipulator is the same, and in the simulation training environment, the running frequency of the wheeled mobile manipulator simulation is set, and the interaction frequency of the coordinated control algorithm of the wheeled mobile manipulator is set; S72: Train and evaluate the coordinated control algorithm for a wheeled mobile manipulator in two different task scenarios, construct a theoretically infinite test set, in which the target position is randomly generated within a preset range, and record the success rate of the task at the end of each training cycle, wherein the success rate is determined by calculating the percentage of successful cases in a validation set that includes a set number of independent trials of the validation set, and the target point position of each trial is randomly generated within a specified range, and the success rate is recorded after each training cycle has passed a set number of training cycles, each environment configuration is independently trained a threshold number of times, and each training includes a set number of training cycles training cycles, and generates a test result including a random target position; S73: According to the success rate of the test results and the observation results of the operability of the robot arm, the effectiveness of the coordinated control algorithm process of the wheeled mobile robot arm is judged. If the effectiveness is lower than the set threshold, return to step S5 for retraining.

[0012] The present invention also provides a wheeled mobile mechanical arm coordinated control system, comprising: Data acquisition module: establish a wheeled mobile manipulator model, solve the target reachability area according to the wheeled mobile manipulator model, calculate the Jacobian matrix of the wheeled mobile manipulator and the operability index of the wheeled mobile manipulator according to the wheeled mobile manipulator model, calculate the maximum linear velocity of the wheeled mobile manipulator and the maximum linear velocity of the wheeled mobile manipulator according to the Jacobian matrix of the wheeled mobile manipulator, and obtain the velocity ellipsoid model of the wheeled mobile manipulator; Model building module: construct a joint state space, a joint action space and a reward function according to the target reachability area, the operability index and the velocity ellipsoid model; construct a coordinated control algorithm flow of a wheeled mobile manipulator based on the DE-MADDPG algorithm according to the joint state space, the joint action space and the reward function; Model training module: Build a physical simulation training environment for wheeled mobile robotic arms based on IsaacSim, set training parameters, design task evaluation indicators, set up training based on the task scenarios, training hyperparameters and task evaluation indicators of the coordinated control algorithm process of the wheeled mobile robotic arms, and record the success rate of the tasks at the end of each training cycle to judge the effectiveness of the coordinated control algorithm process of the wheeled mobile robotic arms.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a coordinated control method of a wheeled mobile robotic arm as described in any one of the above are implemented.

[0014] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 Flow chart of a coordinated control method of a wheeled mobile robotic arm according to an embodiment of the present invention.

[0017] Figure 2 Schematic diagram of kinematics of an embodiment of the present invention.

[0018] Figure 3 Schematic diagram of the reachability area of ​​a robot arm in a coordinated control method for a wheeled mobile robot arm according to an embodiment of the present invention.

[0019] Figure 4 It is a visual diagram of the operability of the coordinated control method of the wheeled mobile mechanical arm according to an embodiment of the present invention.

[0020] Figure 5 This is a model training diagram of the coordinated control method of the wheeled mobile robotic arm according to an embodiment of the present invention.

[0021] Figure 6Schematic diagram of training results of the coordinated control method of a wheeled mobile robotic arm according to an embodiment of the present invention.

[0022] Figure 7 Comparison of training results of the coordinated control method of the wheeled mobile mechanical arm according to the embodiment of the present invention Figure 1 .

[0023] Figure 8 Comparison of training results of the coordinated control method of the wheeled mobile mechanical arm according to the embodiment of the present invention Figure 2 .

[0024] Fig. 9 4 is a block diagram of a coordinated control system of a wheeled mobile robotic arm according to an embodiment of the present invention.

[0025] Fig.10 This is a schematic structural diagram of the coordinated control device for a wheeled mobile mechanical arm provided by the present invention.

[0026] Reference numerals: 101. Data acquisition module; 102. Model building module; 103. Model training module; 810. Processor; 820. Communication interface; 830. Memory; 840. Communication bus. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0028] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0029] Example Combine the following Figures 1 to 10 Description of the Invention like Figure 1As shown, an embodiment of the present invention provides a coordinated control method for a wheeled mobile mechanical arm, comprising the following steps: S1: establishing a wheeled mobile manipulator model, and solving a target reachability area according to the wheeled mobile manipulator model; S2: According to the wheeled mobile manipulator model, calculating the Jacobian matrix of the wheeled mobile manipulator and the operability index of the wheeled mobile manipulator; S3: Calculate the maximum linear velocity and the maximum angular velocity of the wheeled mobile robotic arm according to the Jacobian matrix of the wheeled mobile robotic arm, and obtain a velocity ellipsoid model of the wheeled mobile robotic arm; S4: Construct joint state space, joint action space and reward function according to target reachability area, maneuverability index and velocity ellipsoid model; S5: According to the joint state space, joint action space and reward function, a coordinated control algorithm flow of a wheeled mobile manipulator based on the DE-MADDPG (Decomposed Multi-Agent Deep Deterministic Policy Gradient) algorithm is constructed; S6: Set the task scenario of the coordinated control algorithm of the wheeled mobile robot based on the IsaacSim platform, set the training hyperparameters, and design the task evaluation indicators.

[0030] S7: Perform training settings based on the task scenarios, training hyperparameters, and task evaluation indicators of the coordinated control algorithm process of the wheeled mobile manipulator. Record the success rate of the task at the end of each training cycle to determine the effectiveness of the coordinated control algorithm process of the wheeled mobile manipulator.

[0031] Specifically, step S1 includes the following steps: Figure 2 As shown, with the center of mass of the unmanned vehicle platform as the origin, the horizontal direction to the rear of the unmanned vehicle platform as the positive direction of the x-axis, and the positive direction of the z-axis perpendicular to the ground as the positive direction, the positive direction of the y-axis can form an axial scanning coordinate system of the right-hand system with the positive directions of the x-axis and the z-axis. This coordinate system is a wheeled mobile robotic arm model.

[0032] Find Frank Panda Operating Manual Wheeled Mobile Manipulator Operating Manual Get the Working Radius of Wheeled Mobile Manipulator .

[0033] According to the z-axis coordinate of the target point , the z-axis coordinate of the first joint center of the wheeled mobile robot Calculate the angle between the straight line perpendicular to the ground starting from the target point and the straight line extending from the target to the center of the first joint of the robot arm : ; Get the distance from the center of mass of the unmanned platform to the first joint of the robotic arm on the x-axis: , calculate the reachability area diameter Size: .

[0034] According to the above steps, we can determine the size of the reachable area. The size of the reachable area is as follows Figure 3 As shown, the reachable area diameter The size of the wheeled mobile robot model and the z-axis coordinate of the target point Determined.

[0035] Specifically, step S2 includes the following steps: according to the wheeled mobile manipulator model, calculating the Jacobian matrix of the wheeled mobile manipulator : in, is the end effector position Joint angle The partial guide, is the degree of freedom of the operating space, is the degree of freedom of the joint space. Since the degree of freedom of the space is 6, the degree of freedom of the robot arm is 6. Therefore, ; .

[0036] According to the Jacobian matrix of the wheeled mobile manipulator , calculate the operability index in, Represents determinant evaluation.

[0037] Specifically, step S3 includes the following steps: according to the Jacobian matrix of the wheeled mobile manipulator , The first three rows represent the linear velocity of the end effector of the wheeled mobile robot. The last three rows represent the angular velocity of the end effector of the wheeled mobile manipulator. The Jacobian matrix of the wheeled mobile manipulator Translated as: in, is the linear velocity part, is the angular velocity part; Solve the linear velocity of the end of the wheeled mobile robot and the angular velocity of the end of the wheeled mobile robot : according to and Draw an operational visualization diagram, such as Figure 4 As shown in the figure, the shape and direction of the linear velocity ellipsoid are given by The size of the ellipsoid reflects the speed capability of the robot in different directions; the shape and direction of the angular velocity ellipsoid are determined by The size of the ellipsoid reflects the angular velocity capability of the robot in different directions. The larger the volume of the ellipsoid, the better the maneuverability of the robot.

[0038] Specifically, step S4 of the present invention further includes: According to the structure of the wheeled mobile robotic arm, it is divided into two agents: an unmanned vehicle agent and a robotic arm agent: According to the reachability region and the velocity ellipsoid model, the joint state space of the wheeled mobile manipulator is constructed as ,in, is the displacement of the unmanned vehicle agent relative to the starting position and the yaw in the global frame, are the linear and angular velocities of the joints of the robot agent, is the position coordinate of the end effector of the robot arm, is the target position coordinate; , is the set of real numbers.

[0039] Constructing the joint action space of the two intelligent agents, the unmanned vehicle platform and the robotic arm ,in is the action space of the unmanned vehicle platform, including the linear speed of the unmanned vehicle platform and angular velocity , is the motion space of the robot, including the linear velocity of the end of the robot and the angular velocity of the end of the robot arm All actions are continuous. To stabilize training and avoid overfitting, the value of the action space is normalized to ; Constructing a global reward function : in, is the first control coefficient of the reward amplitude and is the second control coefficient of the reward amplitude, is the distance between the end effector of the wheeled mobile robot and the target point, represents the distance between two points, Indicates that the function takes the minimum value; Constructing a collision penalty function : ; Constructing a local reward function for the reachability region of a wheeled mobile manipulator : in, is the distance between the unmanned vehicle platform and the target point, and the action Consists of two continuous values: Linear speed and angular velocity , the unmanned platform can localize itself in the global map to determine its relative position with respect to the target location; Constructing a local reward function for maneuverability of a wheeled mobile robot : in, is the operating threshold.

[0040] Specifically, initialize the global evaluation network , initialize the global target evaluation network , initialize the action network of the unmanned vehicle agent , the action network of the robot agent , Evaluation network of unmanned vehicle agent and the evaluation network of the robot agent , generate the target position within the set range and set the initial state of the wheeled mobile robot.

[0041] Get the initial state of the simulation environment: , , ,in, Indicates that at time step The state of the joint state space is Indicates that at time step The action space of the autonomous vehicle agent is Indicates that at time step The action space of the robot agent is represents the policy function of the unmanned vehicle agent, Indicates that at time step The observed input data of the unmanned vehicle agent is Indicates that at time step is the policy function of the robot agent, Indicates that at time step The observation input data of the robotic arm agent.

[0042] Perform joint actions ,in, , at time step Get the collision penalty function when , at time step The local reward function of the reachable area when and at time step The local reward function for maneuverability .

[0043] Build storage sample , and store the stored samples into the experience pool, where Indicates that at time step The state of the joint state space.

[0044] Train a global evaluation network, randomly sampling from the experience pool Samples make up a batch of samples : in, , is the sample ordinal number, , the symbol in the formula Indicates an assignment operation, that is, assigning the result of the expression on the right side of the equal sign to the variable on the left side. In the joint state space Take the following The joint action of samples The collision reward function obtained after represents the action taken by the unmanned vehicle agent at the next time step, represents the action taken by the robot agent in the next time step, Indicates that the unmanned vehicle agent is in The observation data for the next time step in samples, Indicates that the robot agent is in the The observation data for the next time step in samples, is the policy function of the unmanned vehicle agent under the new policy at the next time step, is the policy function of the robot agent under the new policy at the next time step; For the The joint state space at the next time step is the sum of the samples.

[0045] Calculating target value and minimizing loss : in, It is The target value of samples, Indicated in In the samples, the agent The global reward obtained at the time step, the agents include the unmanned vehicle agent and the robotic arm agent, , is the discount factor, Indicates that the new strategy Next, use the action network parameters The action value function, Indicated in In the samples, the agent The observed data at the next time step, Indicates that the new strategy Next, the intelligent agent Observation data in the new state actions taken in It is an intelligent agent The global evaluation network uses the global evaluation network parameters to estimate the value of a given state and action; use and Update the global evaluation network and global target evaluation network : in, Indicates a replacement operation. is the evaluation network learning rate, is the global target network parameter, is the inertia update rate; is the loss function About evaluating network parameters gradient.

[0046] Initialize the agent local evaluation network and agent local goal evaluation network , randomly sampled from the experience pool Samples form a small batch of samples ,in, is the second sample ordinal number, ;in, , Represented in the joint state space Take the following The joint action of samples The collision reward function obtained after Indicates that the unmanned vehicle agent is in The observation data for the next time step in samples, Indicates that the robot agent is in the The observation data for the next time step in samples; For the The joint state space at the next time step is the sum of the samples.

[0047] Calculate local objective value and local minimization loss : It is The local target value of samples, Indicated in In the samples, the agent The global reward obtained at the time step, It is an intelligent agent The global evaluation network uses the global evaluation network parameters to estimate the value of a given state and action; use and Update the agent's local evaluation network and local target evaluation network : in, is the local target network parameter; is the loss function On the evaluation of local evaluation network parameters gradient.

[0048] Computational Agents i Strategy parameters : in, is the update rule for the policy parameters, It’s a strategy About strategy parameters The gradient of Among the samples, Observation data of an agent Agents under Joint action , It is a global evaluation network On the joint actions of agents The gradient of It is The local evaluation function of an agent On the joint actions of agents The gradient of It is Among the samples, Observation data of each agent; renew and : in, is the agent strategy parameter for the next time step.

[0049] The specific step S6 includes: setting the task scene to include: task 1: simple environment without obstacles; task 2: complex environment with multiple obstacles. In addition to the self-collision problem, the complex environment with multiple obstacles also needs to consider the collision problem between obstacles and the storage platform; setting the training hyperparameters to: discount factor , experience pool size M = 30000, inertia update rate , simulation time step , the maximum time step of a single round is 500, and the learning rate of the evaluation network is , action network learning rate ; Design task evaluation indicators include average operability, average operability and success rate.

[0050] Table 1 Experimental parameters

[0051] According to the data in Table 1, the embodiment of the present invention uses IsaacSim to build a simulation training environment. Figure 5 As shown. Among them, Figure 5 (a) is a simple barrier-free environment for Task 1. Figure 5 (b) is the complex environment with many obstacles in Task 2.

[0052] Specifically, step S7 training is set as follows: the maximum length of the round is 500 steps, the initial position of the wheeled mobile manipulator is kept consistent, the design and call of the coordinated control algorithm of the wheeled mobile manipulator is implemented based on the RLGames library, two independent multi-layer perceptrons are used for DE-MADDPG, and the input of the coordinated control algorithm of the wheeled mobile manipulator is the same. In the simulation training environment, the operating frequency of the wheeled mobile manipulator simulation is set to 120Hz, and the interaction frequency of the coordinated control algorithm of the wheeled mobile manipulator is set to 60Hz; the coordinated control algorithm of the wheeled mobile manipulator is trained and evaluated in two different task scenarios to build a theoretically infinite test set , where the target position is randomly generated within a preset range, and the success rate of the task is recorded at the end of each training cycle, where the success rate is determined by calculating the percentage of successful cases in a validation set containing 50 independent trials, and the target point position of each trial is randomly generated within the specified range, and the success rate is recorded after every 20 training cycles. Each environment configuration is trained independently 3 times, each training includes 500 training cycles, and a test set containing 100 random target positions is generated; according to the success rate of the test results and the operability observation results of the robot arm, the coordinated control algorithm flow of the wheeled mobile robot arm based on the DE-MADDPG algorithm is evaluated.

[0053] The simulation runs at 120 Hz, the interaction frequency of the reinforcement learning algorithm is set to 60 Hz, and the maximum length of an episode is set to 500 steps. In all experiments, the initial position of the wheeled mobile manipulator remains the same.

[0054] According to the observation of training data, multiple training processes in two tasks show consistent learning curves for the three strategies. Figure 6 As shown. Among them, Figure 6 (a) is a simple barrier-free environment for Task 1. Figure 6 (b) is a complex environment with multiple obstacles in Task 2. In general, the DE-MADDPG-based learning curve proposed in the embodiment of the present invention has the highest learning speed and convergence success rate, that is, better accessibility. Especially in complex task scenarios, the DE-MADDPG-based coordinated control strategy proposed in the embodiment of the present invention has a more obvious convergence speed and success rate.

[0055] For both tasks, we use the average operability of the entire task trajectory to reflect the operability index. Tables 2 and 3 show the experimental results of the simulation environment for 50 tests of the test set. During the evaluation process, the proposed DE-MADDPG-based control strategy achieved the best performance in both task settings, and achieved significant improvements in average operability and success rate compared to the other two control strategies.

[0056] Table 2 Experimental results of reachability and operability in the simulation environment of Task 1

[0057] Table 3 Experimental results of reachability and operability in task 2 simulation environment

[0058] Based on the results of 50 tests in the simulation environment, the Figure 7 and Figure 8 .in, Figure 7 (a) to (d) are simple barrier-free environments for Task 1. Figure 8 (a) to (d) are complex environments with multiple obstacles in Task 2. Horizontal comparison: In the simple environment without obstacles in Task 1, the three control strategies can all achieve the task objectives. The control effects of the two control strategies DE-MADDPG and DDPG (Deep Deterministic Policy Gradient) are significantly better than MADDPG (Multi-Agent Deep Deterministic Policy Gradient). The operability curve shows that DE-MADDPG has significantly improved operability compared with the other two control strategies. In the complex environment with multiple obstacles in Task 2, the control strategies based on MADDPG failed to complete the arrival task. Compared with DDPG, DE-MADDPG has significantly improved control efficiency and operability.

[0059] Longitudinal comparison: The comparison of the two sets of experimental results shows that the control strategy based on MADDPG failed to complete the arrival task in complex environments. Although the control strategy based on DDPG completed the specified tasks in both experimental scenarios, its control efficiency was slow and its operability was poor. In comparison, the control strategy based on DE-MADDPG has higher control efficiency in both environments, significantly improved operability, and better adaptability to different task scenarios.

[0060] like Fig. 9 As shown, Fig. 9 An example of a coordinated control system for a wheeled mobile manipulator is provided, comprising: Data acquisition module 101: Establishing a wheeled mobile manipulator model, solving a target reachability region according to the wheeled mobile manipulator model, calculating the Jacobian matrix of the wheeled mobile manipulator and the operability index of the wheeled mobile manipulator according to the wheeled mobile manipulator model, calculating the maximum linear velocity of the wheeled mobile manipulator and the maximum linear velocity of the wheeled mobile manipulator according to the Jacobian matrix of the wheeled mobile manipulator, and obtaining a velocity ellipsoid model of the wheeled mobile manipulator; Model building module 102: constructing a joint state space, a joint action space and a reward function according to the target reachability area, the operability index and the velocity ellipsoid model, and constructing a coordinated control algorithm flow of a wheeled mobile manipulator based on a DE-MADDPG algorithm according to the joint state space, the joint action space and the reward function; Model training module 103: Build a physical simulation training environment for a wheeled mobile robotic arm based on IsaacSim, set training parameters, design task evaluation indicators, perform training settings based on the task scenarios of the wheeled mobile robotic arm coordinated control algorithm process, the training hyperparameters and the task evaluation indicators, record the success rate of the task at the end of each training cycle, and judge the effectiveness of the wheeled mobile robotic arm coordinated control algorithm process.

[0061] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic equipment may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a coordinated control method for a wheeled mobile manipulator.

[0062] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A coordinated control method for a wheeled mobile robot arm, characterized in that: The following steps are involved: S1: establishing a wheeled mobile manipulator model, and solving a target reachability area according to the wheeled mobile manipulator model; S2: According to the wheeled mobile manipulator model, calculating the Jacobian matrix of the wheeled mobile manipulator and the operability index of the wheeled mobile manipulator; S3: Calculate the maximum linear velocity and the maximum angular velocity of the wheeled mobile robotic arm according to the Jacobian matrix of the wheeled mobile robotic arm, and obtain a velocity ellipsoid model of the wheeled mobile robotic arm; S4: constructing a joint state space, a joint action space and a reward function according to the target reachability area, the operability index and the velocity ellipsoid model; S5: constructing a coordinated control algorithm flow of a wheeled mobile manipulator based on the DE-MADDPG algorithm according to the joint state space, the joint action space and the reward function; S6: Based on the IsaacSim platform, set the task scenario of the coordinated control algorithm process of the wheeled mobile manipulator, set the training hyperparameters, and design the task evaluation indicators; S7: Perform training settings based on the task scenario of the wheeled mobile robotic arm coordinated control algorithm process, the training hyperparameters and the task evaluation indicators, record the success rate of the task at the end of each training cycle, and judge the effectiveness of the wheeled mobile robotic arm coordinated control algorithm process.

2. A coordinated control method for a wheeled mobile mechanical arm according to claim 1, characterized in that: Step S1 includes: S11: With the center of mass of the unmanned vehicle platform as the origin, the horizontal direction directly behind the unmanned vehicle platform as the positive direction of the x-axis, and the vertical direction upward to the ground as the positive direction of the z-axis, the positive direction of the y-axis can form an axial scanning coordinate system of the right-hand system with the positive directions of the x-axis and the z-axis. The axial scanning coordinate system is a wheeled mobile robotic arm model; S12: Look up the wheeled mobile robotic arm operation manual to obtain the working radius of the wheeled mobile robotic arm ; S13: According to the z-axis coordinate of the target point , the z-axis coordinate of the first joint center of the wheeled mobile robot Calculate the angle between the straight line perpendicular to the ground starting from the target point and the straight line extending from the target to the center of the first joint of the robot arm : ； S14: Get the distance from the center of mass of the unmanned platform to the first joint of the robotic arm on the x-axis. , calculate the reachability area diameter Size: The size of the reachable area is the reachable area diameter A sphere with a diameter of .

3. A coordinated control method for a wheeled mobile mechanical arm according to claim 1, characterized in that: Step S2 includes: S21: Calculate the Jacobian matrix of the wheeled mobile manipulator according to the wheeled mobile manipulator model : in, is the end effector position Joint angle The partial guide, is the degree of freedom of the operating space, is the degree of freedom of the joint space; S22: According to the Jacobian matrix of the wheeled mobile manipulator , calculate the operability index in, represents the determinant evaluation, for The transposed matrix of .

4. A coordinated control method for a wheeled mobile mechanical arm according to claim 3, characterized in that: Step S3 includes: S31: According to the Jacobian matrix of the wheeled mobile manipulator , The upper part of represents the linear velocity of the end effector of the wheeled mobile robot. The lower part of represents the angular velocity of the end effector of the wheeled mobile robot. Translated as: in, is the linear velocity part, is the angular velocity part; S32: Solve the linear velocity of the end of the wheeled mobile robot : in, yes The transposed matrix of S33: Determine the angular velocity of the end of the wheeled mobile robot , in, yes The transposed matrix of and Draw the velocity ellipsoid model.

5. A coordinated control method for a wheeled mobile mechanical arm according to claim 3, characterized in that: Step S4 includes: S41: According to the structure of the wheeled mobile robotic arm, the intelligent agents are divided into two agents: an unmanned vehicle agent and a robotic arm agent. According to the reachability region and the velocity ellipsoid model, the joint state space of the wheeled mobile manipulator is constructed as ,in, is the displacement of the unmanned vehicle agent relative to the starting position and the yaw in the global frame, are the linear and angular velocities of the joints of the robot agent, is the position coordinate of the end effector of the robot arm, is the target position coordinate; , is the set of real numbers; S42: Constructing a joint action space of the unmanned vehicle agent and the robotic arm agent according to the reachability region and the velocity ellipsoid model ,in is the action space of the unmanned vehicle agent, including the linear velocity and angular velocity of the unmanned vehicle agent. is the action space of the robot agent, including the linear velocity and angular velocity of the robot agent, and the data of the joint action space is normalized to ; S43: The reward function includes a global reward function, a collision penalty function, a reachability area local reward function and a maneuverability local reward function. Constructing the reward function includes the following steps: S431: Constructing a global reward function : in, is the first control coefficient of the reward range, is the second control coefficient of the reward amplitude, is the distance between the end effector of the wheeled mobile robot and the target point, represents the distance between two points, Indicates that the function takes the minimum value; S432: Construct collision penalty function : ； S433: Constructing a local reward function for the reachability region of a wheeled mobile manipulator : in, is the distance between the unmanned vehicle platform and the target point; S434: Constructing a local reward function for maneuverability of a wheeled mobile manipulator : in, is the operating threshold.

6. A coordinated control method for a wheeled mobile mechanical arm according to claim 5, characterized in that: Step S5 includes: S51: Initialize the global evaluation network , initialize the global target evaluation network , initialize the action network of the unmanned vehicle agent , the action network of the robot agent , Evaluation network of unmanned vehicle agent and the evaluation network of the robot agent , generate the target position within the set range and set the initial state of the wheeled mobile manipulator; S52: Get the initial state of the simulation environment: , , ,in, Indicates that at time step When is the state of the joint state space, Indicates that at time step The action space of the unmanned vehicle agent is Indicates that at time step The action space of the robot agent is represents the policy function of the unmanned vehicle agent, Indicates that at time step The observed input data of the unmanned vehicle agent is Indicates that at time step is the policy function of the robot agent, Indicates that at time step The observation input data of the robot agent; S53: Perform joint actions ,in, , at time step Get the collision penalty function when , at time step The local reward function of the reachable area when and at time step The local reward function for maneuverability ; S54: Build storage sample , and store the stored samples into the experience pool, where Indicates that at time step The state of the joint state space when S55: training a global evaluation network, randomly sampling from the experience pool Samples make up a batch of samples : in, , is the sample ordinal number, , the symbol in the formula Indicates an assignment operation, that is, assigning the result of the expression on the right side of the equal sign to the variable on the left side. Represented in the joint state space Take the following The joint action of samples The collision reward function obtained after represents the action taken by the unmanned vehicle agent at the next time step, represents the action taken by the robot agent in the next time step, Indicates that the unmanned vehicle agent is in The observation data for the next time step in samples, Indicates that the robot agent is in the The observation data for the next time step in samples, is the policy function of the unmanned vehicle agent under the new policy at the next time step, is the policy function of the robot agent under the new policy at the next time step; For the The joint state space at the next time step when the sample is S56: Calculating target value and minimizing loss : in, It is The target value of samples, Indicated in In the samples, the agent The global reward obtained at the time step, the agents include the unmanned vehicle agent and the robotic arm agent, , is the discount factor, Indicates that the new strategy Next, use the action network parameters The action value function, Indicated in In the samples, the agent The observed data at the next time step, Indicates that the new strategy Next, the intelligent agent Observation data in the new state actions taken in It is an intelligent agent The global evaluation network uses the global evaluation network parameters to estimate the value of a given state and action; use and Update the global evaluation network and global target evaluation network : in, Indicates a replacement operation. is the evaluation network learning rate, is the global target network parameter, is the inertia update rate; is the loss function About evaluating network parameters The gradient of S57: Initialize the agent local evaluation network and agent local goal evaluation network , randomly sampled from the experience pool Samples form a small batch of samples ,in, is the second sample ordinal number, ;in, , Represented in the joint state space Take the following The joint action of samples The collision reward function obtained after Indicates that the unmanned vehicle agent is in The observation data for the next time step in samples, Indicates that the robot agent is in the The observation data for the next time step in samples; For the The joint state space at the next time step when the sample is S58: Calculate local target value and local minimization loss : It is The local target value of samples, Indicated in In the samples, the agent The global reward obtained at the time step, It is an intelligent agent The global evaluation network uses the global evaluation network parameters to estimate the value of a given state and action; use and Update the agent's local evaluation network and local target evaluation network : in, is the local target network parameter; is the loss function On the evaluation of local evaluation network parameters The gradient of S59: Computational Agents i Strategy parameters : in, is the update rule for the policy parameters, It’s a strategy About Strategy Parameters The gradient of Among the samples, Observation data of an agent Agents under Joint action , It is a global evaluation network On the joint actions of agents The gradient of It is The local evaluation function of an agent On the joint actions of agents The gradient of It is Among the samples, Observation data of each agent; S510: Update and : in, is the agent strategy parameter for the next time step.

7. A coordinated control method for a wheeled mobile mechanical arm according to claim 1, characterized in that: Step S6 includes: S61: Set up task scenarios, including simple environments without obstacles and complex environments with multiple obstacles; S62: Setting training hyper parameters, the training hyper parameters include: discount factor, experience pool size, inertia update rate, simulation time step, maximum time step of a single round, evaluation network learning rate, action network learning rate; S63: Design task evaluation indicators include average operability, average operability and success rate.

8. A coordinated control method for a wheeled mobile mechanical arm according to claim 1, characterized in that: Step S7 includes: S71: The training is set as follows: the maximum length of the round is set according to the task scenario, training hyperparameters and task evaluation indicators, the initial position of the wheeled mobile manipulator is kept consistent, the design and call of the coordinated control algorithm of the wheeled mobile manipulator is implemented based on the RLGames library, two independent multi-layer perceptrons are used for DE-MADDPG, the input of the coordinated control algorithm of the wheeled mobile manipulator is the same, and in the simulation training environment, the running frequency of the wheeled mobile manipulator simulation is set, and the interaction frequency of the coordinated control algorithm of the wheeled mobile manipulator is set; S72: Train and evaluate the coordinated control algorithm for a wheeled mobile manipulator in two different task scenarios, construct a theoretically infinite test set, in which the target position is randomly generated within a preset range, and record the success rate of the task at the end of each training cycle, wherein the success rate is determined by calculating the percentage of successful cases in a validation set that includes a set number of independent trials of the validation set, and the target point position of each trial is randomly generated within a specified range, and the success rate is recorded after each training cycle has passed a set number of training cycles, each environment configuration is independently trained a threshold number of times, and each training includes a set number of training cycles training cycles, and generates a test result including a random target position; S73: According to the success rate of the test results and the observation results of the operability of the robot arm, the effectiveness of the coordinated control algorithm process of the wheeled mobile robot arm is judged. If the effectiveness is lower than the set threshold, return to step S5 for retraining.

9. A coordinated control method system for a wheeled mobile robot arm, used to execute a coordinated control method for a wheeled mobile robot arm as claimed in any one of claims 1 to 8, characterized in that: include: Data acquisition module: establish a wheeled mobile manipulator model, solve the target reachability area according to the wheeled mobile manipulator model, calculate the Jacobian matrix of the wheeled mobile manipulator and the operability index of the wheeled mobile manipulator according to the wheeled mobile manipulator model, calculate the maximum linear velocity of the wheeled mobile manipulator and the maximum linear velocity of the wheeled mobile manipulator according to the Jacobian matrix of the wheeled mobile manipulator, and obtain the velocity ellipsoid model of the wheeled mobile manipulator; Model building module: constructing a joint state space, a joint action space and a reward function according to the target reachability area, the operability index and the velocity ellipsoid model, and constructing a coordinated control algorithm flow of a wheeled mobile manipulator based on the DE-MADDPG algorithm according to the joint state space, the joint action space and the reward function; Model training module: Build a physical simulation training environment for a wheeled mobile robotic arm based on IsaacSim, set training parameters, design task evaluation indicators, perform training settings based on the task scenarios of the wheeled mobile robotic arm coordinated control algorithm process, the training hyperparameters and the task evaluation indicators, record the success rate of the task at the end of each training cycle, and judge the effectiveness of the wheeled mobile robotic arm coordinated control algorithm process.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the coordinated control method of a wheeled mobile robotic arm as described in any one of claims 1 to 8 are implemented.

Images