Mobile operation robot hybrid control method and device based on guided reinforcement learning

Abstract

The application provides a mobile operating robot hybrid control method and device based on guided reinforcement learning, and belongs to the technical field of mobile operating robot control. The method comprises the following steps: acquiring robot prior skills based on teaching data; performing phase processing and conditional encapsulation on the prior skills to obtain reference priors in a task coordinate system, and constructing the deviation of action and stiffness based on the current phase and the reference priors; splicing the reference priors, the deviation and current state data into observation data, receiving the observation data by a pre-constructed PPO reinforcement learning model, and outputting a residual correction amount; the PPO reinforcement learning model is configured to learn the action compensation difference value relative to the reference priors in different phases; constructing a reference action based on the current state phase and the reference priors, and determining a control action through the reference action and the action compensation difference value; generating a bottom layer execution instruction according to the control action through a pre-configured vehicle arm hybrid motion controller, and controlling and executing a task based on the bottom layer execution instruction.

Description

Technical Field

[0001] This application relates to the field of mobile robot control technology, and in particular to a hybrid control method and apparatus for mobile robots based on guided reinforcement learning. Background Technology

[0002] As intelligent manufacturing evolves towards higher precision and flexibility, the demand for mobile manipulation robots in industrial contact-rich assembly scenarios, such as precision shaft and hole assembly and bolt tightening of dissimilar components, is becoming increasingly urgent. In actual production, such tasks require robots to simultaneously perform large-scale mobile navigation and localized fine manipulation, placing extremely high demands on the system's motion accuracy, contact compliance, and dynamic adaptability. However, under the current technological framework, mobile manipulation robots still face multiple bottlenecks in complex contact-rich tasks: for developers, traditional teach-based or rule-based motion planning requires manual adjustment of trajectory parameters and stiffness gains for each task, resulting in long development cycles and poor generalization; for the robot itself, the heterogeneous arm system exhibits strong spatiotemporal coupling effects during dynamic contact, making it difficult for a single control strategy to simultaneously balance global motion efficiency and local contact stability, and model errors between simulation training and the real physical environment can easily lead to strategy deployment failure.

[0003] To address the aforementioned issues, existing technologies have explored control methods for mobile robots in multiple directions, which can be mainly categorized into four types: The first type is based on traditional motion planning methods, which complete assembly tasks through predefined trajectories and fixed stiffness parameters, exhibiting high determinism in execution; the second type is end-to-end reinforcement learning methods, which directly learn complete motion strategies from the state space, possessing a certain degree of generalization; the third type is pure impedance control methods, which achieve contact compliance by adjusting the end-effector stiffness of the robotic arm; and the fourth type is vehicle-arm decoupling control methods, which separate the planning of chassis navigation and robotic arm operation, simplifying the control logic.

[0004] However, the four methods mentioned above still share two common problems: first, traditional end-to-end reinforcement learning methods lack prior knowledge that simultaneously includes kinematic trajectories and stiffness, resulting in low sample utilization efficiency; second, insufficient coordination between the mobile chassis and robotic arms severely limits the stability of policy execution and the safe deployment capability of real-world systems. These problems collectively restrict the large-scale application of mobile manipulation robots in industrial precision assembly scenarios. Summary of the Invention

[0005] In view of this, this application provides a motion control method and device for mobile operation robots based on guided reinforcement learning, in order to solve the problem that the arm cannot coordinate and efficiently handle the assembly in rich contact scenarios.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a hybrid control method for a mobile robot based on guided reinforcement learning, the method comprising:

[0008] Prior skills of the robot are acquired based on teaching data, including time-varying stiffness trajectory and pose trajectories of the chassis and end effector in the robot.

[0009] The prior skills are phase-processed and conditionally encapsulated to obtain phase-related reference priors in the task coordinate system. Based on the robot's current phase and the reference priors, deviations in action and stiffness are constructed, including:

[0010] The prior skills are transformed from the world coordinate system to the task coordinate system to obtain the prior reference. The prior reference includes the current phase, the phase expression of the pose trajectory of the chassis and the end effector of the robotic arm, and the phase expression of the time-varying stiffness trajectory.

[0011] Based on the prior reference, the phase expression of the chassis and robotic arm end effector pose trajectory corresponding to the look-ahead phase and the phase expression of the time-varying stiffness trajectory are determined.

[0012] The phase representation of the pose trajectory corresponding to the look-ahead phase, the phase representation of the time-varying stiffness trajectory, and the prior reference are encapsulated into the reference prior; the reference prior is configured as follows:

[0013] ;

[0014] in, As a reference prior at time t, For the current mission phase, For the look-ahead phase increment, This represents the phase of the chassis trajectory at time t. This represents the phase of the robot arm's end effector trajectory at time t. This is the phase representation of the time-varying stiffness trajectory at time t. This represents the phase expression of the chassis trajectory corresponding to the forward-looking phase. This represents the phase expression of the robotic arm's end-effector trajectory corresponding to the look-ahead phase. This is the phase representation of the time-varying stiffness trajectory corresponding to the look-ahead phase;

[0015] Based on the phase representation of the pose trajectories of the chassis and the end effector of the robotic arm in the reference prior, the phase representation of the time-varying stiffness trajectory, and the current phase, the deviation between the motion and stiffness is constructed.

[0016] The reference prior, the bias, and the robot's current state data are concatenated into observation data. The observation data is received by a pre-built PPO reinforcement learning model, which outputs residual correction values. The PPO reinforcement learning model is configured to learn the action compensation difference relative to the reference prior under different phases.

[0017] A reference action is constructed based on the current phase and the reference prior, and a control action is determined by the difference between the reference action and the action compensation value.

[0018] The robot is controlled to perform tasks based on the control actions generated by the pre-configured hybrid motion controller.

[0019] In a preferred embodiment of the present invention, the acquisition of the robot's prior skills based on teaching data includes:

[0020] The original timing information is obtained based on the teaching information, and the original timing information includes: original chassis pose timing, original end pose timing, and original time-varying stiffness timing.

[0021] The prior skills are obtained by performing time-series alignment processing on the original time-series information using the Impedance-PRIMP algorithm.

[0022] In another preferred embodiment of the invention, the deviation between the action and the stiffness is configured as follows:

[0023] ;

[0024] in, The deviation at time t, For the vectorization operator of the logarithmic map, It is the Frobenius norm. For the current mission phase, As the chassis, For robotic arm end effectors, Let be the inverse matrix of the phase representation of the chassis trajectory at time t. The actual position and pose of the chassis at time t. Let be the inverse matrix of the phase expression of the robot arm's end-effector trajectory at time t. Let be the actual pose of the robotic arm's end effector at time t. This is the phase representation of the time-varying stiffness trajectory at time t. Let be the time-varying stiffness parameter at time t.

[0025] In another preferred embodiment of the invention, the current state data of the robot is configured as follows:

[0026] ;

[0027] in, and For the observation of the speed of the moving chassis, , , and These represent the position, attitude, linear velocity, and angular velocity observations of the end effector relative to the assembly target, respectively. This refers to the action performed at the previous moment.

[0028] In another preferred embodiment of the present invention, the reward function of the residual PPO reinforcement learning model is configured as follows:

[0029] ;

[0030] in, This represents the original task reward, used to characterize the direct feedback on the completion degree of the core task objective after the strategy executes the action at time t. This indicates a soft guide term related to the reference prior. Its weighting coefficient.

[0031] The soft guidance term is used to encourage the strategy to prioritize searching for feasible solutions within a priori reasonable region, specifically:

[0032] ;

[0033] in, The penalty weight for chassis position deviation. The penalty weight for pose deviation at the robotic arm's end effector. This represents the penalty weight for stiffness deviation.

[0034] In another preferred embodiment of the present invention, a pre-configured arm hybrid motion controller generates low-level execution instructions based on the control actions, including:

[0035] The control parameters of the chassis and the robotic arm are obtained according to the control actions;

[0036] The control parameters of the chassis are subjected to motion scale scaling and impedance shaping to obtain the actual execution speed of the chassis.

[0037] Based on a pre-built joint speed coordination and allocation optimization model, the desired joint speed of the robotic arm is obtained according to the control parameters of the robotic arm.

[0038] Based on the desired joint speed of the robotic arm, the impedance and stiffness of the robotic arm are adjusted to obtain the joint speed after impedance correction.

[0039] The underlying execution command is generated based on the impedance-corrected joint speed and the actual execution speed of the chassis.

[0040] In another preferred embodiment of the present invention, the pre-built joint velocity coordination allocation optimization model is configured as follows:

[0041] ;

[0042] in, Let be the desired joint velocity of the robotic arm. For the joint angle of the robotic arm, , For joint velocity, , Jacobi is the contribution of the robotic arm to the end effector speed. Jacobi for the chassis's contribution to the final velocity. The actual operating speed of the chassis. The error weighting matrix for the final task. Let be the joint velocity regularization coefficient, and , Let be the velocity spinor of the end effector, where , The desired end-point space velocity.

[0043] In another preferred embodiment of the present invention, the step of adjusting the impedance and stiffness of the robotic arm based on the desired joint speed of the robotic arm to obtain the impedance-corrected joint speed includes:

[0044] Based on the expected joint velocity and the actual end-effector velocity of the robotic arm, as well as the reference equilibrium pose and the actual pose of the robotic arm, a pose error is constructed.

[0045] The desired stiffness of the execution layer is determined based on the stiffness prior trajectory, and the mapping relationship between the desired stiffness, the pose error and the desired interaction force of the robotic arm is constructed.

[0046] Solve the mapping relationship to obtain the joint velocity after impedance correction.

[0047] A second aspect of this application provides a hybrid control device for a mobile robot based on guided reinforcement learning, the device comprising:

[0048] The prior skills acquisition module is used to acquire the robot's prior skills based on teaching data. The prior skills include time-varying stiffness trajectories and the pose trajectories of the chassis and end effector in the robot.

[0049] A deviation construction module is used to perform phase-based processing and conditional encapsulation on the prior skills to obtain phase-related reference priors in the task coordinate system, and to construct deviations in action and stiffness based on the robot's current phase and the reference priors, including:

[0050] The prior skills are transformed from the world coordinate system to the task coordinate system to obtain the prior reference. The prior reference includes the current phase, the phase expression of the pose trajectory of the chassis and the end effector of the robotic arm, and the phase expression of the time-varying stiffness trajectory.

[0051] Based on the prior reference, the phase expression of the chassis and robotic arm end effector pose trajectory corresponding to the look-ahead phase and the phase expression of the time-varying stiffness trajectory are determined.

[0052] The phase representation of the pose trajectory corresponding to the look-ahead phase, the phase representation of the time-varying stiffness trajectory, and the prior reference are encapsulated into the reference prior; the reference prior is configured as follows:

[0053] ;

[0054] in, As a reference prior at time t, For the current mission phase, For the look-ahead phase increment, This represents the phase of the chassis trajectory at time t. This represents the phase of the robot arm's end effector trajectory at time t. This is the phase representation of the time-varying stiffness trajectory at time t. This represents the phase expression of the chassis trajectory corresponding to the forward-looking phase. This represents the phase expression of the robotic arm's end-effector trajectory corresponding to the look-ahead phase. This is the phase representation of the time-varying stiffness trajectory corresponding to the look-ahead phase;

[0055] Based on the phase representation of the pose trajectories of the chassis and the end effector of the robotic arm in the reference prior, the phase representation of the time-varying stiffness trajectory, and the current phase, the deviation between the motion and stiffness is constructed.

[0056] The residual correction output module is used to concatenate the reference prior, the deviation, and the robot's current state data into observation data, receive the observation data through a pre-built PPO reinforcement learning model, and output the residual correction amount; the PPO reinforcement learning model is configured to learn the action compensation difference relative to the reference prior under different phases;

[0057] A control action construction module is used to construct a reference action based on the current phase and the reference prior, and to determine the control action by the difference between the reference action and the action compensation value.

[0058] The task execution module is used to generate low-level execution instructions based on the control actions through a pre-configured hybrid motion controller for the robot arm, and to control the robot to perform tasks based on the low-level execution instructions.

[0059] The motion control method for mobile robots based on guided reinforcement learning provided in this application has the following beneficial effects:

[0060] This paper proposes a collaborative optimization method for mobile manipulators based on guided reinforcement learning and hybrid motion control of the robot arm. Building upon existing solutions combining reinforcement learning and impedance control, this application employs input conditions including task observations, prior reference trajectories, and real-time deviations. Task observations encompass the robot's motion state and environmental feedback; the prior reference trajectories are derived from the learned motion and stiffness trajectories; and the real-time deviations represent the error between the current state and the reference trajectory. These are all readily available initial variables, eliminating the need for complex environmental modeling or manual annotation, thus efficiently supporting control decisions. Based on the accurate acquisition of these initial variables, this application treats the mobile chassis, robot arm, and reinforcement learning strategy as a unified whole for collaborative optimization. Residual motion, contact force, and stiffness adjustment are used as intermediate variables. Through numerical analysis of the robot arm coupling dynamics and contact interaction processes, the optimal control strategy is predicted for different task stages. Based on the prediction results, the corresponding robot arm coordinated control parameters and stiffness adjustment strategies are output, directly generating the motion control commands and impedance parameters for the mobile manipulator. This collaborative optimization method can significantly reduce the exploration complexity of reinforcement learning. At the same time, the introduction of reference trajectory priors eliminates the need for precise simulation environment modeling in policy training, effectively narrowing the Sim-to-Real generalization gap and greatly improving the reliability of applications in industrial scenarios. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating a motion control method for a mobile operating robot based on guided reinforcement learning, according to an exemplary embodiment.

[0062] Figure 2 This is a schematic diagram illustrating an IsaacSim simulation training environment according to an exemplary embodiment;

[0063] Figures 3 to 8 This is a schematic diagram illustrating assembly task training according to an exemplary embodiment;

[0064] Figure 9 This is a schematic diagram illustrating the overall training reward curve and error band according to an exemplary embodiment;

[0065] Figure 10 This is a schematic diagram illustrating the change curves of the reward function at different stages according to an exemplary embodiment;

[0066] Figure 11 This is a schematic diagram illustrating the prior motion trajectory of a mobile chassis according to an exemplary embodiment;

[0067] Figure 12 This is a schematic diagram illustrating the prior motion trajectory and stiffness trajectory of a robotic arm according to an exemplary embodiment;

[0068] Figures 13 to 15 This is a schematic diagram illustrating the prior motion trajectory of a mobile chassis according to an exemplary embodiment;

[0069] Figures 16 to 17 This is a schematic diagram showing a comparison of the velocity time-varying curves of the method described herein and the method without virtual impedance, according to an exemplary embodiment.

[0070] Figures 18 to 19 This is a schematic diagram showing a comparison of the time-varying acceleration curves of the method described in this paper and the method without virtual impedance, according to an exemplary embodiment.

[0071] Figure 20 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0073] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0074] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0075] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0076] Example 1:

[0077] Figure 1 This is a flowchart of an embodiment of the guided reinforcement learning-based motion control method for mobile robots provided in this application. Please refer to... Figure 1 The method provided in this embodiment includes steps S1 to S5:

[0078] Step S1: Obtain the robot's prior skills based on the teaching data. The prior skills include the time-varying stiffness trajectory and the pose trajectories of the chassis and end effector in the robot.

[0079] In step S1, the teaching data refers to the robot manually executing the target task once, and the robot recording various sensor data during the execution process. Prior skills refer to the motion template of the robot-arm coordination.

[0080] Optionally, step S1 specifically includes:

[0081] Step S11: Obtain the original timing information based on the teaching information. The original timing information includes: original chassis pose timing, original end pose timing, and original time-varying stiffness timing.

[0082] Step S12: The prior skill is obtained by performing time alignment processing on the original time series information using the Impedance-PRIMP algorithm.

[0083] The specific timing alignment is represented as follows:

[0084] ;

[0085] in, This is the chassis reference pose set. This is a set of reference poses for the robotic arm's end effector. For time-varying stiffness sets, For time step indexing on the discretized trajectory. , This represents the total number of steps in the teaching trajectory.

[0086] Step S1 constructs prior skills based on teaching data, including the pose trajectories of the chassis and end effector and the time-varying stiffness trajectories. The coordinated motion of the arm is encoded into motion primitives with unified phase parameterization. This provides a reference action benchmark, conditional bias observation, and guided reward constraints for the residual reinforcement learning strategy, reducing the strategy search space and improving the training stability and execution success rate under rich contact tasks.

[0087] Step S2: Perform phase-based processing and conditional encapsulation on the prior skills to obtain phase-related reference priors in the task coordinate system, and construct the deviations of action and stiffness based on the robot's current phase and reference priors.

[0088] In step S2, phase conversion processing refers to the process of converting phase... Prior skills. Conditional encapsulation refers to the encapsulation of phase. Prior skills and phase anticipation The prior skills are combined under the following conditions. Reference priors refer to a conditionally encapsulated set of prior skills used to characterize prior skills at a given phase. Under these conditions, the robot arm should be in the ideal state according to human teaching experience. Deviation refers to the difference between the robot's current actual motion state and the ideal state corresponding to the same phase in prior skills.

[0089] Optionally, step S2 specifically includes:

[0090] Step S21: Transform the prior skills from the world coordinate system to the task coordinate system to obtain the prior reference. The prior reference includes the phase, the phase expression of the pose trajectory of the chassis and the end effector of the robotic arm, and the phase expression of the time-varying stiffness trajectory.

[0091] Among them, prior reference refers to the expression of the reference prior in different contexts, and the content represented is consistent with the reference prior. Phase representation refers to parameterizing the pose trajectory and impedance parameters of the mobile manipulator chassis and robotic arm from the physical time domain into a function representation with phase, so as to uniformly describe the spatial form and compliance characteristics of the robot-arm cooperative skill as the task progresses.

[0092] Step S22: Based on prior references, determine the phase expression of the pose trajectory of the chassis and the end effector of the robotic arm corresponding to the look-ahead phase, and the phase expression of the time-varying stiffness trajectory.

[0093] Step S23: Encapsulate the phase representation of the pose trajectory corresponding to the look-ahead phase, the phase representation of the time-varying stiffness trajectory, and the prior reference into a reference prior.

[0094] The reference prior is represented as:

[0095] ;

[0096] in, As a reference prior at time t, For the current mission phase, For the look-ahead phase increment, This represents the phase of the chassis trajectory at time t. This represents the phase of the robot arm's end effector trajectory at time t. This is the phase representation of the time-varying stiffness trajectory at time t. This represents the phase expression of the chassis trajectory corresponding to the forward-looking phase. This represents the phase expression of the robotic arm's end-effector trajectory corresponding to the look-ahead phase. This is the phase representation of the time-varying stiffness trajectory corresponding to the look-ahead phase.

[0097] Step S24: Construct the deviation based on the phase expression of the pose trajectory of the chassis and the end effector of the robotic arm in the reference prior and the phase expression of the time-varying stiffness trajectory.

[0098] The deviation is expressed as:

[0099] ;

[0100] in, The deviation at time t, For the vectorization operator of the logarithmic map, It is the Frobenius norm. For the current mission phase, As the chassis, For robotic arm end effectors, Let be the inverse matrix of the phase representation of the chassis trajectory at time t. The actual position and pose of the chassis at time t. Let be the inverse matrix of the phase expression of the robot arm's end-effector trajectory at time t. Let be the actual pose of the robotic arm's end effector at time t. This is the phase representation of the time-varying stiffness trajectory at time t. Let be the time-varying stiffness parameter at time t.

[0101] Step S2 obtains a reference prior by phasing and conditionally encapsulating the prior skills, and encodes the cooperative motion of the mobile robot arm into a continuous reference function of the normalized task progress to achieve teaching trajectory alignment; by constructing the deviation of action and stiffness based on the robot's current phase and the reference prior, input conditions are provided for the residual correction amount of the subsequent strategy output.

[0102] Step S3: The reference prior, bias and current state data of the robot are concatenated into observation data. The observation data is received through the pre-built PPO reinforcement learning model, and the residual correction is output. The PPO reinforcement learning model is configured to learn the action compensation difference relative to the reference prior under different phases.

[0103] In step S3, the robot's current state data is obtained through robot observation, which includes the robot's motion state during instantaneous actions, including the acquisition of noisy chassis speed, end-effector pose and speed, and the previous action.

[0104] The PPO reinforcement learning model refers to a shared Actor-Critic deep network driven by Impedance-PRIMP prior-conditional observations, with residual actions as outputs, and trained using the Clipped-PPO objective function. It is used to learn fine-tuned control strategies to adapt to environmental disturbances near the teaching reference channel.

[0105] Optionally, step S3 specifically includes:

[0106] Step S31: Obtain the robot's current state data, and combine the reference prior, bias and the robot's current state data into observation data.

[0107] The robot's current state data is configured as follows:

[0108] ;

[0109] in, The current state data of the robot at time t, and For the purpose of observing the speed of the moving chassis, , , and These represent the position, attitude, linear velocity, and angular velocity observations of the end effector relative to the assembly target, respectively. This refers to the action performed at the previous moment.

[0110] The robot's current state data is obtained through robot observation, and combined with the reference prior, the bias, and the robot's current state data to form observation data. The observation data is represented as follows:

[0111] ;

[0112] in, This is the robot's current state data. For reference prior, This is a deviation.

[0113] Step S32: Configure the reward function and optimization objective of the PPO reinforcement learning model.

[0114] The reward function is used to narrow the effective exploration space of the policy and improve the policy calculation speed.

[0115] The optimization objective is to update the Actor-Critic network parameters using gradients based on the advantage signal calculated from the reward function, aiming to make the policy tend towards high-yield, high-reward actions. Simultaneously, a pruning mechanism limits the magnitude of single-step policy changes to ensure stable training convergence. The optimization objective remains unchanged in this scheme; it is an inherent property of the PPO reinforcement learning model.

[0116] The reward function of the pre-built PPO reinforcement learning model is configured as follows:

[0117] ;

[0118] in, This represents the original task reward, used to characterize the direct feedback on the completion degree of the core task objective after the strategy executes the action at time t. This indicates a soft guide term related to the reference prior. This is the weighting coefficient for the soft guide term, used to control its size.

[0119] The original task reward can be represented as:

[0120] ;

[0121] in, This refers to the task reward at time t; multi-scale keypoint reward. , as well as Used to measure the geometric alignment between the insert and the target hole; motion change penalty Used to suppress and control chattering; stage reward With success reward Depict the initial insertion and the final insertion completion states separately; relative action penalty Used to limit overly aggressive end-effector movements; contact force penalty Used to improve contact safety; successfully predicts error penalties. This is used to improve the strategy's ability to determine the task completion status.

[0122] The original task reward representation is transformed into a differentiable optimization objective by integrating multi-scale key point rewards, action change penalties, stage rewards, success rewards, relative action penalties, contact force penalties, and success prediction error penalties. This guides the reinforcement learning strategy to converge efficiently to the task endpoint and standardizes the compliant and safe behavior of the robot in a contact-rich environment.

[0123] The soft guidance term is used to encourage the strategy to prioritize searching for feasible solutions within a priori reasonable region, specifically:

[0124] ;

[0125] in, This refers to the soft guiding term at time t. The penalty weight for chassis position deviation. The penalty weight for pose deviation at the robotic arm's end effector. This represents the penalty weight for stiffness deviation.

[0126] The reward function combines the original task reward with the soft guidance term to constrain the robot's policy update to the feasible manifold neighborhood defined by the teaching prior, thereby narrowing the effective exploration space of the policy and improving the policy calculation speed.

[0127] The optimization objective of the PPO reinforcement learning model is configured as follows:

[0128] ;

[0129] in, This refers to the optimization objective of the PPO reinforcement learning model. For the estimation of the advantage function, This is the cutting factor. The ratio of the probability of the new and old strategies under conditional observations. Let be the expected value of the optimization at time t.

[0130] Step S33: Receive the observation data through the pre-built PPO reinforcement learning model and output the residual correction amount.

[0131] The residual correction is expressed as:

[0132] ;

[0133] in, This represents the conditionalized actor-side input. This is the residual correction amount. For successful quantity prediction, Indicates the action dimension. For policy networks.

[0134] The learning process of the PPO reinforcement learning model is represented as follows:

[0135] 1. Define the input as the environmental state and prior trajectories of the chassis, end effector, and time-varying stiffness reference trajectories; the output is the residual strategy network;

[0136] 2. Initialize network parameters and value network parameters;

[0137] 3. Reset the environment and initialize phase, historical actions, and cache;

[0138] 4. Input environmental conditions and prior knowledge of chassis, end effector, and time-varying stiffness reference trajectories;

[0139] 5. Construct conditional observations based on environmental conditions and reference prior knowledge;

[0140] 6. Construct reference actions based on reference priors, and generate interactive actions using residual strategies;

[0141] 7. Apply interactive actions to the environment to obtain the original task reward and the completion marker;

[0142] 8. Calculate the soft bootstrap term, and combine the original task reward with the soft bootstrap term to obtain the reward function;

[0143] 9. Transfer the current transfer sample to the cache, and update the phase and historical actions;

[0144] 10. Repeat steps 6 through 9 until a termination flag is encountered, then output the final residual policy network.

[0145] Step S3 concatenates the reference prior, bias, and current robot state data into a conditionalized observation input PPO model, enabling the residual strategy to explicitly perceive the ideal cooperative state and deviation degree under the current task phase. This reduces the input ambiguity and search difficulty of the reinforcement learning strategy. At the same time, the output of the PPO model is limited to the residual correction amount relative to the reference prior, so that the strategy only needs to be fine-tuned in the low-dimensional residual space adjacent to the teaching channel to adapt to environmental disturbances. This preserves the robot-arm cooperation and compliance characteristics of the prior skills, and endows the mobile operation robot with online adaptability and training stability in contact with a variety of tasks.

[0146] Step S4: Construct a reference action based on the current phase and the reference prior, and determine the control action by the difference between the reference action and the action compensation.

[0147] In step S4, the reference action is represented as the ideal open-loop instruction of the arm-arm coordinated motion in the teaching experience, which is used by the residual reinforcement learning strategy to output the residual correction amount based on the reference action to adapt to environmental disturbances.

[0148] The residual correction is the motion compensation difference. The residual correction represents the fine-tuning compensation amount relative to the Impedance-PRIMP prior reference motion under a given task phase. It is used to enable the mobile manipulator to adapt to unmodeled environmental disturbances such as ground slippage and workpiece tolerances while retaining the cooperative and compliant characteristics of the teach pendant arm.

[0149] Based on the current phase and the reference prior, a corresponding reference action is constructed. ,

[0150] The control action is determined based on the reference action and the residual correction amount:

[0151] ;

[0152] in, This refers to the control action at time t, where the reference action and residual correction are combined through a dimension-by-dimensional addition method. This is the residual correction amount. For successful prediction.

[0153] Step S4 obtains the chassis and end effector reference actions obtained from the Impedance-PRIMP prior skill mapping by looking up the task phase table. The residual correction amount output by the residual reinforcement learning strategy is combined with the reference actions to determine the final control action. This allows the system to fine-tune unmodeled disturbances such as ground slip and workpiece tolerances online while preserving the prior skill cooperative relationship and stage adaptive compliance characteristics. This reduces the policy search space and improves the training stability and execution safety under rich contact tasks.

[0154] Step S5: Using a pre-configured hybrid motion controller for the robot arm, generate underlying execution instructions based on the control actions, and control the robot to perform tasks based on the underlying execution instructions.

[0155] In step S5, the hybrid motion controller of the vehicle arm refers to the decomposition of the synthetic control action output by the residual reinforcement learning strategy into chassis speed commands and robotic arm end-effector pose correction commands, which are respectively sent to the speed tracker of the moving chassis and the Cartesian space impedance controller of the robotic arm.

[0156] Optionally, step S5 specifically includes:

[0157] Step S51: Obtain the control parameters for the chassis and robotic arm respectively based on the control actions:

[0158] ;

[0159] in, This refers to the control parameters of the chassis and robotic arm at time t. This indicates the task space action of the end effector. This indicates the speed and movement of the chassis.

[0160] Step S52: Perform motion scale scaling and impedance shaping on the chassis control parameters to obtain the actual execution speed of the chassis.

[0161] By introducing chassis virtual impedance shaping, the dynamic equation for chassis execution speed is solved:

[0162] ;

[0163] in, The virtual inertia matrix of the chassis. This is the virtual damping matrix for the chassis. This refers to the actual operating speed of the chassis. These are the control parameters for the chassis after scaling the motion dimensions. This is the chassis reference speed.

[0164] Step S53: Based on the pre-built joint speed coordination and allocation optimization model, obtain the desired joint speed of the robotic arm according to the control parameters of the robotic arm:

[0165] The pre-built joint velocity coordination allocation optimization model is configured as follows:

[0166] ;

[0167] in, Let be the desired joint velocity of the robotic arm. For the joint angle of the robotic arm, , For joint velocity, , Jacobi for the chassis's contribution to the final velocity. The actual operating speed of the chassis. The error weighting matrix for the final task. Let be the joint velocity regularization coefficient, and , Let be the velocity spinor of the end effector, where .

[0168] Step S54: Adjust the impedance and stiffness of the robotic arm based on the desired joint speed to obtain the impedance-corrected joint speed.

[0169] The robotic arm's execution layer introduces task space impedance control, denoted as the current position and attitude of the end effector. The terminal velocity is The end-effector reference equilibrium pose and reference velocity are respectively and The end-effector pose error and velocity error are defined as follows:

[0170] ;

[0171] in, For end-effector pose error, For the end velocity error, Indicates the end position. Indicates the desired position at the end. Represents the attitude rotation matrix. Represents the desired rotation matrix. The vectorized logarithmic mapping representing attitude error.

[0172] The desired interaction force at the end is:

[0173] ;

[0174] in, For the desired interaction force at the end, Here is the task space stiffness matrix. The task space damping matrix is... For end-effector pose error, These are end-velocity errors. They are used to introduce necessary compliance while ensuring reference motion tracking, in order to reduce the force impact during contact establishment and insertion propulsion phases.

[0175] The desired stiffness of the execution layer is constructed based on the phase-dependent stiffness prior trajectory obtained from prior skill learning, and is represented as:

[0176] ;

[0177] in, The phase-dependent stiffness reference is provided by prior skills. This is an online correction term generated based on the current contact status and tracking error. This represents a symmetric positive definite projection operator, used to ensure that the stiffness matrix always maintains its symmetric positive definite property. Damping matrix. according to The spectral characteristics were selected to ensure that the impedance behavior has good dissipation and numerical stability.

[0178] When the robotic arm uses a torque interface, the joint control input is:

[0179] ;

[0180] in, For the joint torque command of the robotic arm, For gravity compensation, For the transpose of the Jacobian matrix of the robotic arm, The desired interaction force at the end.

[0181] When the system uses a joint velocity or position interface, the impedance-corrected joint velocity is obtained. Balance pose with end reference As a reference input for the underlying impedance controller.

[0182] Step S55: Generate low-level execution commands based on the impedance-corrected joint speed and the actual execution speed of the chassis.

[0183] The actual execution speed is the chassis execution speed obtained from solving step S2. .

[0184] Combining the impedance-corrected joint speed and the actual operating speed of the chassis, the arm-coordinating actuator is:

[0185] ;

[0186] in, To execute commands at the lower level, The joint velocity is the impedance-corrected value. This refers to the actual operating speed of the chassis.

[0187] The vehicle-arm coordinated actuator refers to the wheel drive unit of the mobile chassis and the servo units of each joint of the robotic arm. It is configured to synchronously receive chassis speed commands and robotic arm joint position / torque commands and time-varying stiffness parameters issued by the vehicle-arm hybrid motion controller, drive the wheel motors and joint motors to perform motion respectively, and upload the feedback data from the chassis odometer and joint encoder for state estimation and phase update, so as to realize the coordinated physical execution of the chassis and robotic arm according to the same task phase.

[0188] Step S5 uses a pre-configured hybrid motion controller to map the high-level control actions output by the residual reinforcement learning strategy into chassis speed commands and robotic arm joint speed commands. By utilizing chassis virtual impedance shaping and robot arm coordination allocation optimization, heterogeneous motion collaboration between the mobile chassis and the robotic arm is achieved. At the same time, by combining task space impedance control and phase-related stiffness adjustment, the robot exhibits stage-adaptive compliant interaction characteristics during contact-rich assembly, suppressing contact impact and chassis disturbance transmission, and improving the success rate of safe tasks and the deployment stability from simulation to reality for the mobile operation robot in complex unstructured environments.

[0189] In summary, this embodiment provides a motion control method for mobile robots based on guided reinforcement learning. By using a PPO reinforcement learning model to learn control actions from reference priors, compared to directly using the PPO model to learn the overall action in the conventional case, this significantly reduces the time and exploration complexity of reinforcement learning. At the same time, the introduction of reference trajectory priors eliminates the need for precise simulation environment modeling for policy training, effectively narrowing the Sim-to-Real generalization gap and greatly improving the reliability of applications in industrial scenarios.

[0190] Based on the above-mentioned guided reinforcement learning-based motion control method for mobile robots, an effectiveness verification experiment was conducted, as detailed below:

[0191] Simulation Experiment

[0192] Experimental basis

[0193] The training hardware platform used in this paper is a high-performance workstation with an Intel Xeon w9-3495X processor, supporting a maximum turbo frequency of approximately 4.8 GHz; and 128 GB of DDR5 memory (2GB RAM). The graphics processing unit (GPU) uses an NVIDIA RTX 6000 Ada Generation processor with 48 GB of memory, providing computational support for policy network inference, parallel sampling, and GPU physics rendering during reinforcement learning training. The operating system is Ubuntu 22.04 LTS, the simulation software is IsaacSim version 5.0.0, and the reinforcement learning framework is IsaacLab version 2.2.1.

[0194] In terms of simulation environment construction, the robot body was first imported based on the Unified Robot Description Format (URDF) for mobile manipulation robots, and the joint drive attributes, colliders, inertial parameters, initial pose, and simulation-related physical attributes were configured. Combined with the requirements of specific task scenarios, a simulation environment for typical contact-rich assembly operations was constructed, providing a consistent model foundation for subsequent reinforcement learning training and simulation-to-real-world verification. The IsaacSim simulation training environment constructed in this paper is as follows: Figure 2 As shown. In terms of policy training, the PPO algorithm was improved. By combining the learned motion and stiffness priors of the mobile manipulator, an observer, reward function, and actor-commentator structure were designed to achieve joint learning of the coordinated movements of the mobile chassis and the robotic arm. Finally, the trained policy can output high-level control commands oriented towards task execution, providing a policy foundation for the stable deployment of the subsequent hybrid motion controller.

[0195] Corresponding to the simulation platform, a physical experimental platform for mobile operation robots was built. The platform consists of a mobile chassis, a Franka robotic arm, an experimental table, and bolt pin hole assembly workpieces, which are used to verify the deployment and control effects of simulation training strategies in real physical environments.

[0196] Experimental Task

[0197] To verify the effectiveness of the proposed prior knowledge-injected guided reinforcement learning method for mobile manipulation robots in contact-rich assembly tasks, this section analyzes it from two aspects: basic performance and ablation experiments. First, the overall learning effect and assembly execution capability of the proposed method are evaluated by considering reward convergence, task success rate, and changes in key state variables during training. Second, by removing stiffness priors and comparing them with the proposed method, the role of stiffness priors in policy convergence speed, contact stability, and task completion quality is analyzed.

[0198] Experimental results

[0199] <Basic Performance>

[0200] First, we verify the training effect of the residual PPO reinforcement learning algorithm based on prior skills. During the learning phase, the hyperparameters of the residual PPO reinforcement learning algorithm are shown in Table 1.

[0201] Table 1. Network hyperparameters of the residual PPO algorithm

[0202]

[0203] This study analyzes the trends in reward function changes, task success rate improvements, and key state variables during the reinforcement learning training process. By combining the end-effector pose error, contact force changes, and the coordinated motion of the chassis and robotic arm during assembly task execution, the behavioral characteristics of the strategy in contact-rich scenarios can be analyzed. Based on the motion output and contact force changes exhibited at different stages—contact establishment, initial engagement, and final assembly completion—the task completion status can be evaluated. The training screenshot of the final assembly task is shown below. Figures 3 to 8 As shown.

[0204] Based on the prior-guided residual PPO reinforcement algorithm, the task execution of the mobile manipulation robot can be further divided into four stages. First, in the first stage, the robot chassis moves and gradually approaches the target position on the table, allowing the robotic arm to enter the accessible workspace and establishing suitable initial pose conditions for subsequent fine operations. Then, in the second stage, the robotic arm's end effector moves to a hovering point above the target to complete the initial alignment of the task area. Next, in the third stage, the robotic arm moves vertically downwards from the hovering point, further approaching the pre-contact point to achieve precise pre-positioning of the shaft hole assembly position. Finally, in the fourth stage, the end effector continues to press down along the assembly direction and performs the insertion operation, thereby completing the shaft hole assembly task. According to the reward function formula, the changes in the overall reward and the rewards at each stage during training are as follows: Figure 9 and Figure 10 As shown.

[0205] from Figure 9 It can be seen that the overall reward value is low in the early stage of training, but then rises rapidly, indicating that the agent quickly mastered the basic strategy of the chassis approaching the target area. In this stage, the reward increases first, indicating that the robot can effectively reduce the distance between the chassis and the target position on the table and gradually establish a reasonable operational configuration. As training progresses, the reward curve fluctuates, indicating that the task focus gradually shifts from chassis maneuvering to robotic arm operation. During this process, the reward gradually increases, corresponding to the robotic arm approaching the hovering point above the target and then probing downwards towards the pre-contact point, indicating that the agent gradually learns the phased control strategy from chassis adjustment to precise end-effector alignment. Since a stable insertion action has not yet been fully mastered, there is still a certain risk of collision and action deviation during training, therefore the overall reward will show local fluctuations.

[0206] from Figure 10 Further analysis reveals a clear stratification of rewards across the four stages. The first stage offers the highest reward, indicating that the subtask of the chassis approaching the target area is relatively easy to learn. The rewards in the second and third stages gradually increase and stabilize, demonstrating the gradual development of the robotic arm's hovering alignment and pre-contact approach capabilities. The fourth stage offers a relatively low reward, indicating that the final insertion process is more challenging and demands stricter pose accuracy and contact control. In the later stages of training, the overall reward curve gradually stabilizes, and the error band converges significantly, indicating that the agent's exploratory behavior gradually decreases and the policy parameters stabilize. At this point, the robot can reliably complete the entire task process from chassis approach, end-effector hovering, pre-contact probing, to final insertion, demonstrating that the designed staged reward mechanism effectively guides the PPO algorithm to achieve task learning and convergence.

[0207] <Ablation Experiment>

[0208] To further analyze the role of motion and stiffness priors, while maintaining consistency in the task scenario, network structure, training algorithm, and other reward settings, only motion or stiffness-related prior information was removed, and the results were compared with the complete method. First, during human teaching, motion trajectory data of the mobile chassis and motion and stiffness trajectory data of the robotic arm were simultaneously collected to form complete vehicle-arm cooperative trajectory data. The prior trajectory of the mobile robot is as follows: Figure 11 and Figure 12 As shown.

[0209] based on Figure 11 and Figure 12 The real human teaching motion and stiffness data shown are used as priors for ablation experiments, and the experimental results are shown in Table 2.

[0210] Table 2 Comparison of reinforcement learning results with and without prior knowledge

[0211]

[0212] Table 2 shows that the introduction of prior knowledge effectively improves the training efficiency of reinforcement learning. Compared with methods without prior knowledge, introducing only motion trajectory priors or stiffness trajectory priors can shorten the training time, indicating that motion priors help constrain the policy search direction, while stiffness priors help improve the stability of force interactions during the contact phase. Furthermore, when both motion trajectory priors and stiffness trajectory priors are simultaneously integrated, the method exhibits a faster convergence speed in different training phases, with the average training time decreasing from 2.83 h without prior knowledge to 1.78 h. The results demonstrate that the joint guidance of motion and stiffness priors can simultaneously improve task execution search efficiency and contact process stability, reducing training time by approximately 37% compared to traditional reinforcement learning methods without prior knowledge, thus validating the effectiveness of the proposed prior fusion reinforcement learning strategy.

[0213] To further analyze the impact of different prior information on the final performance of reinforcement learning strategies, based on the evaluation of training efficiency, this paper compares various prior configurations in three aspects: task success rate, position tracking accuracy, and contact force control stability. The results are as follows: Figure 13 As shown.

[0214] exist Figures 13 to 15 In the diagram, A represents no prior knowledge, B represents prior knowledge only of the motion trajectory, C represents prior knowledge only of the stiffness trajectory, and D represents prior knowledge of both motion and stiffness trajectories. Compared to method A (without prior knowledge), the introduction of single prior information improved the performance of all strategies to varying degrees: the success rate of method B increased from 58.4% to 74.9%, and the root mean square error of position decreased from 8.2 mm to 5.1 mm; the success rate of method C increased to 71.6%, and the root mean square error of force decreased from 6.7 to 4.3. In contrast, method D, which simultaneously introduced prior knowledge of motion and stiffness trajectories, achieved the best results in all three metrics, with a success rate of 89.3%, a root mean square error of position reduced to 3.3 mm, and a root mean square error of force reduced to 2.8. The results show that prior knowledge of motion trajectory is more helpful in improving trajectory tracking accuracy, while prior knowledge of stiffness trajectory has a more significant effect on improving contact force control. When both types of prior information are introduced at the same time, the optimal comprehensive effect can be achieved in terms of task success rate, position accuracy and contact stability, thus verifying the effectiveness of multi-prior collaborative guidance for reinforcement learning strategy learning.

[0215] <Physical Real-Machine Experiment>

[0216] To further verify the deployability and practical performance of the proposed method in real systems, this paper conducts a simulation-to-real-machine experiment for shaft-hole assembly tasks. The optimized strategy trained in the IsaacLab simulation environment is deployed onto a real mobile robot, and the assembly task is completed using the hybrid motion controller for the robot arm designed in this section. The generalization ability and robustness of the proposed method in real-world environments are analyzed.

[0217] The physical simulation experiments primarily verify whether the mobile robot can stably complete assembly tasks, maintain smooth movement and reduce contact impact during contact and insertion, and ensure stable cooperation between the mobile chassis and the robotic arm during task execution. If the strategy trained in the simulation can reproduce the assembly behavior well on the real machine platform, it indicates that the method presented in this paper has good simulation-to-real-world transfer capability.

[0218] After the learning strategy was deployed, the actual operation process and experimental results of the mobile operation robot showed that the robot could complete continuous operations from workpiece approach and end adjustment to final assembly in real-world scenarios, verifying the deployment effect of the prior-guided reinforcement learning strategy and the feasibility of hybrid motion control of the robot arm.

[0219] <Comparative Experiment and Evaluation Indicators>

[0220] To verify the effectiveness of the proposed hybrid motion controller for the vehicle arm in complex, high-contact tasks, comparative and ablation experiments were conducted focusing on four aspects: task success, contact safety, execution efficiency, and chassis motion smoothness.

[0221] First, the Safe Success Rate (SSR) is used to measure the ability to complete tasks while meeting safety constraints. Specifically, in total... In this experiment, if a certain experiment simultaneously satisfies the conditions of successful task execution and peak contact force not exceeding the safety threshold... If no jamming or safety shutdown occurs during the execution process, the experiment is recorded as a safe success. SSR is defined as the proportion of safe success experiments to the total number of experiments.

[0222] Secondly, peak contact force is used. The impact intensity during the contact establishment and insertion propulsion phases. For a single experiment, Defined as task execution time interval Inner end contact force modulus The smaller this value, the weaker the instantaneous impact on the robot during contact, and the better the compliant interaction performance of the system.

[0223] Next, the execution efficiency is measured by task completion time, denoted as... This represents the total time elapsed from the start of the task to the completion of assembly. The smaller this value, the more efficiently the method can complete the assembly task.

[0224] Finally, a chassis smoothness index is used to measure the continuity of chassis speed changes. To eliminate the influence of the inconsistency between the dimensions of linear velocity and angular velocity, the chassis speed is normalized, and the smoothness index is defined as follows:

[0225]

[0226] in, and They represent the first Chassis linear velocity and angular velocity at each sampling time. and This indicates the maximum permissible values ​​for the chassis's linear velocity and angular velocity. This represents the number of discrete sampling points in a single experiment. To control the cycle. The smaller the value, the smoother the chassis speed change, which is more conducive to reducing the transmission of disturbances during the contact phase.

[0227] The comparison methods include the chassis-less virtual impedance method and the chassis-less stiffness adjustment method. The chassis-less virtual impedance method eliminates the chassis virtual impedance adjustment step, making the actual chassis execution speed directly equal to the strategy output reference speed. The chassis-less stiffness adjustment method retains the chassis virtual impedance shaping and arm coordination distribution, but removes phase-related stiffness priors and online stiffness corrections, using only fixed task space impedance parameters. .

[0228] <Experimental Results and Analysis>

[0229] The quantitative results of different methods on the four indicators are shown in Table 3. All results are expressed as mean. Statistics in the form of "standard deviation".

[0230] Table 3. Quantitative comparison results of the proposed method with different ablation methods

[0231]

[0232] Table 3 shows that the proposed method achieved the highest success rate for safe tasks while exhibiting the lowest peak contact force. This indicates that both chassis virtual impedance adjustment and phase-dependent stiffness adjustment contribute to improving reliable execution capabilities in complex contact tasks and reducing shocks during contact establishment and insertion. The most significant performance degradation occurred after removing chassis virtual impedance adjustment, demonstrating the crucial role of smooth chassis speed adjustment in overall performance. Removing phase-dependent stiffness adjustment also resulted in a deterioration in the method's overall performance, suggesting that fixed stiffness is insufficient to meet the contact requirements at different stages.

[0233] Therefore, chassis virtual impedance adjustment and phase-related stiffness adjustment improve system performance from the aspects of motion smoothness and contact compliance, respectively, indicating that the method in this paper has a better overall performance in complex, contact-rich tasks.

[0234] Finally, the impact of chassis virtual impedance adjustment on the system's dynamic characteristics is analyzed from the perspective of time-domain response. Figures 16 to 19 The time-varying velocity and acceleration curves of the proposed method and the method without virtual impedance are presented respectively.

[0235] according to Figure 16 and Figure 17 It can be seen that the method in this paper is... The proposed method exhibits better continuity and stability in both directional velocity and synthetic linear velocity, with smaller overall fluctuation amplitude. In contrast, the method without virtual impedance shows more pronounced velocity fluctuations, with the synthetic linear velocity peaking at approximately 0.5 m / s around 2 s, while the proposed method maintains a relatively smooth variation range. If the time-averaged synthetic linear velocity is used as the Cartesian spatial linear velocity index, the average linear velocity estimated based on the results shown in the figure is approximately 13.6% lower than that without virtual impedance. This result demonstrates that introducing virtual impedance adjustment into the chassis actuator layer effectively suppresses abrupt velocity changes, resulting in a smoother chassis motion response.

[0236] according to Figure 18 and Figure 19 It can be seen that the method in this paper... The directional acceleration and composite line acceleration fluctuations are significantly weaker than those of the method without virtual impedance, which exhibits prominent spike responses at multiple times, with the peak composite line acceleration exceeding 7 m / s². 2 The method presented in this paper maintains a relatively low overall impedance. This result demonstrates that chassis virtual impedance adjustment not only improves speed continuity but also effectively reduces instantaneous impacts and dynamic oscillations during motion, thereby reducing the transmission of chassis disturbances to the end-contact process. Combined with the quantitative results in Table 3, it can be shown that the method presented in this paper, while improving the smoothness of system motion, helps to enhance the stability and safety of the assembly task execution process.

[0237] Example 2:

[0238] A second aspect of this application provides a motion control device for a mobile robot based on guided reinforcement learning, the device comprising:

[0239] The prior skills acquisition module is used to acquire the robot's prior skills based on teaching data. The prior skills include time-varying stiffness trajectories and the pose trajectories of the chassis and end effector in the robot.

[0240] A deviation construction module is used to perform phase-based processing and conditional encapsulation on the prior skills to obtain phase-related reference priors in the task coordinate system, and to construct deviations in action and stiffness based on the robot's current phase and the reference priors, including:

[0241] The prior skills are transformed from the world coordinate system to the task coordinate system to obtain the prior reference. The prior reference includes the current phase, the phase expression of the pose trajectory of the chassis and the end effector of the robotic arm, and the phase expression of the time-varying stiffness trajectory.

[0242] Based on the prior reference, the phase expression of the chassis and robotic arm end effector pose trajectory corresponding to the look-ahead phase and the phase expression of the time-varying stiffness trajectory are determined.

[0243] The phase representation of the pose trajectory corresponding to the look-ahead phase, the phase representation of the time-varying stiffness trajectory, and the prior reference are encapsulated into the reference prior.

[0244] Based on the phase representation of the pose trajectories of the chassis and the end effector of the robotic arm in the reference prior, the phase representation of the time-varying stiffness trajectory, and the current phase, the deviation between the motion and stiffness is constructed.

[0245] The residual correction output module is used to concatenate the reference prior, the deviation, and the robot's current state data into observation data, receive the observation data through a pre-built PPO reinforcement learning model, and output the residual correction amount; the PPO reinforcement learning model is configured to learn the action compensation difference relative to the reference prior under different phases;

[0246] A control action construction module is used to construct a reference action based on the current phase and the reference prior, and to determine the control action by the difference between the reference action and the action compensation value.

[0247] The task execution module is used to generate low-level execution instructions based on the control actions through a pre-configured hybrid motion controller for the robot arm, and to control the robot to perform tasks based on the low-level execution instructions.

[0248] In summary, the motion control device for a mobile robot based on guided reinforcement learning provided in this embodiment obtains control actions by learning reference priors using a PPO reinforcement learning model. Compared with directly learning the overall action using a PPO model in the conventional case, this significantly reduces the time and exploration complexity of reinforcement learning. At the same time, the introduction of reference trajectory priors eliminates the need for precise simulation environment modeling for policy training, effectively narrowing the Sim-to-Real generalization gap and greatly improving the reliability of applications in industrial scenarios.

[0249] Example 3:

[0250] Thirdly, embodiments of this application provide an electronic device, Figure 20 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 20 As shown, the electronic device may include a processor 11 and a memory 12 storing computer program instructions.

[0251] Specifically, the processor 11 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0252] The memory 12 may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory 12 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 12 may include removable or non-removable (or fixed) media. Where appropriate, the memory 12 may be internal or external to a data processing device. In a particular embodiment, the memory 12 is non-volatile memory. In a particular embodiment, the memory 12 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0253] The memory 12 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 11.

[0254] The processor 11 reads and executes computer program instructions stored in the memory 12 to implement any of the guided reinforcement learning-based hybrid control methods for mobile robots in the above embodiments.

[0255] In one embodiment, the electronic device may further include a communication interface 13 and a bus 10. Wherein, as... Figure 20 As shown, the processor 11, memory 12, and communication interface 13 are connected through bus 10 and communicate with each other.

[0256] The communication interface 13 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 13 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0257] Bus 10 includes hardware, software, or both, that couples components of an electronic device together. Bus 10 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 10 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 10 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0258] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0259] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0260] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0261] Fourthly, embodiments of this application provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the hybrid control method for a mobile operating robot based on guided reinforcement learning provided in the first aspect.

[0262] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0263] In a possible implementation, this application can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform steps implementing the hybrid control method for a mobile operating robot based on guided reinforcement learning provided in the first aspect.

[0264] The program code for executing this application can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0265] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A hybrid control method for a mobile robot based on guided reinforcement learning, characterized in that, The method includes: Prior skills of the robot are acquired based on teaching data, including time-varying stiffness trajectory and pose trajectories of the chassis and end effector in the robot. The prior skills are phase-processed and conditionally encapsulated to obtain phase-related reference priors in the task coordinate system. Based on the robot's current phase and the reference priors, deviations in action and stiffness are constructed, including: The prior skills are transformed from the world coordinate system to the task coordinate system to obtain the prior reference. The prior reference includes the current phase, the phase expression of the pose trajectory of the chassis and the end effector of the robotic arm, and the phase expression of the time-varying stiffness trajectory. Based on the prior reference, the phase expression of the chassis and robotic arm end effector pose trajectory corresponding to the look-ahead phase and the phase expression of the time-varying stiffness trajectory are determined. The phase representation of the pose trajectory corresponding to the look-ahead phase, the phase representation of the time-varying stiffness trajectory, and the prior reference are encapsulated into the reference prior; the reference prior is configured as follows: ； in, As a reference prior at time t, For the current mission phase, For the look-ahead phase increment, This represents the phase of the chassis trajectory at time t. This represents the phase of the robot arm's end effector trajectory at time t. This is the phase representation of the time-varying stiffness trajectory at time t. This represents the phase expression of the chassis trajectory corresponding to the forward-looking phase. This represents the phase expression of the robotic arm's end-effector trajectory corresponding to the look-ahead phase. This is the phase representation of the time-varying stiffness trajectory corresponding to the look-ahead phase; Based on the phase representation of the pose trajectories of the chassis and the end effector of the robotic arm in the reference prior, the phase representation of the time-varying stiffness trajectory, and the current phase, the deviation between the motion and stiffness is constructed. The reference prior, the bias, and the robot's current state data are concatenated into observation data. The observation data is received by a pre-built PPO reinforcement learning model, which outputs residual correction values. The PPO reinforcement learning model is configured to learn the action compensation difference relative to the reference prior under different phases. A reference action is constructed based on the current phase and the reference prior, and a control action is determined by the difference between the reference action and the action compensation value. The robot is controlled to perform tasks based on the control actions generated by the pre-configured hybrid motion controller.

2. The hybrid control method for a mobile robot based on guided reinforcement learning according to claim 1, characterized in that, The prior skills of the robot acquired based on teaching data include: The original timing information is obtained based on the teaching information, and the original timing information includes: original chassis pose timing, original end pose timing, and original time-varying stiffness timing. The prior skills are obtained by performing time-series alignment processing on the original time-series information using the Impedance-PRIMP algorithm.

3. The hybrid control method for a mobile robot based on guided reinforcement learning according to claim 1, characterized in that, The deviation between the action and stiffness is configured as follows: ； in, The deviation at time t, For the vectorization operator of the logarithmic map, It is the Frobenius norm. For the current mission phase, As the chassis, For robotic arm end effectors, Let be the inverse matrix of the phase representation of the chassis trajectory at time t. The actual position and pose of the chassis at time t. Let be the inverse matrix of the phase expression of the robot arm's end-effector trajectory at time t. Let be the actual pose of the robotic arm's end effector at time t. This is the phase representation of the time-varying stiffness trajectory at time t. Let be the time-varying stiffness parameter at time t.

4. The hybrid control method for a mobile robot based on guided reinforcement learning according to claim 1, characterized in that, The robot's current state data is configured as follows: ； in, and For the observation of the speed of the moving chassis, , , and These represent the position, attitude, linear velocity, and angular velocity observations of the end effector relative to the assembly target, respectively. This refers to the action taken at the previous moment.

5. The hybrid control method for a mobile robot based on guided reinforcement learning according to claim 1, characterized in that, The reward function for the residual PPO reinforcement learning model is configured as follows: ； in, This represents the original task reward, used to characterize the direct feedback on the completion degree of the core task objective after the strategy executes the action at time t. This indicates a soft guide term related to the reference prior. These are the weighting coefficients for the soft-guided term, used to control its size. The soft guidance term is used to encourage the strategy to prioritize searching for feasible solutions within a priori reasonable region, specifically: ； in, The penalty weight for chassis position deviation. The penalty weight for pose deviation at the robotic arm's end effector. This is the penalty weight for stiffness deviation.

6. The hybrid control method for a mobile robot based on guided reinforcement learning according to claim 1, characterized in that, The pre-configured hybrid motion controller for the vehicle arm generates underlying execution commands based on the control actions, including: The control parameters for the chassis and the robotic arm are obtained based on the control actions. The control parameters of the chassis are subjected to motion scale scaling and impedance shaping to obtain the actual execution speed of the chassis. Based on a pre-built joint speed coordination and allocation optimization model, the desired joint speed of the robotic arm is obtained according to the control parameters of the robotic arm. Based on the desired joint speed of the robotic arm, the impedance and stiffness of the robotic arm are adjusted to obtain the joint speed after impedance correction. The underlying execution command is generated based on the impedance-corrected joint speed and the actual execution speed of the chassis.

7. A hybrid control method for a mobile robot based on guided reinforcement learning according to claim 6, characterized in that, The pre-built joint velocity coordination allocation optimization model is configured as follows: ； in, Let be the desired joint velocity of the robotic arm. For the joint angle of the robotic arm, , For joint velocity, , Jacobi is the contribution of the robotic arm to the end effector speed. Jacobi for the chassis's contribution to the final velocity. The actual operating speed of the chassis. The error weighting matrix for the final task. Let be the joint velocity regularization coefficient, and , Let be the velocity spinor of the end effector, where , The desired end-point space velocity.

8. A hybrid control method for a mobile robot based on guided reinforcement learning according to claim 6, characterized in that, The process of adjusting the impedance and stiffness of the robotic arm based on the desired joint speed to obtain the impedance-corrected joint speed includes: Based on the expected joint velocity and the actual end-effector velocity of the robotic arm, as well as the reference equilibrium pose and the actual pose of the robotic arm, a pose error is constructed. The desired stiffness of the execution layer is determined based on the stiffness prior trajectory, and the mapping relationship between the desired stiffness, the pose error and the desired interaction force of the robotic arm is constructed. Solve the mapping relationship to obtain the joint velocity after impedance correction.

9. A hybrid control device for a mobile robot based on guided reinforcement learning, characterized in that, The device includes: The prior skills acquisition module is used to acquire the robot's prior skills based on teaching data. The prior skills include time-varying stiffness trajectories and the pose trajectories of the chassis and end effector in the robot. A deviation construction module is used to perform phase-based processing and conditional encapsulation on the prior skills to obtain phase-related reference priors in the task coordinate system, and to construct deviations in action and stiffness based on the robot's current phase and the reference priors, including: The prior skills are transformed from the world coordinate system to the task coordinate system to obtain the prior reference. The prior reference includes the current phase, the phase expression of the pose trajectory of the chassis and the end effector of the robotic arm, and the phase expression of the time-varying stiffness trajectory. Based on the prior reference, the phase expression of the chassis and robotic arm end effector pose trajectory corresponding to the look-ahead phase and the phase expression of the time-varying stiffness trajectory are determined. The phase representation of the pose trajectory corresponding to the look-ahead phase, the phase representation of the time-varying stiffness trajectory, and the prior reference are encapsulated into the reference prior; the reference prior is configured as follows: ； in, As a reference prior at time t, For the current mission phase, For the look-ahead phase increment, This represents the phase of the chassis trajectory at time t. This represents the phase of the robot arm's end effector trajectory at time t. This is the phase representation of the time-varying stiffness trajectory at time t. This represents the phase expression of the chassis trajectory corresponding to the forward-looking phase. This represents the phase expression of the robotic arm's end-effector trajectory corresponding to the look-ahead phase. This is the phase representation of the time-varying stiffness trajectory corresponding to the look-ahead phase; Based on the phase representation of the pose trajectories of the chassis and the end effector of the robotic arm in the reference prior, the phase representation of the time-varying stiffness trajectory, and the current phase, the deviation between the motion and stiffness is constructed. The residual correction output module is used to concatenate the reference prior, the deviation, and the robot's current state data into observation data, receive the observation data through a pre-built PPO reinforcement learning model, and output the residual correction amount; the PPO reinforcement learning model is configured to learn the action compensation difference relative to the reference prior under different phases; A control action construction module is used to construct a reference action based on the current phase and the reference prior, and to determine the control action by the difference between the reference action and the action compensation value. The task execution module is used to generate low-level execution instructions based on the control actions through a pre-configured hybrid motion controller for the robot arm, and to control the robot to perform tasks based on the low-level execution instructions.

Images