A Robot Whole-Body Control Method and System Based on Generative Reinforcement Learning

Abstract

This invention discloses a robot whole-body control method and system based on generative reinforcement learning. It constructs a multimodal state encoder based on a neural network, encoding joint angular positions, velocities, torques, chassis pose, chassis velocity, and RGBD visual images into a joint state representation. A generative whole-body control strategy based on a conditional diffusion model is constructed, adding noise to the initial control quantity through a forward diffusion process and iteratively denoising through a backward diffusion process to generate the whole-body control quantity. A composite optimization objective function is established, including diffusion model optimization terms and value network optimization terms. The strategy, encoder, and value network parameters are synchronously updated based on an interactive dataset. Initial noisy actions are sampled from a standard normal distribution, and the final control quantity is obtained through iterative denoising via backward diffusion, driving the chassis and robotic arm to perform collaborative execution. This addresses the problems of insufficient expressive power of existing end-to-end reinforcement learning strategies, difficulty in adapting to high-dimensional action spaces, and low sample efficiency.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence and robot control technology, specifically to a method and system for the whole-body control of a mobile operating robot based on multimodal generative reinforcement learning. Background Technology

[0002] Mobile robots have wide applications in industrial manufacturing, home services, aerospace, and other fields. Effective control of mobile robots directly affects their operational efficiency and control performance. Traditional control methods separate the control of the mobile chassis and the robotic arm to achieve mobile operation, which has limitations such as low operational efficiency and poor adaptability to dynamic environments, severely restricting the autonomous operation level of service robots.

[0003] Reinforcement learning can learn an end-to-end policy through continuous interaction, enabling coordinated full-body control of a mobile chassis and robotic arm, greatly improving the operational efficiency and adaptability to complex environments of mobile robots. However, existing end-to-end reinforcement learning methods have two major drawbacks: first, the policy network has limited expressive power, making it difficult to adapt to high-dimensional full-body motion spaces and capture multimodal behavior distributions, resulting in poor full-body control performance; second, when directly processing multimodal high-dimensional data, a large amount of interactive training data is required to learn an effective policy, resulting in low sample efficiency.

[0004] There are also whole-body motion planning and control methods for wheeled robots, which realize coordinated motion planning of the torso and arm based on hierarchical optimization. However, these methods require modeling of the robot's body dynamics, have poor expressiveness of control strategies, poor adaptability to dynamic environments, and have not solved the problems of high-dimensional whole-body motion space adaptation and efficient utilization of multimodal data.

[0005] Therefore, there is an urgent need for a reinforcement learning strategy learning framework with strong expressiveness to achieve end-to-end optimization of the whole-body movements of mobile robots under multimodal observation, thereby improving sample efficiency and control accuracy. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a robot whole-body control method and system based on generative reinforcement learning, which realizes end-to-end optimization of the whole-body movements of a mobile robot under multimodal observation, and improves sample efficiency and control accuracy.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for full-body control of a mobile manipulator based on multimodal generative reinforcement learning includes the following steps: S1. Construct a multimodal state encoder to encode the multimodal observation data into a joint state representation; the observation data includes joint angle position, velocity, torque, chassis pose, chassis velocity, and RGBD visual image; S2, construct a generative whole-body control strategy, map the joint state representation to the whole-body control quantity; the strategy is based on a noise prediction module fitted by a multilayer perceptron, the forward diffusion process adds noise to the initial control quantity, and the reverse diffusion process iteratively denoises and generates the final control quantity; S3, establish a composite optimization objective function that includes diffusion model optimization terms and value network optimization terms, and synchronously update the optimization strategy, encoder and value network parameters based on the interactive dataset; S4 deploys the optimized strategy on the mobile robot, collects observation data in real time to generate a joint state representation, samples the initial noisy action from the standard normal distribution, and obtains the final control quantity by iteratively denoising through the back diffusion process, driving the chassis and robotic arm to execute collaboratively.

[0008] Furthermore, the multimodal state encoder is composed of a first multilayer perceptron and a convolutional neural network; the first multilayer perceptron is used to encode joint angular position, joint velocity, joint torque, chassis pose, and chassis velocity time-series data; the convolutional neural network is used to encode RGBD visual image data; the joint state representation is obtained by splicing the outputs of the first multilayer perceptron and the convolutional neural network.

[0009] Furthermore, the generative whole-body control strategy is constructed based on a conditional diffusion model, with a noise prediction module fitted by a second multilayer perceptron as the core, mapping a given joint state representation to a probability distribution of an action sequence.

[0010] Furthermore, the value network adopts a dual-Q network structure, which is constructed from a third multilayer perceptron and a fourth multilayer perceptron.

[0011] Furthermore, the motion space of the whole-body control quantities of the mobile operation robot includes the chassis motion quantity set and the robotic arm motion quantity set; the chassis motion quantity set includes linear velocity and angular velocity; the robotic arm motion quantity set includes the angular velocity of each joint and the opening and closing degree of the end effector gripper; the motion space satisfies preset constraints.

[0012] Furthermore, the conditional diffusion model optimization term predicts the noise in the noisy action of the time step based on the noise prediction module, and optimizes the generative whole-body control strategy parameters and multimodal state encoder parameters by minimizing the mean square error between the predicted noise and the actual noise; the cumulative attenuation coefficient of the time step is calculated based on the preset noise scheduling parameters.

[0013] Furthermore, the value network optimization term is based on a double-Q network structure, and the target Q value is calculated through the target Q network update formula. Overestimation of the value is avoided through double-Q learning. The target Q network parameters are updated using a moving average method.

[0014] Specifically, the target Q network update formula is:

[0015] in For the first A target Q network ( ), parameters are , For the first A target Q network ( ), parameters are , For time step The joint state indicates that For time step The total control volume of the mobile operating robot. For time step Multimodal observation data, For target-generative whole-body control, Sampled from the target-generative whole-body control policy network Initial action of time step, For the reward function, γ The discount factor is used to avoid overestimation through double-Q learning.

[0016] Furthermore, the composite optimization objective function is a weighted sum of the optimization terms of the conditional diffusion model and the value network optimization terms, wherein the balance coefficient is used to adjust the weight ratio of the two optimization terms.

[0017] Specifically, the objective function, which includes both the conditional diffusion model optimization term and the value function optimization term, is:

[0018] in This is the balance coefficient.

[0019] Furthermore, the iterative update formula for the reverse diffusion process is as follows: based on the noise prediction module predicting the noise in the noisy action at the current time step, and combining the single-step attenuation coefficient and the cumulative attenuation coefficient of the time step, the noise is gradually removed until the final whole-body control quantity is obtained.

[0020] Furthermore, the interactive dataset was obtained by manually controlling the robot to complete the part grasping task and combining it with a pre-trained basic strategy.

[0021] Furthermore, the training employs the Adam optimizer, with the learning rate, batch size, and number of training epochs dynamically adjusted based on convergence performance.

[0022] Furthermore, the input of the convolutional neural network of the multimodal state encoder is the concatenated RGB image channel and depth image channel. The network structure includes multiple convolutional layers and fully connected layers, and the activation function is ReLU.

[0023] A full-body control system for a mobile manipulator based on multimodal generative reinforcement learning, comprising: The multimodal observation data acquisition module is used to acquire joint angular positions, joint velocities, joint torques, chassis pose, chassis speed, and RGBD visual images of the mobile robot in real time. A multimodal state encoder module is used to encode the multimodal observation data into a joint state representation; A generative whole-body control strategy module is used to map the joint state representation into whole-body control variables of a mobile manipulator based on a conditional diffusion model. The network training optimization module is used to establish a composite optimization objective function that includes optimization terms of the conditional diffusion model and optimization terms of the value network, and to update the parameters of each network synchronously. The strategy reasoning module is used to generate the final whole-body control quantity through the back diffusion process of the conditional diffusion model to generate noise reduction, and drive the chassis and robotic arm to perform collaborative execution.

[0024] Compared with existing technologies, the present invention, a method and system for whole-body control of a mobile manipulator based on multimodal generative reinforcement learning, has the following advantages: (1) Strong strategy expression ability: This invention uses a conditional diffusion model to construct a generative whole-body control strategy. The diffusion model has strong expressiveness for complex multimodal distributions, can adapt to high-dimensional whole-body motion space, effectively capture the collaborative control law of chassis and robotic arm, avoid outlier action output, and significantly improve the expressive ability and generalization performance of the control strategy.

[0025] (2) End-to-end whole-body coordination: This invention uses a multi-modal state encoder to fuse multi-source heterogeneous data such as joint angle position, joint velocity, joint torque, chassis pose, chassis velocity and RGBD visual image to generate a unified joint state representation, thereby realizing end-to-end whole-body coordination control of the mobile chassis and the robotic arm. It eliminates the need for precise dynamic modeling of the robot body and reduces the complexity of the control system.

[0026] (3) High sample efficiency: This invention establishes a composite optimization objective function that includes conditional diffusion model optimization term and value function optimization term. It avoids overestimation of value function through double Q network structure. Combined with Adam optimizer to synchronously update multimodal state encoder, generative whole body control strategy and value function parameters, it achieves efficient policy learning under limited interactive samples and significantly improves training sample efficiency.

[0027] (4) High control accuracy and safety: This invention uses the reverse diffusion process of the conditional diffusion model to perform strategy reasoning, samples the initial noisy action from the standard normal distribution, and obtains the final whole-body control quantity through iterative denoising. The action space meets the preset constraints, ensuring the rationality and safety of control quantities such as chassis linear velocity, angular velocity, robotic arm joint angular velocity and gripper opening and closing degree. Attached Figure Description

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 A flowchart illustrating a method for full-body control of a mobile manipulator based on multimodal generative reinforcement learning, provided in a first aspect embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the diffusion process of the generative whole-body control strategy network in the whole-body control method for a mobile operating robot provided in the first aspect embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the update process of the multimodal state encoder and the generative whole-body control strategy network in the whole-body control method for a mobile operating robot provided in the first aspect embodiment of the present invention.

[0029] Figure 4 This is a physical simulation diagram of a mobile operating robot provided in a second aspect embodiment of the present invention.

[0030] Figure 5 A schematic diagram of a mobile operating robot provided in a second aspect embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] Example 1 refer to Figure 1 The present invention proposes a method for full-body control of a mobile robot based on multimodal generative reinforcement learning, comprising the following steps: Step S1: Construct a neural network-based multimodal state encoder, which is used to encode the multimodal observation data of the mobile manipulator into a joint state representation; Step S2: Construct a generative whole-body control strategy for the mobile manipulator based on the conditional diffusion model, and map the joint state representation to the whole-body control variables of the mobile manipulator; Step S3: Establish an optimization objective function that includes conditional diffusion model optimization terms and value function optimization terms, and optimize the parameters of the generative whole-body control strategy, multimodal state encoder, and value function; Step S4: Deploy the optimized generative whole-body control strategy on the mobile operation robot prototype, and perform strategy reasoning through the reverse process of the conditional diffusion model to achieve whole-body collaborative control of the mobile operation robot.

[0033] The physical prototype of the mobile operation robot mentioned in this invention takes a mobile operation robot including a UR robotic arm and a differential drive chassis as an example; The motion control of the mobile operation robot mentioned in this invention takes the whole-body collaborative control of a mobile operation robot used for assembling small parts as an example.

[0034] Furthermore, in step S1, a time step is set. Multimodal observation data of mobile operating robots The multimodal observation data specifically includes: joint angle position vectors. Joint velocity vector Joint torque vector Chassis position Chassis speed RGB images Depth images ,Right now:

[0035] The multimodal state encoder consists of a first multilayer perceptron and a convolutional neural network. The input layer dimension of the first multilayer perceptron is... The hidden layer is Layer, number of neurons is The activation function used is ReLU; the output layer dimension is... ; used to encode the above structured data, the encoding process is as follows:

[0036] The input to the concatenated neural network is the... Channel image (RGB image) Channel and depth images (channel), its network structure is Number of convolutional layers (number of convolutional kernels) Core size stride is (with ReLU activation function) and Fully connected layers (with a number of neurons) ); used to encode visual image data, the encoding process is as follows:

[0037] The joint state representation is obtained by concatenating the outputs of the first multilayer perceptron and the convolutional neural network:

[0038] in for dimensional vector, These are the fitting parameters for the multimodal state encoder, enabling the fusion and redundancy compression of multimodal observation data.

[0039] Further, in step S2, the generative whole-body control strategy is modeled as follows:

[0040] in For generative whole-body control policy network, the parameters are: , For a given joint state representation Generate action sequence afterward probability distribution Given a joint state With time step Noisy actions The probability distribution after that, For the whole-body control of mobile operating robots express Follow the mean variance is The Gaussian distribution.

[0041] The forward diffusion process of generative whole-body control strategies is as follows:

[0042]

[0043] in Given the initial whole-body control variables of the mobile robot With joint state representation The time step generated later is arrive The probability distribution, For a given time step action and joint state representation Post-generation time step Noisy Actions The probability distribution, This represents the initial control parameters for the entire mobile robot. Preset noise scheduling parameters.

[0044] Full-body control of mobile operating robots The action space has 9 dimensions, specifically defined as:

[0045] The constraint ranges for each parameter are as follows: (Chassis linear velocity) (Chassis angular velocity) (Robotic arm joint angular velocity). (Gripper opening / closing degree, 0 for closed, 1 for open). Motion space satisfies:

[0046] in .

[0047] The objective function, which includes both the conditional diffusion model optimization term and the value function optimization term, is:

[0048] Where the balance coefficient The value is 0.3, and the optimization term for the conditional diffusion model is:

[0049] in These are parameters for a generative whole-body control strategy. These are the parameters for the multimodal state encoder. for Uniform distribution on Indicates random noise. For multimodal observation data, For interactive datasets, This is the noise prediction module for the policy network fitted to the second multilayer perceptron. For time step The cumulative attenuation coefficient is defined as follows: .

[0050] The core of the conditional diffusion model optimization term is the noise prediction module fitted by the second multilayer perceptron. The input layer dimension of the second multilayer perceptron is The hidden layer is Layer, number of neurons is The activation function is Mish; the output layer dimension is... (Consistent with the dimensions of the whole-body control of the mobile robot); diffusion time step ,noise The scheduling parameters adopt linear scheduling ( ).

[0051] The value function optimization term is:

[0052] in For value function parameters, The initial action sampled from the generative whole-body control strategy. The value function employs a dual-Q network structure, including... and .

[0053] The value function is constructed from the third and fourth multilayer perceptrons, with the input layer being... Dimension (the sum of the joint state representation and the total control variables of the mobile robot); hidden layer is Layer, number of neurons is The activation function is Mish; the output layer dimension is... Output the action value respectively and .

[0054] The target Q network update formula is:

[0055] in For the first A target Q-network with parameters as follows: , For the first A target Q network ( ), parameters are , For time step The joint state indicates that For time step The amount of control over the entire robot. For time step Multimodal observation data, A target-generating whole-body control strategy. Sampled from the target-generative whole-body control policy network Initial action of time step, For the reward function, γ The discount factor is used to avoid overestimation through double-Q learning.

[0056] Furthermore, in step S3, the robot is manually controlled to complete 500 part-grabbing tasks. Combined with data collected from pre-trained basic strategies, an interactive dataset is constructed.

[0057] Training uses the Adam optimizer (learning rate is 100%). The batch size is 256, the training epochs are 1000, and each training epoch includes: Sample batch data and calculate and The parameters of the multimodal state encoder are updated synchronously through gradient descent. Generative whole-body control strategy parameters Dual-Q network parameters .

[0058] Every 100 gradient steps, the target network parameters are updated using a moving average:

[0059] in , These correspond to two Q networks respectively.

[0060] Furthermore, in step S4, the optimized model is deployed on a physical prototype of a mobile manipulation robot, which includes a UR robotic arm and a differential drive chassis. The strategy reasoning process is as follows: First, real-time multimodal observation data is collected, and a joint state representation is generated using a multimodal state encoder. Then, initial noisy actions are sampled from a standard normal distribution, and a noise prediction module fitted by a second multilayer perceptron is used for iterative back-diffusion denoising to obtain the final whole-body control variables for the mobile robot. The bottom-level controller receives the whole-body control variables of the mobile robot, repeats the above reasoning process, and dynamically updates the actions. The reasoning process is as follows:

[0061] in For time step The single-step decay coefficient, until Obtain the final robot whole-body control quantity The chassis and robotic arm work together to execute commands.

[0062] When the strategy outputs the gripper to close, the tactile sensor detects the pressure value, confirming a successful grip.

[0063] Compared with existing technologies, the core innovation of the multimodal generative reinforcement learning-based whole-body control method for mobile manipulators provided in this invention lies in the generative whole-body control strategy for mobile manipulators based on a conditional diffusion model. This strategy leverages the strong expressiveness of the diffusion model to adapt to a high-dimensional motion space, capturing the collaborative control patterns of the chassis and robotic arm, and avoiding outlier actions. Multimodal observation data encoding is merely a data preprocessing method, achieving the fusion and compression of heterogeneous data. This invention effectively solves the problems of insufficient policy expressiveness, low sample efficiency, and poor control performance in the whole-body control of mobile manipulators caused by existing end-to-end reinforcement learning, achieving high-performance whole-body collaborative control.

[0064] Although this invention uses terms such as encoder, policy network, and diffusion model frequently, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

[0065] Example 2 This embodiment takes a physical prototype of a mobile operation robot including a UR robotic arm and a differential drive chassis as the implementation object, and takes the whole-body collaborative control of the mobile operation robot for small parts grasping (suitable for parts that can be manipulated by human fingers) as the application scenario, and describes in detail the specific implementation process of the technical solution of the present invention.

[0066] 1. System Hardware Configuration like Figure 4 and Figure 5 As shown, the physical prototype of the mobile manipulation robot includes: a differential drive mobile chassis equipped with a drive wheel and an encoder for real-time feedback of chassis pose and speed; a UR5 six-DOF robotic arm mounted on a platform above the chassis, with angle encoders and torque sensors at each joint; an end effector that is a two-finger electric gripper equipped with a tactile sensor; an RGBD camera mounted above the wrist of the robotic arm for observing the pose of the target part; and an industrial computer equipped with an NVIDIA 3090 graphics card for neural network inference and training.

[0067] 2. Multimodal observation data acquisition At the start of each control cycle, the multimodal observation data acquisition module synchronously collects the following data: (1) Joint angle position vector of robotic arm: The angle values ​​of 6 joints are read by UR5 joint encoder, with a range of [-2π, 2π]; (2) Joint velocity vectors of the robotic arm: The angular velocities of the six joints are obtained by differential calculation using the encoder; (3) Joint torque vector of robotic arm: The output torque of 6 joints is read by the joint torque sensor; (4) Chassis pose: Two-dimensional plane coordinates are obtained through the chassis encoder. and orientation angle ; (5) Chassis speed: including linear speed With angular velocity ; (6) RGB image: size 128×128, 3 channels; (7) Depth image: size 128×128, 1 channel.

[0068] The above data constitutes multimodal observation data at time step t.

[0069] 3. Construction of a multimodal state encoder The multimodal state encoder consists of a first multilayer perceptron and a convolutional neural network.

[0070] The first multilayer perceptron has an input layer dimension of 23 (6-dimensional joint angle position + 6-dimensional joint velocity + 6-dimensional joint torque + 3-dimensional chassis pose + 2-dimensional chassis velocity), three hidden layers with 512 neurons in each layer, using ReLU activation function, and an output layer dimension of 128. The encoding process is as follows: the joint angle position, joint velocity, joint torque, chassis pose, and chassis velocity are concatenated into a 23-dimensional vector, input into the first multilayer perceptron, and the output is a 128-dimensional encoded vector.

[0071] The input to the convolutional neural network is a concatenated 4-channel image (3 RGB channels + 1 depth channel), with an image size of 128×128. The network structure consists of four convolutional layers: the first layer has 32 kernels, an 8×8 kernel size, and a stride of 4; the second layer has 64 kernels, a 4×4 kernel size, and a stride of 2; the third layer has 64 kernels, a 3×3 kernel size, and a stride of 1; and the fourth layer has 128 kernels, a 3×3 kernel size, and a stride of 1. All convolutional layers use ReLU as the activation function. This is followed by two fully connected layers with 512 and 128 neurons, respectively. The encoding process is as follows: the RGB and depth images are concatenated to form a 4-channel input, spatial features are extracted by the convolutional layers, and then compressed into a 128-dimensional encoded vector by the fully connected layers.

[0072] The joint state representation is obtained by splicing the 128-dimensional vector output by the first multilayer perceptron and the 128-dimensional vector output by the convolutional neural network, resulting in a final dimension of 256, thus achieving the fusion and redundancy compression of multimodal observation data.

[0073] 4. Construction of Generative Whole-Body Control Strategy The generative whole-body control strategy is based on a conditional diffusion model and uses a noise prediction module fitted by a second multilayer perceptron as its core.

[0074] The motion space has 9 dimensions, specifically defined as: chassis linear velocity (range [-0.5, 0.5] m / s), chassis angular velocity (range [-1.0, 1.0] rad / s), angular velocities of the six joints of the robotic arm (range [-2.0, 2.0] rad / s for each joint), and gripper opening / closing degree (range [0, 1], 0 for closed and 1 for open).

[0075] The second multilayer perceptron has an input layer dimension of 265 (256-dimensional joint state representation + 9-dimensional noisy action + 1-dimensional time step encoding), 3 hidden layers, each with 512 neurons, and uses the Mish activation function. The output layer dimension is 9 (consistent with the dimension of the whole body control).

[0076] Diffusion time step Set to 100, the noise scheduling parameter adopts linear scheduling: The cumulative attenuation coefficient increases linearly from 0.0001 to 0.02. The cumulative attenuation coefficient is defined as: ,in From 1 to t.

[0077] Forward diffusion process: from initial whole-body control level Departure, according to Gaussian noise is added gradually, after Step to obtain noisy action .

[0078] 5. Composite Optimization Objective Function The composite optimization objective function is: The balance coefficient Set it to 0.3.

[0079] Conditional diffusion model optimization terms Based on noise prediction module Predicting time steps Noisy Actions The noise in the image, and compared with the actual noise. Compare and calculate the mean squared error. Value function optimization term. Timing difference error is calculated based on a dual-Q network.

[0080] 6. Training Process By manually controlling the robot to complete 500 parts grasping tasks and collecting data in conjunction with pre-trained basic strategies, an interactive dataset was constructed. Training was performed using the Adam optimizer with a learning rate of 3×10⁻⁶. -4 The batch size is 256, and the training rounds are 1000.

[0081] Each training round includes: randomly sampling a batch of data from the interactive dataset, and computing... and The parameters of the multimodal state encoder are updated synchronously through gradient descent. Generative whole-body control strategy parameters Dual-Q network parameters .

[0082] 7. Strategic Reasoning and Execution The optimized model was deployed on a physical prototype of the mobile manipulator robot. The policy inference process in each control cycle (20Hz) is as follows: (1) Real-time acquisition of current multimodal observation data; (2) Generate joint state representations through a multimodal state encoder; (3) Sample the initial noisy action from the standard normal distribution; (4) From the time step Start reverse iteration, predict noise based on the noise prediction module, and update the action; (6) Repeat step (4) until Thus, the final amount of control over the whole body is obtained; (7) The bottom controller receives the whole body control quantity, executes the chassis linear velocity and angular velocity to drive the chassis movement, executes the angular velocity of each joint of the robotic arm to control the movement of the robotic arm, and executes the gripper opening and closing degree to control the gripper action. (8) When the strategy outputs the gripper to close and the tactile sensor detects that the pressure value exceeds the threshold of 0.5N, the part is confirmed to be successfully gripped.

[0083] 8. Experimental Results In the task of grasping small parts, the mobile robot in this embodiment successfully achieved full-body coordinated control of chassis movement and robotic arm grasping actions. Compared with traditional hierarchical control methods, the part grasping success rate increased from 65% to 73%, achieving stable full-body coordinated motion and verifying the effectiveness of the present invention in achieving high-performance full-body coordinated control under multimodal observation.

[0084] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A robot whole body control method based on generative reinforcement learning, characterized by Includes the following steps: S1. Construct a multimodal state encoder to encode the multimodal observation data into a joint state representation; the observation data includes joint angle position, velocity, torque, chassis pose, chassis velocity, and RGBD visual image; S2, construct a generative whole-body control strategy, map the joint state representation to the whole-body control quantity; the strategy is based on a noise prediction module fitted by a multilayer perceptron, the forward diffusion process adds noise to the initial control quantity, and the reverse diffusion process iteratively denoises and generates the final control quantity; S3, establish a composite optimization objective function that includes diffusion model optimization terms and value network optimization terms, and synchronously update the optimization strategy, encoder and value network parameters based on the interactive dataset; S4 deploys the optimized strategy, collects observation data in real time to generate a joint state representation, samples the initial noisy action from the standard normal distribution, iterates the back diffusion process to remove noise and obtain the final control quantity, and drives the chassis and robotic arm to perform collaborative execution.

2. The method of claim 1, wherein, The multimodal state encoder consists of a first multilayer perceptron and a convolutional neural network; the first multilayer perceptron is used to encode joint angular position, joint velocity, joint torque, chassis pose, and chassis velocity time-series data; the convolutional neural network is used to encode RGBD visual image data; the joint state representation is obtained by splicing the outputs of the first multilayer perceptron and the convolutional neural network.

3. The method of claim 1, wherein, The generative whole-body control strategy is constructed based on a conditional diffusion model, with a noise prediction module fitted by a second multilayer perceptron as the core, which maps a given joint state representation into a probability distribution of an action sequence.

4. The method according to claim 1, characterized in that, The value network adopts a dual-Q network structure, which is constructed by a third multilayer perceptron and a fourth multilayer perceptron.

5. The method according to claim 1, characterized in that, The motion space of the mobile robot's whole-body control quantities includes the chassis motion quantity set and the robotic arm motion quantity set; the chassis motion quantity set includes linear velocity and angular velocity; the robotic arm motion quantity set includes the angular velocity of each joint and the opening and closing degree of the end effector gripper; the motion space satisfies preset constraints.

6. The method according to claim 1, characterized in that, The conditional diffusion model optimization term is based on the noise prediction module to predict the noise in the noisy action of the time step, and optimizes the generative whole-body control strategy parameters and multimodal state encoder parameters by minimizing the mean square error between the predicted noise and the actual noise; the cumulative attenuation coefficient of the time step is calculated based on the preset noise scheduling parameters.

7. The method according to claim 1, characterized in that, The value network optimization term is based on a double-Q network structure. The target Q value is calculated through the target Q network update formula, and overestimation is avoided through double-Q learning. The target Q network parameters are updated using a moving average method.

8. The method according to claim 1, characterized in that, The composite optimization objective function is a weighted sum of the optimization terms of the conditional diffusion model and the value network, where the balance coefficient is used to adjust the weight ratio of the two optimization terms.

9. The method according to claim 1, characterized in that, The iterative update formula for the reverse diffusion process is as follows: based on the noise prediction module, the noise in the noisy action at the current time step is predicted, and combined with the single-step attenuation coefficient and the cumulative attenuation coefficient of the time step, the noise is gradually removed until the final whole-body control quantity is obtained.

10. A whole-body robot control system based on generative reinforcement learning, characterized in that, include: The multimodal observation data acquisition module is used to acquire joint angular positions, joint velocities, joint torques, chassis pose, chassis speed, and RGBD visual images of the mobile robot in real time. A multimodal state encoder module is used to encode the multimodal observation data into a joint state representation; A generative whole-body control strategy module is used to map the joint state representation into whole-body control variables of a mobile manipulator based on a conditional diffusion model. The network training optimization module is used to establish a composite optimization objective function that includes optimization terms of the conditional diffusion model and optimization terms of the value network, and to update the parameters of each network synchronously. The strategy reasoning module is used to generate the final whole-body control quantity through the back diffusion process of the conditional diffusion model to generate noise reduction, and drive the chassis and robotic arm to perform collaborative execution.

Images