A method and system for obstacle crossing control of reconfigurable wheeled robots combining deep reinforcement learning

Abstract

This invention provides a method and system for obstacle crossing control of a reconfigurable wheeled robot, incorporating deep reinforcement learning. The method relates to the technical field of robot obstacle crossing control and includes the following steps: constructing and initializing a ground-based physical obstacle crossing simulation environment; establishing a kinematic and dynamic model of the reconfigurable wheeled robot; constructing the state space and motion space of the reconfigurable robot; introducing a multi-objective reward function; constructing a policy and value network architecture for a deep reinforcement learning PPO algorithm; and loading and configuring the optimized policy model into the controller of the reconfigurable wheeled robot in the simulation environment to achieve obstacle crossing control of the robot. This invention provides a ground-based obstacle crossing control method for reconfigurable wheeled robots that is adaptable to complex and varied terrain, possesses strong robustness, strong real-time decision-making capabilities, and effectively optimizes energy consumption.

Description

Technical Field

[0001] This invention relates to the technical field of obstacle crossing control for robots, and more particularly to a method and system for obstacle crossing control of reconfigurable wheeled robots that incorporates deep reinforcement learning. Background Technology

[0002] Reconfigurable wheeled robots, with their unique morphing mechanisms, exhibit distinctive advantages in diverse and complex environments, especially in obstacle crossing on land, where their reconfigurability allows them to adapt to different types of obstacles. However, traditional methods for controlling obstacle crossing on land face numerous challenges.

[0003] 1. Existing rule-based or pre-set strategy-based solutions, such as motion rules formulated through expert experience, are prone to obstacle crossing failure when faced with random geometric obstacles, dynamic interference, or complex obstacle crossing environments that are not pre-set due to their limited coverage of obstacle types and lack of real-time perception and autonomous decision-making capabilities. Furthermore, they require a large amount of manual intervention and parameter tuning, making it difficult to meet the autonomous operation requirements of reconfigurable wheeled robots.

[0004] 2. Control methods that rely on precise robot kinematics and dynamics models, such as inverse dynamics control or model predictive control (MPC), have control performance that is highly dependent on model accuracy. However, in actual terrain interactions, there are a large number of nonlinear factors (such as contact impact, friction changes, and unmodeled joint flexibility) that can easily lead to model mismatch, thereby affecting control accuracy and stability. This is especially true for wheeled robots with complex reconfigurable mechanisms such as folding arms and variable diameter wheels, where multibody dynamics modeling is extremely complex. Furthermore, model parameters need to be recalibrated during shape switching, which is time-consuming and labor-intensive. Failure to fully consider mechanical constraints may also lead to the risk of mechanical damage.

[0005] 3. Methods that transform obstacle crossing control problems into constrained optimization problems, such as trajectory optimization or nonlinear model predictive control (NMPC), can theoretically generate optimal strategies, but solving high-dimensional optimization problems in real time places stringent demands on the computing power of embedded controllers, easily leading to control delays and failing to meet the high real-time requirements of control in dynamic environments. These methods also often rely on prior environmental information, making it difficult to adapt to rapidly changing obstacle crossing scenarios when obstacles change randomly or environmental information is incomplete.

[0006] More importantly, the aforementioned traditional control methods typically prioritize task completion rate, control accuracy, or stability during design, failing to incorporate the mechanical energy consumption generated during the robot's multimodal motion into their core considerations. This can lead to robots employing high-energy-consuming obstacle-crossing strategies, severely limiting their endurance and operational range in the field or during extended missions. This is especially true for reconfigurable wheeled robots that rely on battery power, where energy efficiency is a critical bottleneck for achieving long-endurance, long-distance autonomous operations. Summary of the Invention

[0007] To address the aforementioned technical problems of reconfigurable wheeled robots, including poor environmental adaptability in complex terrain environments, insufficient autonomous decision-making capabilities, high dependence on accurate robot modeling, stringent real-time computing power requirements, and, most notably, a lack of effective consideration of robot mechanical energy consumption in obstacle-crossing strategy design, leading to low energy utilization efficiency, this invention provides a ground obstacle-crossing control method and system for reconfigurable wheeled robots that incorporates deep reinforcement learning. This invention offers a ground obstacle-crossing control method for reconfigurable wheeled robots that is adaptable to complex and varied terrain, possesses strong robustness, exhibits strong real-time decision-making capabilities, and effectively optimizes energy consumption.

[0008] The technical means employed in this invention are as follows:

[0009] A method for obstacle-crossing control of a reconfigurable wheeled robot combining deep reinforcement learning, characterized by the following steps:

[0010] Step 1: Build and initialize the ground physical obstacle crossing simulation environment;

[0011] Step 2: Establish the kinematic and dynamic models of the reconfigurable wheeled robot;

[0012] Step 3: Construct the state space and motion space of the reconfigurable robot;

[0013] Step 4: Introduce a multi-objective reward function; the multi-objective reward function includes: a quantified reward based on mechanical power consumption, a target approach guidance reward, and a sparse event constraint reward; the quantified reward for mechanical power consumption includes: robot movement energy consumption, folding arm mechanism energy consumption, and variable diameter wheel shape adjustment energy consumption;

[0014] Step 5: Construct the policy and value network architecture of the deep reinforcement learning PPO algorithm, initialize its network parameters, and obtain the optimized policy model through interactive reinforcement learning driven by course learning in the ground obstacle crossing simulation environment.

[0015] Step 6: Load and configure the optimized strategy model into the controller of the reconfigurable wheeled robot in the simulation environment to achieve obstacle crossing control of the robot.

[0016] Furthermore, establishing the reconfigurable robot kinematic model includes:

[0017] A kinematic model of the foldable arm is established. Based on geometric relationships, the overall width W of the robot is determined by the length of the robot body and the length and angle of the foldable arm.

[0018] +2 ;

[0019] in, Indicates the fixed width of the robot's body; This indicates the distance from the joint of the robot's folding arm to the edge of the wheel; Indicates the folding angle of the robot;

[0020] The overall height H of the robot is determined by the distance from the body to the edge of the wheels, plus the length of the foldable arm and the folding angle.

[0021] ;

[0022] Where h represents the fixed height of the robot body, Indicates the length of the folding arm;

[0023] The kinematic model of the robot with the variable diameter wheel is established as follows:

[0024] The variable diameter wheel is modeled based on the straight scissor element (SSE), and the wheel radius is calculated using the SSE unfolding angle β.

[0025] ;

[0026] Wherein, the radius of the central rotating disk is The length of the scissor bar is Number of units n=3, wheel thickness is ,and Achieve wheel diameter Dynamic adjustment.

[0027] Furthermore, establishing the dynamic model of the reconfigurable robot includes the following steps:

[0028] Establish a joint torque dynamics model: Based on the Newton-Euler method, derive the joint torque of the folding arm of the amphibious robot. The formula is:

[0029] ;

[0030] in, Indicates the normal force of a single leg. For friction, The coefficient of friction;

[0031] Constructing an energy consumption model: Mechanical energy consumption is calculated by integrating the changes in joint torque and angle, using the following formula:

[0032] ;

[0033] in, Indicates driving torque. This indicates the amount of change in the driving joint. This indicates the torque of the servo motor. This represents the constant angle change required to complete the change in the shape of the variable diameter wheel, where T is the number of training steps.

[0034] Furthermore, in step 3, the state space of the reconfigurable robot is constructed; the real-time kinematic state information of the robot body includes: the robot's position, the robot's posture, the robot's linear velocity, and the robot's angular velocity.

[0035] Kinematic state vector for: = ;

[0036] in, This represents the robot's coordinates in the global coordinate system. The Euler angles represent the robot's orientation. This represents the robot's linear velocity. This represents the robot's angular velocity;

[0037] The real-time form of the robot's reconfigurable components includes: the folding angle of the foldable arm and the wheel diameter of the variable diameter wheel;

[0038] The morphological vector of the robot's reconfigurable component is = ;

[0039] The state space vector representation of a reconfigurable robot is constructed as follows: ;in, It indicates local terrain features.

[0040] Furthermore, in step 3, the robot's motion space includes: the robot's basic movement motions and the deformation motions of the robot's reconfigurable components;

[0041] The robot's basic movement motion is represented by its forward linear velocity and turning angular velocity, namely:

[0042] ;

[0043] The deformation motion of the robot's reconfigurable components is represented by the rate of change of the folding angle of the foldable arm and the rate of change of the shape adjustment parameters of the variable-diameter wheel, namely:

[0044] ;

[0045] The robot's motion space is then represented as: .

[0046] Furthermore, in step 4, the total reward of the multi-objective reward function at each time step t... Includes: sparse reward components and dense reward components;

[0047] ;

[0048] Wherein, the sparse reward component Defined as:

[0049] ;

[0050] The dense reward component The overall structure is as follows:

[0051] ;

[0052] Among them, dense reward component The numerical values ​​and the calculated total mechanical energy consumption of the robot during obstacle crossing. Proportional, that is = - ;

[0053] The for:

[0054] = + ;

[0055] in, Indicates the energy consumption of the folding arm extension:

[0056] = ;

[0057] in, Indicates the torque of the folding arm joint. This indicates the change in the folding angle of the folding arm;

[0058] Indicates wheel drive energy consumption:

[0059] = ;

[0060] in, Indicates the torque driving the joint. This indicates the amount of change in the angle of the driven joint;

[0061] Energy consumption due to changes in wheel diameter:

[0062] = ;

[0063] in, This indicates the torque of the servo motor. This represents the constant angle value of rotation required to complete the change of the variable diameter wheel's shape;

[0064] The set of auxiliary and constraint terms consists of multiple sub-items that are weighted and summed by their respective weights:

[0065] = + + + ;

[0066] in, = This indicates a reward for approaching the target. This indicates the distance between the current location and the target point. 3 indicates the slope adjustment parameter; This indicates a penalty for smoothness of motion, used to suppress violent movements; = ; This represents the continuous motion vectors executed by the robot at the current time step, including basic movement actions and reconfigurable component actions; This represents the square of the L2 norm of the action vector. Indicates the penalty coefficient;

[0067] Indicates a penalty for the aircraft leaning backward:

[0068] = ;

[0069] in, This indicates the robot's real-time pitch angle. This indicates the preset safe pitch angle threshold. Indicates the penalty coefficient;

[0070] Indicates the penalty for limiting the angle of the folding joint:

[0071] = - (0, ));

[0072] and These are the maximum and minimum preset physical angle limits of the folding joint, respectively. Indicates the penalty coefficient; This represents the weight coefficient corresponding to each penalty item. Each penalty item is multiplied by its corresponding weight when calculating the total reward.

[0073] The present invention also includes a reconfigurable wheeled robot obstacle-crossing control system incorporating deep reinforcement learning, comprising:

[0074] Environment Interaction and Difficulty Adaptation Module: Used to build and manage the ground physical obstacle crossing simulation environment, initialize the robot model and obstacle layout, provide real-time environmental observation and physical feedback, and dynamically adjust the difficulty parameters according to the training process;

[0075] Kinematics and Dynamics Modeling Module: Used to build kinematic and dynamic models of reconfigurable wheeled robots;

[0076] State and Action Space Construction Module: Used to acquire and process the kinematic state information of the robot body, the morphological information of reconfigurable parts, and the local terrain feature information of the environment, so as to construct the state space and action space.

[0077] Reward function calculation and evaluation module: used to introduce and calculate multi-objective reward functions, which include quantitative rewards based on mechanical energy consumption, target approach rewards, and constraint penalty terms;

[0078] Strategy learning and decision generation module: used to implement the deep reinforcement learning algorithm PPO, construct the policy and value network, and perform course learning-driven interactive training based on the state space, action space and reward function, thereby generating low-level joint space control signals for the robot.

[0079] Control strategy deployment and execution module: used to deploy the control strategy to the simulation platform and convert the low-level joint space control signals into physical instructions that the robot can execute, so as to drive it to achieve energy-efficient autonomous obstacle crossing control in complex environments.

[0080] Compared with the prior art, the present invention has the following advantages:

[0081] This invention, through an innovative multi-objective reward function, particularly the quantification and penalty of key mechanical energy consumption, enables a deep reinforcement learning agent to learn energy-aware obstacle-crossing strategies. This significantly reduces the total mechanical energy consumption of reconfigurable wheeled robots during complex obstacle-crossing processes, greatly improving the robot's endurance and operational range. Simultaneously, this invention fully leverages the advantages of deep reinforcement learning, combining a high-fidelity physical simulation environment and comprehensive state and action space design. This allows the robot to autonomously learn strategies to adapt to complex and varied terrain through interaction with the environment, significantly enhancing its environmental adaptability, autonomous decision-making ability, and robustness, overcoming the excessive reliance on precise modeling in traditional control methods. Furthermore, this invention employs a curriculum-driven training strategy, gradually increasing the environmental difficulty and the weight of energy consumption penalties in the reward function. This effectively guides the deep reinforcement learning process, accelerates policy convergence, and makes the learned strategies more stable, safe, and efficient in complex scenarios. By using a deep reinforcement learning framework to achieve end-to-end control strategy learning, this invention simplifies control strategy design, avoids complex and difficult-to-derive explicit control law design, and enables high-dimensional collaborative control of multi-dimensional motion and deformation of reconfigurable wheeled robots. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 The system architecture diagram of the reconfigurable wheeled robot ground obstacle crossing control method based on energy-efficient deep reinforcement learning provided by the present invention is shown.

[0084] Figure 2 This is a diagram showing the reconfigurable wheeled robot preparing to cross the first narrow passage in the energy-efficient deep reinforcement learning-based ground obstacle crossing control method for reconfigurable wheeled robots of this invention.

[0085] Figure 3 This diagram illustrates the process of a reconfigurable wheeled robot traversing the first narrow passage in the energy-efficient deep reinforcement learning-based ground obstacle-crossing control method for reconfigurable wheeled robots of this invention.

[0086] Figure 4 This is a diagram showing the reconfigurable wheeled robot preparing to cross a second narrow passage in the energy-efficient deep reinforcement learning-based ground obstacle crossing control method for reconfigurable wheeled robots of this invention.

[0087] Figure 5This diagram illustrates the process of a reconfigurable wheeled robot traversing a second narrow passage in the energy-efficient deep reinforcement learning-based ground obstacle-crossing control method for reconfigurable wheeled robots of this invention.

[0088] Figure 6 This is a diagram showing the reconfigurable wheeled robot traversing a second narrow passage in the ground obstacle-crossing control method for reconfigurable wheeled robots based on energy-efficient deep reinforcement learning, as described in this invention.

[0089] Figure 7 This invention provides a diagram for the reconfigurable wheeled robot to cross a step obstacle in its ground obstacle crossing control method for reconfigurable wheeled robots based on energy-efficient deep reinforcement learning.

[0090] Figure 8 This diagram illustrates the process of a reconfigurable wheeled robot crossing a step obstacle in the ground obstacle crossing control method for reconfigurable wheeled robots based on energy-efficient deep reinforcement learning, as described in this invention.

[0091] Figure 9 This is a diagram illustrating the reconfigurable wheeled robot's successful crossing of a step obstacle in the energy-efficient deep reinforcement learning-based ground obstacle crossing control method for reconfigurable wheeled robots of this invention.

[0092] Figure 10 This is the average reward map for training the reinforcement learning algorithm in this invention. Detailed Implementation

[0093] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0094] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0095] like Figure 1-10As shown, this invention provides a method for obstacle crossing control of a reconfigurable wheeled robot combining deep reinforcement learning, comprising the following steps:

[0096] Step 1: Construct and initialize a ground-based physical obstacle-crossing simulation environment. This environment includes stepped obstacles and two consecutive narrow passages, and also features a reconfigurable wheeled robot model with diverse configurations. This robot model has foldable robotic arms and variable-diameter wheels, and can automatically adjust its shape according to the environment.

[0097] Step 2: Establish the kinematic and dynamic models of the reconfigurable wheeled robot; establishing the kinematic model of the reconfigurable robot includes:

[0098] A kinematic model of the foldable arm is established. Based on geometric relationships, the overall width W of the robot is determined by the length of the robot body and the length and angle of the foldable arm.

[0099] +2 ;

[0100] in, Indicates the fixed width of the robot's body; This indicates the distance from the joint of the robot's folding arm to the edge of the wheel; Indicates the folding angle of the robot;

[0101] The overall height H of the robot is determined by the distance from the body to the edge of the wheels, plus the length of the foldable arm and the folding angle.

[0102] ;

[0103] Where h represents the fixed height of the robot body, Indicates the length of the folding arm;

[0104] The kinematic model of the robot with the variable diameter wheel is established as follows:

[0105] The variable diameter wheel is modeled based on the straight scissor element (SSE), and the wheel radius is calculated using the SSE unfolding angle β.

[0106] ;

[0107] Wherein, the radius of the central rotating disk is The length of the scissor bar is Number of units n=3, wheel thickness is ,and Achieve wheel diameter Dynamic adjustment.

[0108] The process of establishing a dynamic model of the reconfigurable robot includes the following steps:

[0109] Establish a joint torque dynamics model: Based on the Newton-Euler method, derive the joint torque of the folding arm of the amphibious robot. The formula is:

[0110] ;

[0111] in, Indicates the normal force of a single leg. For friction, The coefficient of friction;

[0112] Constructing an energy consumption model: Mechanical energy consumption is calculated by integrating the changes in joint torque and angle, using the following formula:

[0113] ;

[0114] in, Indicates driving torque. This indicates the amount of change in the driving joint. This indicates the torque of the servo motor. This represents the constant angle change required to complete the change in the shape of the variable diameter wheel, where T is the number of training steps.

[0115] Step 3: Construct the state space and motion space of the reconfigurable robot. In Step 3, the state space of the reconfigurable robot is constructed. (State space is the set of all observable and describable states of the entire robot at a given moment, including kinematic states, component morphological states, environmental characteristics, etc.; kinematic state is only a part of the state space, mainly describing the overall kinematic quantities of the robot (position, attitude, velocity, angular velocity).) The real-time kinematic state information of the robot body includes: the robot's position, robot's attitude, robot's linear velocity, and robot's angular velocity.

[0116] Kinematic state vector for: = ;

[0117] in, This represents the robot's coordinates in the global coordinate system. The Euler angles represent the robot's orientation. This represents the robot's linear velocity. This represents the robot's angular velocity;

[0118] The real-time configuration of the robot's reconfigurable components includes: the folding angle of the foldable arm and the wheel diameter of the variable-diameter wheels; the folding angle of the foldable arm is obtained through a C# script, which is attached to the robot. First, define `floatcurrentArmAngle;` / / current folding arm angle.

[0119] float lastArmAngle; / / Angle of the folding arm at the previous moment

[0120] OnEpisodeBegin() is a core method in the Unity ML-Agents toolkit. Its main function is to reset the environment to its initial state at the beginning of each training round (episode), preparing for a new round of training or testing.

[0121] You can think of it as a "Start New Game" or "Replay This Level" button in a video game. This method is automatically called whenever the agent fails, succeeds, or reaches a certain time step, ending one round and starting the next. (See `public override void OnEpisodeBegin()`).

[0122] Add `currentArmAngle = 90.0f;` to the method.

[0123] Initialize the folding arm angle and update lastArmAngle = currentArmAngle;

[0124] Adding `currentArmAngle = leftArm.transform.localEulerAngles.z;` inside the `void UpdateStateAndCalculateEnergy(){}` method will retrieve the real-time angle of the foldable arm.

[0125] The same principle applies to variable diameter wheels; by writing relevant code in a script and then attaching the script to the moving robot, the real-time status can be obtained.

[0126] The morphological vector of the robot's reconfigurable component is = ;

[0127] Local terrain features are collected by sensors, which can acquire the height difference, slope, width and surface roughness of obstacles within a 5-meter range in front;

[0128] The state space vector representation of a reconfigurable robot is constructed as follows: ;in, It indicates local terrain features.

[0129] In step 3, the robot's motion space includes: the robot's basic movement motions and the deformation motions of the robot's reconfigurable components;

[0130] The robot's basic movement motion is represented by its forward linear velocity and turning angular velocity, namely:

[0131] ;

[0132] The deformation motion of the robot's reconfigurable components is represented by the rate of change of the folding angle of the foldable arm and the rate of change of the shape adjustment parameters of the variable-diameter wheel, namely:

[0133] ;

[0134] The robot's motion space is then represented as: .

[0135] Step 4: Introduce a multi-objective reward function; the multi-objective reward function includes: a quantified reward based on mechanical power consumption, a goal-approaching guidance reward, and a sparse event constraint reward; the quantified reward of mechanical power consumption includes: robot movement energy consumption, folding arm mechanism energy consumption, and variable diameter wheel shape adjustment energy consumption; in Step 4, the total reward of the multi-objective reward function at each time step t... Includes: sparse reward components and dense reward components;

[0136] ;

[0137] Wherein, the sparse reward component Defined as:

[0138] ;

[0139] The dense reward component The overall structure is as follows:

[0140] ;

[0141] Among them, dense reward component The numerical values ​​and the calculated total mechanical energy consumption of the robot during obstacle crossing. Proportional, that is = - ;

[0142] The for:

[0143] = + ;

[0144] in, Indicates the energy consumption of the folding arm extension:

[0145] = ;

[0146] in, Indicates the torque of the folding arm joint. This indicates the change in the folding angle of the folding arm;

[0147] Indicates wheel drive energy consumption:

[0148] = ;

[0149] in, Indicates the torque driving the joint. This indicates the amount of change in the angle of the driven joint;

[0150] Energy consumption due to changes in wheel diameter:

[0151] = ;

[0152] in, This indicates the torque of the servo motor. This represents the constant angle value of rotation required to complete the change of the variable diameter wheel's shape;

[0153] The set of auxiliary and constraint terms includes multiple sub-terms that are weighted and summed by their respective weights (the mechanical work penalty term is the main term and the main innovation of this invention patent; auxiliary terms, such as the target approach reward, are used to assist the mechanical work penalty term in guiding the robot toward the target when training the robot to overcome obstacles; constraint terms, such as the body tilting penalty, are used to reduce tilting and falling over during the training process of the robot to overcome obstacles, and are a penalty term that restricts joint movement).

[0154] = + + + ;

[0155] in, = This indicates a reward for approaching the target. This indicates the distance between the current location and the target point. 3 indicates the slope adjustment parameter; This indicates a penalty for smoothness of motion, used to suppress violent movements; = ; This represents the continuous motion vectors executed by the robot at the current time step, including basic movement actions and reconfigurable component actions; This represents the square of the L2 norm of the action vector. This represents the penalty coefficient; in reinforcement learning, the action smoothness penalty is used to suppress drastic changes in robot motion between consecutive time steps. If the difference in motion between two time steps is large, it indicates that the control signal is "jittering," leading to high motor energy consumption, excessive mechanical wear, and even instability. Smooth changes result in more stable and energy-efficient robot operation. Ensuring the formula is smooth and differentiable facilitates gradient optimization. In summary, this formula provides a unified measure of the amount of change across all dimensions of robot motion.

[0156] Indicates a penalty for the aircraft leaning backward:

[0157] = ;

[0158] in, This indicates the robot's real-time pitch angle. This indicates the preset safe pitch angle threshold. Indicates the penalty coefficient;

[0159] Indicates the penalty for limiting the angle of the folding joint:

[0160] = - (0, ));

[0161] and These are the maximum and minimum preset physical angle limits of the folding joint, respectively. Indicates the penalty coefficient; This represents the weight coefficient corresponding to each penalty item. Each penalty item is multiplied by its corresponding weight when calculating the total reward.

[0162] Step 5: Construct the policy and value network architecture of the deep reinforcement learning PPO algorithm, initialize its network parameters, and obtain an optimized policy model through interactive reinforcement learning driven by a ground obstacle crossing simulation environment. This architecture is designed to efficiently process and fuse data from different types. The input layer of this network architecture is used to accept reconfigurable wheeled robot proprioceptive data and environmental perception data to construct a comprehensive observation space.

[0163] Input layer design: Proprioceptive perception data includes the robot's position, posture, linear velocity, angular velocity and other kinematic states; environmental perception data includes environmental information from the robot's sensors. This data is acquired through cameras and processed through data fusion technology to form multi-dimensional feature vectors.

[0164] Data fusion: The multimodal fusion network architecture processes different types of input data through multiple channels. Specifically, proprioceptive perception data and environmental perception data are initially processed through independent neural network channels, and then fused into a unified input vector, which is then passed to subsequent network layers.

[0165] Output layer design: The output layer of the architecture is used to generate low-level joint space control signals for the robot. The specific control signals include the following three parts:

[0166] Hub motor torque command: Through the network output, the torque command required for each hub motor is generated. This command controls the robot's wheel speed and power output, enabling the robot to move smoothly in complex terrain.

[0167] Control commands for each joint of the folding arm: Based on the robot's posture and terrain information, control commands are output for each joint of the folding arm. These control commands adjust the angle of the folding arm according to real-time conditions to achieve robot posture adjustment and improve obstacle-crossing ability.

[0168] Variable diameter wheel adjustment command: The network generates a variable diameter wheel adjustment command based on environmental perception data to adapt to different terrains and obstacles. This command adjusts the wheel diameter in real time to optimize obstacle-crossing efficiency and energy consumption.

[0169] Step 6: Load and configure the optimized strategy model into the controller of the reconfigurable wheeled robot in the simulation environment to achieve obstacle crossing control of the robot.

[0170] The present invention also includes a reconfigurable wheeled robot obstacle-crossing control system incorporating deep reinforcement learning, comprising:

[0171] Environment Interaction and Difficulty Adaptation Module: Used to build and manage the ground physical obstacle crossing simulation environment, initialize the robot model and obstacle layout, provide real-time environmental observation and physical feedback, and dynamically adjust the difficulty parameters according to the training process;

[0172] Kinematics and Dynamics Modeling Module: Used to build kinematic and dynamic models of reconfigurable wheeled robots;

[0173] State and Action Space Construction Module: Used to acquire and process the kinematic state information of the robot body, the morphological information of reconfigurable parts, and the local terrain feature information of the environment, so as to construct the state space and action space.

[0174] Reward function calculation and evaluation module: used to introduce and calculate multi-objective reward functions, which include quantitative rewards based on mechanical energy consumption, target approach rewards, and constraint penalty terms;

[0175] Strategy learning and decision generation module: used to implement the deep reinforcement learning algorithm PPO, construct the policy and value network, and perform course learning-driven interactive training based on the state space, action space and reward function, thereby generating low-level joint space control signals for the robot.

[0176] Control strategy deployment and execution module: used to deploy the control strategy to the simulation platform and convert the low-level joint space control signals into physical instructions that the robot can execute, so as to drive it to achieve energy-efficient autonomous obstacle crossing control in complex environments.

[0177] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for obstacle-crossing control of a reconfigurable wheeled robot combining deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Build and initialize the ground physical obstacle crossing simulation environment; Step 2: Establish the kinematic and dynamic models of the reconfigurable wheeled robot; Step 3: Construct the state space and motion space of the reconfigurable robot; Step 4: Introduce a multi-objective reward function; The multi-objective reward function includes: a quantified reward based on mechanical power consumption, a target approach guidance reward, and a sparse event constraint reward; the quantified reward of mechanical power consumption includes: robot movement energy consumption, folding arm mechanism energy consumption, and variable diameter wheel shape adjustment energy consumption; in step 4, the total reward of the multi-objective reward function at each time step t... Includes: sparse reward components and dense reward components; ； Wherein, the sparse reward component Defined as: ； The dense reward component The overall structure is as follows: ； Among them, dense reward component The numerical values ​​and the calculated total mechanical energy consumption of the robot during obstacle crossing. Proportional, that is = - ; The for: = + ； in, Indicates the energy consumption of the folding arm extension: = ； in, Indicates the torque of the folding arm joint. This indicates the change in the folding angle of the folding arm; Indicates wheel drive energy consumption: = ； in, Indicates the torque driving the joint. This indicates the amount of change in the angle of the driven joint; Energy consumption due to changes in wheel diameter: = ； in, This indicates the torque of the servo motor. This represents the constant angle value of rotation required to complete the change of the variable diameter wheel's shape; The set of auxiliary and constraint terms consists of multiple sub-items that are weighted and summed by their respective weights: = + + + ； in, = This indicates a reward for approaching the target. This indicates the distance between the current location and the target point. 3 indicates the slope adjustment parameter; This indicates a penalty for smoothness of motion, used to suppress violent movements; = ; This represents the continuous motion vectors executed by the robot at the current time step, including basic movement actions and reconfigurable component actions; This represents the square of the L2 norm of the action vector. Indicates the penalty coefficient; Indicates a penalty for the aircraft leaning backward: = ； in, This indicates the robot's real-time pitch angle. This indicates the preset safe pitch angle threshold. Indicates the penalty coefficient; Indicates the penalty for limiting the angle of the folding joint: = - (0， ))； and These are the maximum and minimum preset physical angle limits of the folding joint, respectively. Indicates the penalty coefficient; This represents the weight coefficient corresponding to each penalty item. Each penalty item is multiplied by its corresponding weight when calculating the total reward. Step 5: Construct the policy and value network architecture of the deep reinforcement learning PPO algorithm, initialize its network parameters, and obtain the optimized policy model through interactive reinforcement learning driven by course learning in the ground obstacle crossing simulation environment. Step 6: Load and configure the optimized strategy model into the controller of the reconfigurable wheeled robot in the simulation environment to achieve obstacle crossing control of the robot.

2. The obstacle-crossing control method for a reconfigurable wheeled robot combining deep reinforcement learning according to claim 1, characterized in that, Establishing a reconfigurable robot kinematic model includes: Establish a kinematic model of the foldable arm. Based on geometric relationships, the overall width W of the robot is determined by the length of the body and the length and angle of the foldable arm. +2 ； in, Indicates the fixed width of the robot's body; This indicates the distance from the joint of the robot's folding arm to the edge of the wheel; Indicates the folding angle of the robot; The overall height H of the robot is determined by the distance from the body to the edge of the wheels, plus the length of the foldable arm and the folding angle. ； Where h represents the fixed height of the robot body, Indicates the length of the folding arm; The kinematic model of the robot with the variable diameter wheel is established as follows: The variable diameter wheel is modeled based on the straight scissor element (SSE), and the wheel radius is calculated using the SSE unfolding angle β. ； Wherein, the radius of the central rotating disk is The length of the scissor bar is Number of units n=3, wheel thickness is ,and Achieve wheel diameter Dynamic adjustment.

3. The obstacle-crossing control method for a reconfigurable wheeled robot combining deep reinforcement learning according to claim 1, characterized in that, Establishing a dynamic model of the reconfigurable robot includes the following steps: Establish a joint torque dynamics model: Based on the Newton-Euler method, derive the joint torque of the folding arm of the amphibious robot. The formula is: ； in, Indicates the normal force of a single leg. For friction, The coefficient of friction; Constructing an energy consumption model: Mechanical energy consumption is calculated by integrating the changes in joint torque and angle, using the following formula: ； in, Indicates driving torque. This indicates the amount of change in the driving joint. This indicates the torque of the servo motor. This represents the constant angle change required to complete the change in the shape of the variable diameter wheel, where T is the number of training steps.

4. The obstacle-crossing control method for a reconfigurable wheeled robot combining deep reinforcement learning according to claim 1, characterized in that, In step 3, the state space of the reconfigurable robot is constructed; the real-time kinematic state information of the robot body includes: the robot's position, the robot's posture, the robot's linear velocity, and the robot's angular velocity. Kinematic state vector for: = ; in, This represents the robot's coordinates in the global coordinate system. The Euler angles represent the robot's orientation. This represents the robot's linear velocity. This represents the robot's angular velocity; The real-time form of the robot's reconfigurable components includes: the folding angle of the foldable arm and the wheel diameter of the variable diameter wheel; The morphological vector of the robot's reconfigurable component is = ; The state space vector representation of a reconfigurable robot is constructed as follows: ;in, It indicates local terrain features.

5. The obstacle-crossing control method for a reconfigurable wheeled robot combining deep reinforcement learning according to claim 1, characterized in that, In step 3, the robot's motion space includes: the robot's basic movement motions and the deformation motions of the robot's reconfigurable components; The robot's basic movement motion is represented by its forward linear velocity and turning angular velocity, namely: ； The deformation motion of the robot's reconfigurable components is represented by the rate of change of the folding angle of the foldable arm and the rate of change of the shape adjustment parameters of the variable-diameter wheel, namely: ； The robot's motion space is then represented as: .

6. A reconfigurable wheeled robot obstacle-crossing control system incorporating deep reinforcement learning, employing the method described in any one of claims 1-5, characterized in that, include: Environment Interaction and Difficulty Adaptation Module: Used to build and manage the ground physical obstacle crossing simulation environment, initialize the robot model and obstacle layout, provide real-time environmental observation and physical feedback, and dynamically adjust the difficulty parameters according to the training process; Kinematics and Dynamics Modeling Module: Used to build kinematic and dynamic models of reconfigurable wheeled robots; State and Action Space Construction Module: Used to acquire and process the kinematic state information of the robot body, the morphological information of reconfigurable parts, and the local terrain feature information of the environment, so as to construct the state space and action space. Reward function calculation and evaluation module: used to introduce and calculate multi-objective reward functions, which include quantitative rewards based on mechanical energy consumption, target approach rewards, and constraint penalty terms; Strategy learning and decision generation module: used to implement the deep reinforcement learning algorithm PPO, construct the policy and value network, and perform course learning-driven interactive training based on the state space, action space and reward function, thereby generating low-level joint space control signals for the robot. Control strategy deployment and execution module: used to deploy the control strategy to the simulation platform and convert the low-level joint space control signals into physical instructions that the robot can execute, so as to drive it to achieve energy-efficient autonomous obstacle crossing control in complex environments.

Images