A method and system for omnidirectional motion control of a salamander-like robot based on reinforcement learning

Abstract

A method and system for omnidirectional motion control of a salamander-inspired robot based on reinforcement learning, belonging to the field of electromechanical system control. The method includes: acquiring robot state information and motion commands; generating periodic phase variables through a phase integration module and outputting target actions for each joint by a trajectory generator; designing a reward function based on behavioral constraints to constrain motion directionality and posture stability; utilizing the morphological symmetry of the salamander-inspired robot's body structure to symmetrically enhance state-action samples to expand training data; and employing a reinforcement learning algorithm to iteratively update the control strategy based on reward feedback and symmetrically enhanced data to obtain a stable and efficient omnidirectional motion control model. This invention can achieve reinforcement learning-driven omnidirectional motion strategy generation without the need for a pre-set gait model, possessing advantages such as natural motion patterns, high control accuracy, and strong adaptability, and can be widely applied in fields such as complex terrain detection and intelligent inspection.

Description

Technical Field

[0001] This application belongs to the field of electromechanical system control, and in particular relates to an omnidirectional motion control method and system for a salamander-like robot based on reinforcement learning. Background Technology

[0002] Salamander-inspired quadruped robots are a type of biomimetic robot that mimics the body structure of amphibians such as salamanders. They are characterized by a low center of gravity, high stability, and a variety of movement patterns, enabling efficient movement in complex and varied terrain environments. Therefore, these robots have broad application prospects in disaster relief, environmental monitoring, and field exploration.

[0003] Existing methods for motion control of salamander-inspired robots primarily rely on manually designed control strategies. One type of method achieves periodic movement through predefined gait patterns; another, inspired by the Central Pattern Generator (CPG), achieves smooth switching between different motion modes by adjusting control parameters. While these methods can generate stable rhythmic movements, they still heavily depend on expert experience, and their structural constraints limit the diversity of motion behavior, making it difficult to achieve highly flexible and adaptable omnidirectional motion control.

[0004] In recent years, with the development of intelligent control technologies such as deep reinforcement learning, researchers have begun to explore the automatic acquisition of motion control strategies through learning-driven approaches. Some studies directly output joint angles through end-to-end neural networks and use reference trajectories or foot contact signals as learning targets to achieve various gait patterns. However, these methods often rely on pre-collected reference motion data, and for bio-inspired robots such as salamanders, the lack of available high-quality motion datasets limits the generalization ability and scalability of such methods.

[0005] Another type of no-reference learning method attempts to discover diverse motion patterns through unsupervised skills or based on an objective function that maximizes entropy. However, the generated motions often lack physical rationality, exhibiting chaotic movements or poor coordination, making them difficult to apply directly to real robotic systems.

[0006] Furthermore, in the omnidirectional motion control of salamander-inspired robots, the exploration of strategies in different directions is often unbalanced, easily getting trapped in local optima in some directions, leading to asymmetric motion performance. Recent studies have shown that utilizing the morphological symmetry of robots as physical priors can effectively improve learning efficiency and motion consistency, thereby obtaining more natural motion behavior. However, existing research does not fully utilize the symmetry patterns in the motion of salamander-inspired robots and lacks a systematic control framework for omnidirectional motion.

[0007] Therefore, how to enable salamander-like quadruped robots to autonomously learn diverse, stable, and symmetrical omnidirectional gaits and achieve balanced state-motion space exploration without the need for reference trajectories has become a pressing technical problem in this field. Summary of the Invention

[0008] To address the problems of monotonous gait patterns, asymmetrical behavior, and low learning efficiency in existing salamander-inspired robot motion control, this application provides a reinforcement learning-based omnidirectional motion control method and system for salamander-inspired robots. This method enables the salamander-inspired robot to autonomously learn diverse, stable, and symmetrical omnidirectional gaits without a reference trajectory, thereby improving the robot's autonomous movement capability and control robustness in complex environments.

[0009] In a first aspect, this application provides a reinforcement learning-based omnidirectional motion control method for a salamander-inspired robot, comprising the following steps:

[0010] Acquire the state information and motion commands of the salamander-like robot; construct a state vector based on the state information and task commands, and take the task-level commands, body posture changes, phase encoding of limbs and torso, and phase coverage statistics of each leg as inputs;

[0011] Based on the state vector, the target servo position command is generated using the phase integral and trajectory generation module, thereby driving the robot's joints to achieve periodic motion.

[0012] Based on a reward function that includes motion constraints and behavior guidance, the actions generated by the reinforcement learning agent are evaluated and optimized.

[0013] A data augmentation strategy utilizing morphological symmetry is proposed, which generates equivalent samples through symmetric group transformation to improve the generalization and symmetry constraints of the strategy.

[0014] The policy network and value network are iteratively updated using reinforcement learning algorithms to obtain a stable and efficient omnidirectional motion control policy.

[0015] Furthermore, the state vector Defined as:

[0016]

[0017] in These represent the desired forward and lateral velocities, respectively. Indicates the desired angular velocity; These represent the translation and rotation errors between the desired pose and the actual pose, respectively. Indicates the first Phase variables of a joint or body segment; Used to implement periodic encoding of phase; Indicates the first Phase coverage of a leg within a complete cycle is used to measure the periodic integrity of the gait.

[0018] Furthermore, the phase integration and trajectory generation module generates the target servo angle and motion vector according to the following rules:

[0019] First, the reinforcement learning policy network outputs action vectors at each time step.

[0020]

[0021] in: These represent the phase integral rates of the four legs: left front, left rear, right front, and right rear, respectively. This represents the phase integral rate of the torso segment;

[0022] Subsequently, the phase variable is updated according to the following integration rule:

[0023]

[0024] in, To control the time step, Phase velocity represents the rate of change of phase per unit time. All outputs are from a reinforcement learning policy network, used to adjust the phase change rate of each body part, thereby achieving flexible omnidirectional gait adjustment;

[0025] Based on the phase variable, the trajectory generator calculates the target angle of the servo motor:

[0026]

[0027] in, It is the amplitude constant; This refers to the servo motor offset angle; Indicates the first One leg The target angle of each servo motor; through the above phase integration and trajectory generation process, the salamander-inspired robot can achieve continuous, coordinated and periodically stable omnidirectional gait control.

[0028] Furthermore, the reward function Defined as a weighted sum of multiple sub-items:

[0029]

[0030] The sub-items are as follows: Forward motion reward: encourages the robot to generate effective displacement along the target direction; Lateral penalty: suppresses invalid lateral motion perpendicular to the main motion direction; Turning reward: provides a positive reward when turning in the desired angular velocity direction; Targetless turning penalty: restricts the robot from generating random rotations or unnecessary posture perturbations; Targetless translation penalty: prevents the robot from continuing to move when stationary or with a fixed target; Phase coverage reward: encourages the leg phase to be evenly distributed throughout the complete cycle to maintain gait coordination and stability; and the weight parameters for each reward or penalty item.

[0031] Furthermore, the morphological symmetry data augmentation is based on the left-right symmetry characteristics of the robot's body structure, through symmetry groups. Acting on the state space With action space Generate equivalent symmetric samples ,in, Indicates identity transformation, This represents a symmetrical reflection transformation. This represents a left-right symmetrical reflection transformation. This represents a composite transformation of two symmetric operations, which can improve the consistency and generalization ability of the policy in the symmetric action space without changing the semantics of the task.

[0032] Secondly, this application provides an omnidirectional motion control system for a salamander-inspired robot based on reinforcement learning, comprising:

[0033] 1. Status Acquisition Module: Configured to acquire robot status information and external motion commands;

[0034] 2. Phase Integrator and Trajectory Generation Module: Configured to generate periodic trajectories through phase integration and output motion signals;

[0035] 3. Reward Feedback Module: Configured to calculate behavioral constraint rewards and provide policy optimization signals;

[0036] 4. Symmetry Enhancement Module: Configured to enhance the morphological symmetry of training samples;

[0037] 5. Policy Update Module: Configured to iteratively update policy parameters based on reinforcement learning algorithms.

[0038] Thirdly, this application also provides an electronic device, including:

[0039] One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the stored one or more computer programs to cause the electronic device to perform the method described in the first aspect above.

[0040] Fourthly, this application also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0041] Fifthly, this application also provides a computer program product, including a computer program that, when run on one or more processors, implements the method of any one of the first aspects mentioned above.

[0042] Compared with the prior art, the beneficial effects of this application are:

[0043] The reinforcement learning framework proposed in this invention can autonomously learn diverse gait patterns without relying on any reference trajectory or external dataset. By introducing a phase variable driving mechanism, it achieves coordinated control of the spine and limbs of a salamander-inspired robot, significantly improving motion stability and flexibility. The designed phase coverage reward encourages the robot to explore a wider gait space, effectively avoiding the occurrence of static or repetitive gaits. Combined with a morphological symmetry enhancement strategy, the learning process maintains structural symmetry in the state-action space, significantly improving motion consistency and the generalization ability of the control strategy. The system has a clear structure and reasonable module division, and can be directly applied on simulation and real hardware platforms, demonstrating good scalability and practical value. Attached Figure Description

[0044] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0045] Figure 1 This is a schematic diagram of an omnidirectional motion control method for a salamander-inspired robot based on reinforcement learning.

[0046] Figure 2 This is an experimental platform diagram of an omnidirectional motion control method for a salamander-inspired robot based on reinforcement learning. Detailed Implementation

[0047] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the following detailed description is exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0048] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0049] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other. Example 1

[0050] like Figure 1 As shown, this embodiment provides an omnidirectional motion control method for a salamander-like robot based on reinforcement learning; including:

[0051] S101: Acquire robot status information and motion commands;

[0052] A state vector is constructed based on the aforementioned state information and task instructions. The state vector is denoted as:

[0053]

[0054] This is used to comprehensively describe the robot's motion state at time t, including task-level instructions, changes in body posture, phase information of limbs and torso, and phase coverage statistics of each leg.

[0055] in These represent the desired forward and lateral velocities, respectively. Indicates the desired angular velocity; These represent the translation and rotation errors between the desired pose and the actual pose, respectively. Indicates the first Phase variables of a joint or body segment; Used to implement periodic encoding of phase; Indicates the first Phase coverage of a leg within a complete cycle is used to measure the periodic integrity of the gait.

[0056] As an optional implementation, the state information can be acquired through a motion capture system to obtain the position and attitude angle information of the salamander-like robot in the world coordinate system;

[0057] As another implementation, the state information can also be collected in real time by the robot's inertial measurement unit (IMU), joint encoder, and vision sensor.

[0058] Regarding motion commands, let the command vector be:

[0059]

[0060] Among them, translation command Defined as a two-dimensional vector in the robot's body coordinate system:

[0061]

[0062] in, Indicates the desired direction of horizontal movement (yaw angle).

[0063] When no translation command is given, define .

[0064] Rotation command These correspond to three cases: clockwise rotation, no rotation, and counterclockwise rotation, respectively.

[0065] During training and testing, translation and rotation commands were combined into seven different motion patterns to enrich the data distribution for policy learning.

[0066] During robot movement, due to the periodicity of gait, its linear velocity and angular velocity fluctuate significantly over time, making direct tracking unstable.

[0067] Therefore, within a given time interval The robot's pose changes are recorded internally to obtain steady-state motion information.

[0068] At each time step, the robot's pose in the world coordinate system is recorded in a length of [length missing]. In the stack.

[0069] By transforming the robot's pose from the most recent moment to the body coordinate system of the earliest moment, the changes in translation and rotation of the robot within that time interval can be calculated:

[0070] ,

[0071] To stabilize the learning process of phase information, the oscillation phases of the four legs and spinal segments are encoded using sine and cosine forms:

[0072] ,

[0073] In the formula, LF, LH, RF, RH, and S represent: left foreleg, left hindleg, right foreleg, right hindleg, and spine, respectively.

[0074] This representation method can smoothly vary along the unit circle, avoiding... Discontinuity caused by jumps.

[0075] To further quantify the movement performance of each leg during the gait cycle, a phase coverage index was calculated.

[0076] Specifically, the phase sequence is arranged in ascending order, the circular difference between adjacent phases and the closure difference between the beginning and end are calculated, and the largest gap is taken. The length of the uncovered area is .

[0077] Therefore, the phase coverage rate is:

[0078]

[0079] in, Indicates the first The coverage ratio of each leg.

[0080] This metric reflects the extent to which each leg traverses the phase of a robot's gait in a complete gait cycle, and helps to assess gait balance.

[0081] In summary, the robot state vector Composed of task instructions, attitude changes, phase encoding, and coverage, it can describe the robot's current motion state and provide high-dimensional state input for subsequent phase integration and trajectory generation processes.

[0082] The aforementioned state information can be collected collaboratively by the motion capture system, IMU sensors, and encoders, forming the basis for the fusion of global and local state perception of the robot.

[0083] S102: Phase integral and trajectory generator output action;

[0084] As one or more embodiments, such as Figure 1 As shown, the phase integration and trajectory generation module is used to generate joint angle commands based on the phase variables of each drive component, thereby obtaining the target servo position vector. This drives the robot's joints to perform periodic movements.

[0085] In this embodiment, the salamander-like robot consists of five independently controlled body parts: the left foreleg (LF), the left hind leg (LH), the right foreleg (RF), the right hind leg (RH), and the spine (S).

[0086] Each part consists of phase variables Controlled, of which .

[0087] The trajectory generator maps these phase variables to the desired angles of each joint, thereby outputting the target servo position. .

[0088] For each leg The desired angles of the three joints are defined as follows:

[0089]

[0090]

[0091]

[0092] in, The amplitude is a constant. This is the servo offset angle, used to provide the basic attitude. Indicates the first One leg The target angle of each servo motor.

[0093] The above definition ensures that each leg completes one phase cycle. The process involves two phases: swinging and support, within which the interval... Corresponding oscillation phase, interval Corresponding support phase.

[0094] The joint angles of the spine are defined as follows:

[0095]

[0096] Among them, the phase of the spinal joint Control the rhythm and direction of the overall body bending.

[0097] Based on the above definitions, the output vector of the trajectory generator can be obtained as follows:

[0098]

[0099] This vector is a sequence of attitude commands consisting of 15 target joint angles, which serves as the position reference input for subsequent joint-level controllers (e.g., proportional-integral-derivative controllers, PID controllers).

[0100] The aforementioned control structure decouples trajectory generation from the execution process, improving the overall system's stability and interpretability.

[0101] As an optional implementation, the phase variable is updated over time according to the phase integral law:

[0102]

[0103] in, Phase velocity represents the rate of change of phase per unit time. All outputs are from a reinforcement learning policy network, used to adjust the phase change rate of each body part, thereby achieving flexible omnidirectional gait adjustment; To control the time step, it is typically set to 0.1 s, corresponding to a control frequency of 10 Hz. The phase velocity can take positive or negative values, representing the forward and backward evolution of the phase, respectively, thus giving the robot flexible phase modulation capabilities.

[0104] As another implementation method, the phase velocity is composed of an action vector at each moment:

[0105]

[0106] in, The output from the upper-level reinforcement learning policy network is used to adjust the phase integration rate in real time. These represent the phase integral rates of the four legs: left front, left rear, right front, and right rear, respectively. This represents the phase integral rate of the torso segment.

[0107] Compared to traditional methods that directly optimize joint angles or predefined trajectory parameters, this embodiment achieves indirect control of movement rhythm and coordination by regulating the evolution of phase variables, thereby enabling the agent to autonomously form stable and efficient gait coordination patterns under different terrains and tasks.

[0108] S103: Reward feedback based on behavioral constraints;

[0109] In this embodiment, the behavior-constrained reward feedback is used to guide the reinforcement learning agent to generate high-quality motion behaviors according to the task objectives during the control of the salamander-like robot's movement. Specifically, this step constrains and guides the agent's behavior during the reinforcement learning training phase by setting a reward function, enabling it to maintain the stability and diversity of its motion while achieving the desired speed and turning.

[0110] The reward function is designed to guide the agent along the body coordinate system. shaft and Axial direction tracking of desired target velocity And achieve the desired yaw rate. The reward function consists of multiple sub-items, each used to reward or punish different behavioral characteristics, specifically including the following:

[0111] 1. Forward Motion Reward: Used to reward the robot's effective displacement along the target direction. Its calculation formula is as follows:

[0112]

[0113] in, They represent in shaft and Displacement increment in the axial direction, These are preset coefficients.

[0114] 2. Lateral motion penalty: This is used to penalize invalid motion perpendicular to the desired direction. Its calculation formula is as follows:

[0115]

[0116] in, This is a constant used to adjust the intensity of the penalty.

[0117] 3. Turning Bonus: When a target yaw rate exists. At that time, the robot is encouraged to turn quickly, using the following formula:

[0118]

[0119] in, This represents the actual change in yaw angle. This is the steering weighting coefficient.

[0120] 4. Penalty for Unintended Turning: Penalizes the robot for unnecessary turns when no turning command is given. The formula is:

[0121]

[0122] 5. Penalty for Unintended Translation: When there is no translation command, the robot is penalized for producing undesirable translational behavior. The formula is as follows:

[0123]

[0124] 6. Phase Coverage Reward: Used to encourage coverage of limb oscillations to promote the formation of diverse movement patterns. The formula is:

[0125]

[0126] in, This indicates the oscillation coverage of the limbs.

[0127] In summary, the total value of the reward function is defined as the weighted sum of its components:

[0128]

[0129] in, The weight parameters for each reward or penalty item are set according to the task requirements.

[0130] Through this behavior-constrained reward feedback mechanism, the agent can simultaneously achieve target following, motion smoothing, and phase diversity during reinforcement learning, thereby effectively improving the omnidirectional motion capability and adaptive performance of the salamander-inspired robot.

[0131] S104: Data augmentation based on morphological symmetry;

[0132] In this embodiment, the data augmentation based on morphological symmetry is used to enhance the training samples during the reinforcement learning training process by utilizing the structural symmetry of the salamander-inspired robot, thereby improving the sample efficiency and motion symmetry of policy learning, and thus improving the stability and naturalness of the robot's omnidirectional motion.

[0133] Specifically, the steps involve introducing morphological symmetry constraints during the training phase of a reinforcement learning algorithm (preferably Proximal Policy Optimization, PPO) to generate symmetric samples by applying the equivalent mapping relationship of the robot structure under spatial transformations. The resulting Symmetric Markov Decision Process (SymmetricMDP) ensures that the transition dynamics and reward function are consistent. Under the condition of invariance, achieve symmetric learning of the strategy.

[0134] In this process, the morphological reflection symmetry group of the salamander-inspired robot is first defined:

[0135]

[0136] in, Indicates identity transformation, This represents a symmetrical reflection transformation. This represents a left-right symmetrical reflection transformation. This represents a composite transformation of two symmetric operations.

[0137] In three-dimensional space, salamander-like robots can rearrange the phase relationships between different limbs through the aforementioned symmetric operations to achieve equivalent pose configurations. For any given state space... With action space The aforementioned symmetry group acts on both, transforming the original sample through reflection or combination. Mapping to equivalent symmetric samples .

[0138] During the training phase, for state-action pairs sampled from the environment By applying the above symmetry transformation, several enhanced samples can be obtained:

[0139]

[0140] These augmented samples share the same transition probabilities and reward functions as the original samples, thus maintaining the physical consistency of the learning process.

[0141] To reduce the policy bias introduced by augmented samples, this embodiment sets a zero bias parameter during policy network initialization and uses a large action variance to ensure that the initial policy satisfies approximately symmetry constraints.

[0142]

[0143] This ensures that the original samples and the augmented samples contribute equally to the gradient update and maintains the approximate equivariance of the policy during training.

[0144] The following effects can be achieved through the data augmentation mechanism based on morphological symmetry described above:

[0145] 1. Improve sample efficiency: Expand the effective training samples without increasing the actual number of interactions;

[0146] 2. Promote symmetry in omnidirectional motion: enable the trajectories generated by reinforcement learning strategies to exhibit consistent controllability in different directions;

[0147] 3. Enhance the naturalness and robustness of movement patterns: Through symmetry constraints, enable the robot to form a balanced and coordinated gait.

[0148] Therefore, the morphological symmetry data augmentation step not only improves the stability and generalization ability of policy training, but also structurally ensures the geometric consistency and biosimulation of the salamander-inspired robot's motion behavior.

[0149] S105: Strategy Iteration Update;

[0150] In one embodiment, step S105 is used to perform policy iterative updates to train and optimize the motion control strategy of the salamander-like robot based on the Proximal Policy Optimization (PPO) algorithm.

[0151] In this embodiment, both the policy network and the value network are implemented using a multi-layer perceptron (MLP) structure. They share the same feature extraction backbone network, differing only in the output layer dimension. The policy network generates the action distribution, while the value network predicts the expected reward value of the current state. The two networks are jointly optimized using the PPO algorithm to improve the stability of policy updates and sample utilization efficiency.

[0152] In one optional implementation, at the beginning of each training round, the phase variables are uniformly initialized, and the gait categories are randomly shuffled and then sequentially traversed. Simultaneously, corresponding motion commands are uniformly sampled for each gait category. The training process is performed using the MuJoCo physics engine, which possesses high-speed and high-precision dynamic simulation capabilities to generate state transition data for the salamander-inspired robot under different gaits and orientations.

[0153] In another embodiment, to achieve effective transfer from simulation to reality, the ground friction coefficient, robot mass distribution, and joint PID parameters are finely adjusted so that the dynamic characteristics of the simulation model are consistent with those of the actual robot, thereby improving the transferability of the strategy in the physical environment.

[0154] In one specific implementation, such as Figure 2 As shown, the experimental platform includes a salamander-inspired robot, a Qualisys motion capture system, a Wi-Fi communication module, and a ground control station. The motion capture system tracks reflective markers placed on the robot's fore and rear sections to acquire its planar pose information in real time, which is then used to establish the robot's body coordinate system. The ground control station runs the trained policy network, receives real-time pose data via Wi-Fi, calculates corresponding servo control commands, and sends them to the robot, achieving end-to-end closed-loop motion control.

[0155] In this embodiment, the objective function for PPO optimization can be expressed as:

[0156]

[0157] in, This represents the ratio of the probability of the current policy to that of the old policy. This is the generalized advantage estimation (GAE). is the shear coefficient.

[0158] By optimizing the above objective function, the policy network parameters are iteratively updated. In order to obtain the optimal strategy that can achieve efficient omnidirectional motion under different states and directions of motion. Example 2

[0159] This embodiment provides an omnidirectional motion control system for a salamander-like robot based on reinforcement learning; including:

[0160] The state acquisition module is configured to: acquire the robot's state information in real time, including joint angles, body posture, linear velocity, angular velocity and environmental feedback signals, through sensor components set on the salamander robot's body or an external motion capture system; at the same time, generate motion commands according to external task instructions or preset control targets, and use the state information and motion commands together as input to the control strategy;

[0161] The phase integration and trajectory generation module is configured to: integrate the time step through a phase integration mechanism to obtain a periodic phase variable; and input the phase variable and robot state information together into the trajectory generator to generate the target motion trajectory of each joint, so as to ensure the continuity and rhythm of the robot's motion, thereby forming the basic driving signal for omnidirectional motion.

[0162] The reward feedback module is configured to: calculate a comprehensive reward signal for guiding the optimization of the reinforcement learning policy, the reward signal including but not limited to motion directionality constraints, energy consumption constraints and posture stability constraints; and optimize the parameters of the policy network and value network based on the reward feedback to achieve balanced and coordinated omnidirectional motion behavior.

[0163] The symmetry enhancement module is configured to: utilize the morphological symmetry of the salamander-inspired robot's body structure to perform symmetric transformations on state-action pairs, including left-right mirror transformations, front-back swaps, and combinations thereof;

[0164] By expanding the training sample set through symmetry enhancement, the generalization ability and symmetry of the reinforcement learning model under different directional motions can be improved.

[0165] The policy optimization module is configured to: employ a reinforcement learning algorithm to iteratively update the policy network parameters using the aforementioned reward signal and symmetrically enhanced sample data; and optimize the policy by minimizing the objective function.

[0166] It should be noted that the aforementioned state acquisition module, phase integration and trajectory generation module, reward feedback module, symmetry enhancement module, and policy optimization module correspond to steps S101 to S105 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. As part of the system, these modules can be executed in a computer system, such as a set of computer-executable instructions.

[0167] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0168] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Example 3

[0169] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0170] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0171] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0172] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0173] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0174] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. Example 4

[0175] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0176] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for omnidirectional motion control of a salamander-like robot based on reinforcement learning, characterized in that, Includes the following steps: Acquire the state information and motion commands of the salamander-like robot; construct a state vector based on the state information and task commands, and take the task-level commands, body posture changes, phase encoding of limbs and torso, and phase coverage statistics of each leg as inputs; Based on the state vector, the target servo position command is generated using the phase integral and trajectory generation module, thereby driving the robot's joints to achieve periodic motion. Based on a reward function that includes motion constraints and behavior guidance, the actions generated by the reinforcement learning agent are evaluated and optimized. A data augmentation strategy utilizing morphological symmetry is proposed, which generates equivalent samples through symmetric group transformation to improve the generalization and symmetry constraints of the strategy. The policy network and value network are iteratively updated using reinforcement learning algorithms to obtain a stable and efficient omnidirectional motion control policy. The state vector is defined as follows: in These represent the desired forward and lateral velocities, respectively. Indicates the desired angular velocity; These represent the translation and rotation errors between the desired pose and the actual pose, respectively. Indicates the first Phase variables of a joint or body segment; Used to implement periodic encoding of phase; Indicates the first The phase coverage of a leg within a complete cycle is used to measure the periodic integrity of the gait. The phase integration and trajectory generation module generates the target servo angle and motion vector according to the following rules: First, the reinforcement learning policy network outputs action vectors at each time step. in: These represent the phase integral rates of the four legs: left front, left rear, right front, and right rear, respectively. This represents the phase integral rate of the torso segment; Subsequently, the phase variable is updated according to the following integration rule: in, To control the time step, Phase velocity represents the rate of change of phase per unit time. All outputs are from a reinforcement learning policy network, used to adjust the phase change rate of each body part, thereby achieving flexible omnidirectional gait adjustment; Based on the phase variable, the trajectory generator calculates the target angle of the servo motor: in, It is the amplitude constant; This refers to the servo motor offset angle; Indicates the first One leg The target angle of each servo motor; through the above phase integration and trajectory generation process, the salamander-inspired robot can achieve continuous, coordinated and periodically stable omnidirectional gait control.

2. The method as described in claim 1, characterized in that, The reward function is defined as a weighted sum of multiple sub-items: Among them, multiple sub-items These are: Forward motion reward: encourages the robot to generate effective displacement along the target direction; Lateral penalty: suppresses invalid lateral motion perpendicular to the main motion direction; Turning reward: provides a positive reward when turning in the desired angular velocity direction; Targetless turning penalty: restricts the robot from generating random rotations or unnecessary posture perturbations; Targetless translation penalty: prevents the robot from continuing to move when stationary or with a fixed target; Phase coverage reward: encourages the leg phase to be evenly distributed throughout the complete cycle to maintain gait coordination and stability. These are the weight parameters for each reward or penalty item.

3. The method as described in claim 1, characterized in that, The data augmentation of morphological symmetry is based on the left-right symmetry characteristics of the robot's body structure, through symmetry groups. Acting on the state space With action space Generate equivalent symmetric samples ,in, Indicates identity transformation, This represents a symmetrical reflection transformation. This represents a left-right symmetrical reflection transformation. This represents a composite transformation of two symmetric operations, which can improve the consistency and generalization ability of the policy in the symmetric action space without changing the semantics of the task.

4. The method as described in claim 1, characterized in that, Both the policy network and the value network adopt a multilayer perceptron structure and share a feature extraction layer. The policy network outputs action distribution parameters to generate phase velocity instructions, and the value network is used to estimate long-term rewards. The two are jointly trained through a reinforcement learning algorithm to achieve stable policy updates and efficient use of samples.

5. A reinforcement learning-based omnidirectional motion control system for a salamander-like robot, used to implement the method described in any one of claims 1-4, characterized in that, include: The state acquisition module is used to acquire pose, velocity and phase information in real time through the robot's own sensors or an external motion capture system; The phase integration and trajectory generation module is used to generate the target angle of the servo motor and drive each joint based on the phase integration equation and trajectory function. The reward feedback module is used to calculate the comprehensive reward function and guide the agent's policy update; The symmetry enhancement module is used to perform morphological symmetry transformations to generate equivalent training samples. The strategy optimization module is used to iteratively update the strategy using reinforcement learning algorithms, enabling autonomous control of the robot's omnidirectional motion.

6. An electronic device, characterized in that, include: One or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the processor executes the computer program, it implements the method of any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-4.

Images