Samander-imitated robot omnidirectional motion control method and system based on reinforcement learning

Abstract

The invention discloses a salamander-imitating robot omni-directional motion control method and system based on reinforcement learning, and belongs to the field of electromechanical system control. The method comprises the following steps: acquiring robot state information and a motion instruction; a periodic phase variable is generated through a phase integration module, and target actions of all joints are output through a track generator; designing a reward function based on behavior constraint, and constraining motion directivity and attitude stability; the morphological symmetry of the body structure of the salamander-imitating robot is utilized to symmetrically enhance the state-action sample so as to expand training data; and a reinforcement learning algorithm is adopted, and a stable and efficient omnidirectional motion control model is obtained according to a reward feedback and symmetric enhancement data iteration updating control strategy. The method can realize the generation of the omni-directional motion strategy driven by reinforcement learning without presetting a gait model, has the advantages of natural motion mode, high control precision and strong adaptability, and can be widely applied to the fields of complex terrain detection, intelligent inspection and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of electromechanical system control, and particularly relates to a method and system for omnidirectional motion control of a salamander robot based on reinforcement learning. BACKGROUND

[0002] A salamander quadruped robot is a type of bionic robot that imitates the body structure of amphibians such as salamanders. It has the characteristics of low center of gravity, high stability, and rich motion forms, and can achieve efficient motion in complex and variable terrain environments. Therefore, this type of robot has broad application prospects in disaster rescue, environmental monitoring, and field exploration.

[0003] Existing salamander robot motion control methods mainly rely on artificially designed control strategies. One method achieves periodic motion by predefining gait patterns, and another method is inspired by the Central Pattern Generator (CPG) in biology, and achieves smooth switching between different motion patterns by adjusting control parameters. Although this method can generate stable rhythmic motion, it still relies heavily on expert experience, and the structural constraints limit the diversity of motion behaviors, making it difficult to achieve high flexibility and high adaptability of omnidirectional motion control.

[0004] In recent years, with the development of intelligent control technologies such as deep reinforcement learning, researchers have begun to try to automatically obtain motion control strategies through learning-driven methods. Some studies use end-to-end neural networks to directly output joint angles, and use reference trajectories or foot contact signals as learning targets to achieve multiple gait patterns. However, this method often relies on pre-collected reference motion data, and for salamander-inspired robots, there is a lack of available high-quality motion data sets, limiting the generalization ability and scalability of this method.

[0005] Another type of reference-free learning method attempts to obtain diverse motion patterns through unsupervised skill discovery or entropy-maximizing objective functions, but the generated motion often lacks physical rationality, exhibiting chaotic or poorly coordinated actions, making it difficult to be directly applied to real robot systems.

[0006] In addition, in the omnidirectional motion control of salamander robots, the exploration of the strategy in different directions is often uneven, and it is easy to fall into local optimization in some directions, resulting in asymmetric motion performance. Recent studies have shown that using the morphological symmetry of the robot as a physical prior can effectively improve learning efficiency and motion consistency, thereby obtaining more natural motion behaviors. However, existing research lacks the use of symmetry rules in salamander robot motion, and lacks a systematic control framework for omnidirectional motion.

[0007] Therefore, how to enable the salamander robot to autonomously learn a diversified, stable and symmetrical omnidirectional gait without referring to a trajectory and achieve a balanced state-action space exploration has become a technical problem to be solved in the field. SUMMARY

[0008] To solve the problems of single gait pattern, asymmetric behavior and low learning efficiency in the motion control of existing salamander robots, the present application provides an omnidirectional motion control method and system for salamander robots based on reinforcement learning. The method can enable the salamander robot to autonomously learn a diversified, stable and symmetrical omnidirectional gait without referring to a trajectory, thereby improving the autonomous motion ability and control robustness of the robot in complex environments.

[0009] In a first aspect, the present application provides an omnidirectional motion control method for salamander robots based on reinforcement learning, comprising the following steps: Obtain the state information and motion instructions of the salamander robot; construct a state vector based on the state information and task instructions, and take the task-level instructions, body posture changes, phase encoding of limbs and trunk, and phase coverage statistics of each leg as inputs; According to the state vector, a target rudder position instruction is generated using a phase integral and trajectory generation module, thereby driving the periodic motion of each joint of the robot; Based on a reward function containing motion constraints and behavior guidance, the actions generated by the reinforcement learning agent are evaluated and optimized; Use a data augmentation strategy based on morphological symmetry to generate equivalent samples through symmetric group transformation to improve the generalization and symmetry constraints of the strategy; Use a reinforcement learning algorithm to iteratively update the strategy network and value network to obtain a stable and efficient omnidirectional motion control strategy.

[0010] Further, the state vector is defined as:

[0011] where represent the desired forward and lateral velocities, respectively; represent the desired angular velocity; represent the translational and rotational errors between the desired pose and the actual pose, respectively; represent the phase variable of the th joint or body segment; is used to implement periodic encoding of the phase; represent the phase coverage of the th leg in a complete cycle, which is used to measure the periodicity of the gait.

[0012] Furthermore, the phase integration and trajectory generation module generates the target servo angle and motion vector according to the following rules: First, the reinforcement learning policy network outputs action vectors at each time step.

[0013] in: These represent the phase integral rates of the four legs: left front, left rear, right front, and right rear, respectively. This represents the phase integral rate of the torso segment; Subsequently, the phase variable is updated according to the following integration rule:

[0014] in, To control the time step, Phase velocity represents the rate of change of phase per unit time. All outputs are from a reinforcement learning policy network, used to adjust the phase change rate of each body part, thereby achieving flexible omnidirectional gait adjustment; Based on the phase variable, the trajectory generator calculates the target angle of the servo motor:

[0015] in, It is the amplitude constant; This refers to the servo motor offset angle; Indicates the first One leg The target angle of each servo motor; through the above phase integration and trajectory generation process, the salamander-inspired robot can achieve continuous, coordinated and periodically stable omnidirectional gait control.

[0016] Furthermore, the reward function Defined as a weighted sum of multiple sub-items:

[0017] The sub-items are as follows: Forward motion reward: encourages the robot to generate effective displacement along the target direction; Lateral penalty: suppresses invalid lateral motion perpendicular to the main motion direction; Turning reward: provides a positive reward when turning in the desired angular velocity direction; Targetless turning penalty: restricts the robot from generating random rotations or unnecessary posture perturbations; Targetless translation penalty: prevents the robot from continuing to move when stationary or with a fixed target; Phase coverage reward: encourages the leg phase to be evenly distributed throughout the complete cycle to maintain gait coordination and stability; and the weight parameters for each reward or penalty item.

[0018] Furthermore, the morphological symmetry data augmentation is based on the left-right symmetry characteristics of the robot's body structure, through symmetry groups. acts on state space with action space , generating equivalent symmetric samples wherein, denotes the identity transformation, denotes the front-back symmetric reflection transformation, denotes the left-right symmetric reflection transformation, denotes the composite transformation of the two symmetric operations, which can improve the consistency and generalization ability of the policy in the symmetric action space without changing the semantics of the task.

[0019] In a second aspect, the present application provides a salamander robot omnidirectional motion control system based on reinforcement learning, comprising: 1. State acquisition module: configured to acquire robot state information and external motion instructions; 2. Phase integration and trajectory generation module: configured to generate periodic trajectories by phase integration and output action signals; 3. Reward feedback module: configured to calculate behavior constraint rewards and provide policy optimization signals; 4. Symmetry enhancement module: configured to perform morphological symmetry enhancement on training samples; 5. Policy update module: configured to perform iterative updates of policy parameters based on reinforcement learning algorithms.

[0020] In a third aspect, the present application further provides an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the stored one or more computer programs to enable the electronic device to perform the method of the first aspect.

[0021] In a fourth aspect, the present application further provides a computer readable storage medium for storing computer instructions, which are executed by a processor to complete the method of the first aspect.

[0022] In a fifth aspect, the present application further provides a computer program product comprising a computer program, which, when running on one or more processors, is used to implement the method of any one of the preceding first aspect.

[0023] Compared with the prior art, the present application has the following beneficial effects: The reinforcement learning framework proposed in the application can autonomously learn diversified gait patterns without relying on any reference trajectory or external data set; by introducing a phase variable driving mechanism, coordinated control of the salamander robot's spine and limbs is achieved, significantly improving motion stability and flexibility; the designed phase coverage reward term encourages the robot to explore a wider gait space, effectively avoiding the occurrence of static or repetitive gaits; combined with the morphological symmetry enhancement strategy, the learning process maintains structural symmetry in the state-action space, significantly improving motion consistency and the generalization ability of the control strategy; the system structure is clear, the module division is reasonable, and it can be directly applied on simulation and real hardware platforms, and has good scalability and practical value. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings accompanying the specification of this application are used to provide further understanding of the application, the illustrative embodiments of the application and the description thereof serve to explain the application, and do not constitute an improper limitation on the application.

[0025] Figure 1 A schematic diagram of a kind of omnidirectional motion control method of salamander robot based on reinforcement learning.

[0026] Figure 2 A kind of experimental platform diagram of the omnidirectional motion control method of salamander robot based on reinforcement learning. DETAILED DESCRIPTION

[0027] The technical solutions of the application will be described in detail below with reference to the drawings and embodiments. It should be pointed out that the following detailed description is exemplary, and is intended to provide further description of the application. Unless otherwise specified, all technical and scientific terms used in the application have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0028] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] The embodiments in the application and the features in the embodiments can be combined with each other without conflict. Embodiment 1

[0030] As Figure 1As shown, the embodiment provides a method for omnidirectional motion control of salamander robot based on reinforcement learning; comprising: S101: obtaining robot state information and motion instructions; Constructing a state vector based on the state information and task instructions. The state vector is denoted as:

[0031] for comprehensively describing the motion state of the robot at time t, which includes task-level instructions, body posture changes, phase information of limbs and trunk, and phase coverage statistics of each leg.

[0032] wherein respectively represent the expected forward and lateral velocities; represents the expected angular velocity; respectively represent the translational and rotational errors between the expected pose and the actual pose; represents the phase variable of the th joint or body segment; for realizing the periodic encoding of the phase; represents the phase coverage rate of the th leg in a complete cycle, which is used to measure the cycle integrity of the gait.

[0033] As an optional implementation, the state information can be obtained by an action capture system to obtain the position and attitude angle information of the salamander robot in the world coordinate system; As another implementation, the state information can also be collected in real time by the inertial measurement unit (IMU) of the robot body, joint encoders, and visual sensors.

[0034] In terms of motion instructions, let the command vector be:

[0035] wherein, the translational instruction is defined as a two-dimensional vector in the robot body coordinate system:

[0036] wherein, represents the expected horizontal motion direction (yaw angle).

[0037] When the translational instruction is not given, define .

[0038] The rotation instruction corresponds to the three cases of clockwise rotation, no rotation, and counterclockwise rotation, respectively.

[0039] During training and testing, the translation and rotation commands are combined into seven different motion patterns to enrich the data distribution for policy learning.

[0040] During the robot motion, due to the periodic characteristics of the gait, the linear and angular velocities fluctuate significantly over time, making direct tracking unstable.

[0041] Therefore, the robot pose changes are recorded in a given time interval to obtain the steady-state motion information.

[0042] At each time step, the robot's pose in the world coordinate system is recorded in a stack with a length of .

[0043] By transforming the pose at the latest time to the body coordinate system at the earliest time, the translation and rotation changes of the robot in this time interval can be calculated: , To stabilize the learning process of the phase information, the sinusoidal and cosine forms are used to encode the oscillation phases of the four legs and the spine segments: , In the formula, LF, LH, RF, RH, and S represent the left front leg, the left rear leg, the right front leg, the right rear leg, and the spine, respectively.

[0044] This representation can smoothly change on the unit circle, avoiding discontinuity caused by jumping.

[0045] To further quantify the motion performance of each leg in the gait cycle, the phase coverage rate is calculated.

[0046] Specifically, the phase sequence is arranged in ascending order, and the circular difference between adjacent phases and the closed difference between the beginning and end are calculated, taking the maximum gap , then the uncovered area length is .

[0047] Thus, the phase coverage rate is:

[0048] where represents the coverage ratio of the th leg.

[0049] This index reflects the degree of phase traversal of each leg in a complete gait cycle, which helps to evaluate the gait balance.

[0050] In summary, the robot state vector Composed of task instructions, attitude changes, phase encoding, and coverage, it can describe the robot's current motion state and provide high-dimensional state input for subsequent phase integration and trajectory generation processes.

[0051] The aforementioned state information can be collected collaboratively by the motion capture system, IMU sensors, and encoders, forming the basis for the fusion of global and local state perception of the robot.

[0052] S102: Phase integral and trajectory generator output action; As one or more embodiments, such as Figure 1 As shown, the phase integration and trajectory generation module is used to generate joint angle commands based on the phase variables of each drive component, thereby obtaining the target servo position vector. This drives the robot's joints to perform periodic movements.

[0053] In this embodiment, the salamander-like robot consists of five independently controlled body parts: the left foreleg (LF), the left hind leg (LH), the right foreleg (RF), the right hind leg (RH), and the spine (S).

[0054] Each part consists of phase variables Controlled, of which .

[0055] The trajectory generator maps these phase variables to the desired angles of each joint, thereby outputting the target servo position. .

[0056] For each leg The desired angles of the three joints are defined as follows:

[0057]

[0058]

[0059] in, The amplitude is a constant. This is the servo offset angle, used to provide the basic attitude. Indicates the first One leg The target angle of each servo motor.

[0060] The above definition ensures that each leg completes one phase cycle. The process involves two phases: swinging and support, within which the interval... Corresponding oscillation phase, interval Corresponding support phase.

[0061] The joint angles of the spine are defined as follows:

[0062] where the phase of the spinal joint controls the rhythm and direction of the overall body bending.

[0063] According to the above definitions, the output vector of the trajectory generator is:

[0064] This vector is a sequence of pose commands composed of 15-dimensional joint angle targets, serving as the position reference input of the subsequent joint-level controller (e.g., proportional-integral-derivative controller, PID).

[0065] The above control structure decouples the trajectory generation and execution processes, improving the stability and interpretability of the overall system.

[0066] As an optional implementation, the phase variable is updated over time according to the phase integral law:

[0067] where, is the phase velocity, representing the rate of change of the phase per unit time, are output by the upper reinforcement learning policy network, used to adjust the phase change speed of each body part, achieving flexible omnidirectional gait adjustment; is the control time step, usually set to 0.1 s, corresponding to a control frequency of 10 Hz. The phase velocity can take positive or negative values, representing forward and reverse evolution of the phase, respectively, thus giving the robot flexible phase modulation capabilities.

[0068] As another implementation, the phase velocity at each time step forms an action vector:

[0069] where, is output by the upper reinforcement learning policy network, used to adjust the phase integral rate in real time. respectively represent the phase integral rates of the left front, left rear, right front, and right rear legs; represents the phase integral rate of the torso segment.

[0070] Compared to traditional methods of directly optimizing joint angles or predefining trajectory parameters, this embodiment achieves indirect control over movement rhythm and coordination by regulating the evolution of the phase variable, enabling the agent to autonomously form stable and efficient gait coordination patterns under different terrains and tasks.

[0071] S103: reward feedback based on behavior constraints; In the embodiment, the reward feedback based on behavior constraints is used to guide the reinforcement learning agent to generate high-quality motion behavior according to the task target in the process of controlling the motion of the robot of the notophthalmus. Specifically, the step sets the reward function to constrain and guide the behavior of the agent in the reinforcement learning training stage, so that the agent can achieve the expected speed and steering while maintaining the stability and diversity of the motion.

[0072] The reward function aims to guide the agent to track the expected target speed along the axis and axis of the body coordinate system , and achieve the expected yaw rate . The reward function is composed of multiple sub-items for rewarding or punishing different behavior characteristics, including the following contents: 1. Forward motion reward term: used to reward the effective displacement of the robot along the target direction, and the calculation formula is:

[0073] wherein, respectively represent the displacement increment in the axis and axis direction, is a preset coefficient.

[0074] 2. Lateral motion penalty term: used to punish the invalid motion perpendicular to the expected direction, and the calculation formula is:

[0075] wherein, is a constant for adjusting the punishment intensity.

[0076] 3. Steering reward term: when there is a target yaw rate , the robot is encouraged to quickly steer, and the formula is:

[0077] wherein, is the actual yaw change amount, is the steering weight coefficient.

[0078] 4. No-target steering penalty term: when there is no steering instruction, the robot is punished for unnecessary rotation, and the formula is:

[0079] 5. No-target translation penalty term: when there is no translation instruction, the robot is punished for non-expected translation behavior, and the formula is:

[0080] 6. Phase coverage reward term: to encourage the coverage of each limb oscillation to promote the formation of diverse motion patterns, the formula is:

[0081] wherein, represents the oscillation coverage of the limbs.

[0082] According to the above, the total value of the reward function is defined as the weighted sum of each sub-term:

[0083] wherein, is the weight parameter of each reward or penalty term, which is set according to the task requirements.

[0084] Through the reward feedback mechanism based on behavior constraints, the agent can realize target following, action smoothing and phase diversity in the reinforcement learning process, thereby effectively improving the omnidirectional motion ability and adaptive performance of the salamander robot.

[0085] S104: data augmentation based on morphological symmetry; In this embodiment, the data augmentation based on morphological symmetry is used to enhance the training samples in the reinforcement learning training process by using the structural symmetry of the salamander robot, so as to improve the sample efficiency and motion symmetry of policy learning, and further improve the stability and naturalness of the omnidirectional motion of the robot.

[0086] Specifically, the step introduces morphological symmetry constraints in the reinforcement learning algorithm (preferably Proximal Policy Optimization, PPO) training stage, and uses the equivalent mapping relationship of the robot structure under spatial transformation to generate symmetric samples. The symmetric Markov decision process (Symmetric MDP) constructed in this way can realize symmetric learning of the policy under the condition of ensuring that the transition state and reward function are invariant.

[0087] In this process, first define the morphological reflection symmetry group of the salamander robot:

[0088] wherein, represents the identity transformation, represents the front-back symmetric reflection transformation, represents the left-right symmetric reflection transformation, represents the composite transformation of the two symmetric operations.

[0089] In three-dimensional space, the salamander robot can rearrange the phase relationship between different limbs through the above symmetry operation to achieve equivalent posture configuration. For any given state space and action space , the above symmetry group acts on both of them, and through reflection or combination transformation, the original sample is mapped to the equivalent symmetric sample .

[0090] In the training phase, for the state-action pair sampled from the environment , by applying the above symmetry transformation, a number of augmented samples can be obtained:

[0091] These augmented samples share the same transition probability and reward function with the original sample, thereby maintaining the physical consistency of the learning process.

[0092] To reduce the policy drift introduced by augmented samples, in this embodiment, a zero bias parameter is set when the policy network is initialized, and a larger action variance is used to make the initial policy satisfy the approximate symmetry constraint:

[0093] so as to ensure that the original sample and the augmented sample have equal contribution in the update gradient, and maintain the approximate invariance of the policy during the training process.

[0094] Through the above data augmentation mechanism based on morphological symmetry, the following effects can be obtained: 1. Improve sample efficiency: expand the effective training samples without increasing the actual interaction times; 2. Promote the symmetry of omnidirectional motion: make the trajectory generated by the reinforcement learning policy show consistent controllability in different directions; 3. Enhance the naturalness and robustness of motion patterns: through symmetry constraint, the robot forms a balanced and coordinated gait.

[0095] Therefore, the morphological symmetry data augmentation step not only improves the stability and generalization ability of policy training, but also structurally guarantees the geometric consistency and biological simulation of the salamander robot motion behavior.

[0096] S105: policy iteration update; In one embodiment, step S105 is configured to perform policy iteration update to train and optimize the motion control policy of the salamander robot based on the Proximal Policy Optimization (PPO) algorithm.

[0097] In this embodiment, both the policy network and the value network are implemented using a Multi-Layer Perceptron (MLP) structure, and share the same feature extraction backbone network, only differing in the output layer dimension. The policy network is used to generate the action distribution, and the value network is used to predict the expected return value of the current state. The two networks are jointly optimized by the PPO algorithm to improve the stability of policy updating and the efficiency of sample utilization.

[0098] In an alternative implementation, the phase variable is uniformly initialized at the beginning of each training round, and the gait categories are randomly shuffled and sequentially traversed. At the same time, the corresponding motion instructions are uniformly sampled for each gait category. The training process is carried out in the MuJoCo physics engine, which has high-speed and high-precision dynamics simulation capabilities, for generating state transition data of the salamander robot in different gaits and directions.

[0099] In another embodiment, to achieve effective migration from simulation to reality (Sim-to-Real), the ground friction coefficient, robot mass distribution, and joint PID parameters are finely adjusted to ensure that the dynamics characteristics of the simulation model are consistent with the actual robot, thereby improving the migratability of the policy in the physical environment.

[0100] In a specific implementation, as shown in Figure 2 , the experimental platform includes a salamander robot, a Qualisys motion capture system, a Wi-Fi communication module, and a ground control station. The motion capture system tracks the reflective marker points placed on the front and rear body segments of the robot to obtain real-time planar pose information, and is used to establish the robot body coordinate system. The ground control station runs the trained policy network, receives real-time pose data through Wi-Fi, and calculates the corresponding rudder control instructions to be sent to the robot, realizing end-to-end closed-loop motion control.

[0101] In this embodiment, the objective function of PPO optimization can be represented as:

[0102] where, is the probability ratio of the current policy and the old policy, is the Generalized Advantage Estimation (GAE), is the clipping coefficient.

[0103] By optimizing the above objective function, the policy network parameters are iteratively updated to obtain the optimal policy that can achieve efficient omnidirectional motion in different gaits and motion directions. Embodiment 2

[0104] The embodiment provides a salamander robot omnidirectional motion control system based on reinforcement learning; comprising: A state acquisition module configured to acquire state information of the robot in real time through a sensor component arranged on a body of the salamander robot or an external motion capture system, the state information including joint angles, body postures, linear velocities, angular velocities and environmental feedback signals; and generate motion instructions according to external task instructions or preset control targets, and input the state information and the motion instructions together as inputs of a control strategy; A phase integral and trajectory generation module configured to integrate time steps through a phase integral mechanism to obtain periodic phase variables; and input the phase variables and the state information of the robot together into a trajectory generator to generate target motion trajectories of each joint, so as to ensure continuity and rhythm of robot motion, thereby forming a basic driving signal of omnidirectional motion; A reward feedback module configured to calculate a comprehensive reward signal for guiding optimization of a reinforcement learning strategy, the reward signal including but not limited to motion directionality constraints, energy consumption constraints and posture stability constraints; and optimize parameters of a strategy network and a value network based on the reward feedback, so as to realize balanced and coordinated omnidirectional motion behavior; A symmetry enhancement module configured to perform symmetric transformation on state-action pair samples by using morphological symmetry of a body structure of the salamander robot, including left-right mirror transformation, front-back exchange and combinations thereof. The training sample set is expanded through symmetry enhancement, so as to improve generalization ability and symmetry of the reinforcement learning model under different direction motions.

[0105] A strategy optimization module configured to use a reinforcement learning algorithm to iteratively update parameters of a strategy network by using the reward signal and the sample data after symmetry enhancement; and perform strategy optimization by minimizing an objective function.

[0106] It should be noted that the state acquisition module, the phase integral and trajectory generation module, the reward feedback module, the symmetry enhancement module and the strategy optimization module correspond to steps S101 to S105 in the first embodiment, and the above modules and corresponding steps have the same examples and application scenarios as those realized by the first embodiment, but are not limited to the content disclosed in the first embodiment. The above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0107] The description of each of the above embodiments has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0108] The proposed system can be implemented in other ways. For example, the system embodiments described above are only illustrative, and the division of the above modules is only a logical functional division. In actual implementation, another division manner can be used, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed. Embodiment 3

[0109] The embodiment also provides an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method in the above embodiment one.

[0110] It should be understood that, in the embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSPs, application-specific integrated circuits ASICs, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0111] The memory can include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0112] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0113] The method in the embodiment one can be directly embodied as hardware processor execution completion, or executed by hardware and software modules in the processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0114] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. Example 4

[0115] The embodiment also provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in the embodiment one.

[0116] The above only provides the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for controlling the omni-directional motion of a salamander robot based on reinforcement learning, characterized in that, The method comprises the following steps: acquiring state information and motion instructions of the salamander robot; constructing a state vector based on the state information and task instructions, taking task-level instructions, body posture changes, phase encoding of limbs and trunk, and phase coverage statistics of each leg as inputs; generating target servo position instructions using a phase integral and trajectory generation module based on the state vector, thereby driving each joint of the robot to achieve periodic motion; evaluating and optimizing the actions generated by the reinforcement learning agent based on a reward function that includes motion constraints and behavior guidance; using a data augmentation strategy based on morphological symmetry to generate equivalent samples through symmetric group transformation to improve the generalization and symmetry constraints of the strategy; iteratively updating the strategy network and value network using a reinforcement learning algorithm to ultimately obtain a stable and efficient omnidirectional motion control strategy.

2. The method of claim 1 wherein, The state vector is defined as: where denote the desired forward and lateral velocities, respectively; denote the desired angular velocity; denote the translational and rotational errors between the desired and actual poses, respectively; denote the phase variable of the th joint or body segment; for implementing a periodic encoding of the phase; denote the phase coverage of the th leg over a full cycle, used to measure the cycle integrity of the gait.

3. The method of claim 1 wherein, The phase integral and trajectory generation module generates target servo angles and action vectors according to the following rules: First, the reinforcement learning strategy network outputs an action vector at each time step wherein: respectively represent the phase-integrated rate of the left front, left rear, right front, and right rear legs; represents the phase-integrated rate of the trunk segment; Subsequently, the phase variable is updated according to the following integral rule: wherein, is a control time step, is a phase velocity, representing the rate of change of the phase per unit time, are all output by the reinforcement learning policy network, used to adjust the phase change rate of each body part to achieve flexible omnidirectional gait adjustment; Based on the phase variable, the trajectory generator calculates the target angle of the servo: wherein, is an amplitude constant; is a rudder bias angle; denotes the target angle of the leg of the th rudder; through the above phase integral and trajectory generation process, the salamander robot can realize continuous, coordinated and periodic stable omnidirectional gait control.

4. The method of claim 1, wherein, The reward function defined as a weighted sum of a plurality of sub-items: wherein the plurality of sub-items respectively: forward motion reward: encourage the robot to produce effective displacement along the target direction; lateral penalty: suppress ineffective lateral motion perpendicular to the main motion direction; turning reward: give positive reward when turning in the direction of desired angular velocity; aimless turning penalty: limit the robot to produce random rotation or unnecessary pose disturbance; aimless translation penalty: avoid the robot to keep moving when stationary or target fixed; phase coverage reward: encourage the leg phases to be evenly distributed within a complete cycle to maintain gait coordination and stability; are weight parameters for each reward or penalty item.

5. The method of claim 1, wherein, The data enhancement of morphological symmetry is based on the left-right symmetry characteristics of the robot body structure, and is implemented by the symmetry group acts on the state space and the action space to generate equivalent symmetric samples wherein, represents an identity transformation, represents a front-back symmetric reflection transformation, represents a left-right symmetric reflection transformation, represents a composite transformation of the two symmetric operations, which can improve the consistency and generalization ability of the policy in the symmetric action space without changing the semantics of the task.

6. The method of claim 1, wherein, Both the strategy network and the value network use a multilayer perceptron structure and share a feature extraction layer; the strategy network outputs action distribution parameters to generate phase speed instructions, and the value network is used to estimate long-term returns; both are jointly trained through a reinforcement learning algorithm to achieve stable strategy updates and efficient sample utilization.

7. A reinforcement learning based newt robot omnidirectional motion control system for implementing the method of any one of claims 1-6, characterized by, It comprises: a state acquisition module for acquiring real-time pose, velocity, and phase information through robot body sensors or external motion capture systems; a phase integral and trajectory generation module for generating servo target angles and driving each joint according to the phase integral equation and trajectory function; a reward feedback module for calculating a comprehensive reward function and guiding the strategy update of the agent; a symmetry enhancement module for performing morphological symmetry transformation to generate equivalent training samples; a strategy optimization module for strategy iterative update using a reinforcement learning algorithm to achieve autonomous control of the robot's omnidirectional motion.

8. An electronic device, characterized by comprising: It comprises: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the computer program is stored in the memory; when the processor executes the computer program, the method of any one of claims 1-6 is realized.

9. A computer readable storage medium characterized by, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1-6.

Images